Fallout 2 mod FO2 Engine Tweaks (Sfall)

sfall Arrays proposition

Am I the only one annoyed so much about how arrays work in sfall?

The problem is that there are 2 types of arrays:
1) "permanent" arrays which exist forever after they are created (unless implicitly freed), even between savegames (a bug in script code can lead to enormous save file size, and you can't easily know what arrays are "unused" to clean them out)
2) "temporary" arrays which exist for single frame and then deleted. They are useful in cases where you only need some small array in one function and don't want to bother about memory leaks. BUT, you can't use them in global script variable (which defined with "variable" keyword outside of procedure), which is really annoying.

HOWEVER, both array types are absolutely not something that regular programmer will understand as "arrays". By default, arrays in all other languages is a piece of information that is stored in memory. When you want array to be saved to file/savegame/somewhere else you usually use some kind of function for that. So you don't want "default" array type to be saved into savegames.
Temporary arrays are a bit different, because in many modern languages, there are things called ref counters, which can automatically free objects/arrays from memory after you don't need them (after last link is removed), but since we don't have such thing (and implementing one would be a pain), "temp_array" as it currently exist in sfall may be an option.

I was using a workaround for this problem in my own scripts:
- defined single procedure which creates permanent array and put it's ID in given sfall global variable
- I never use create_array directly, only use that procedure
But, it's not perfect. In some cases, when I need to pass my array between several scripts (like array of traps), this solution is ok (because I need to somehow pass variable ID between scripts and this is the only way since "imported" variables won't work with global/hook scripts). BUT, in other cases I only use array in one script.
Example:
- I have a list of some PIDs in mod's ini file.
- What I want to do is parse that list (text string) into an array of integers during script initialization (which happens after game load) and use that array efficiently throughout the lifespan of that script
- in this case I have to still store array and it's ID in savegame file, which is unnecessary, since I re-generate this array everytime and use it only in one script.
- what I want (and expect to be able) to do is use global script variable (not sfall global) like this:

Code:

variable pidList;

procedure start begin
  if (...) then begin
     pidList := create_array(10, 4); 
    // etc...

And not worry about savegames, because in real world arrays only automatically stored in memory.
In this case I don't need to worry about memory leak, because I will create this array only at script load, and I don't need to free it because it won't go to my savegame and will be freed from memory after game load.


I have a working implementation of third array type which don't automatically go into savegames in my current sfall build. But, there is several issues with that:
- To retain backward compatibility with older scripts, I will need to leave "create_array", "temp_array", "fix_array" functions as they were and add some kind of new methods. BUT, this will not fix the main problem with arrays (create_array will remain and will still be dangerous to use).
- I don't really want to add new opcodes to both sfall, compiler and decompiler, too...
- If my implementation will work differently from sfall master, I will have to apply my changes every time someone changes sfall
- While currently, RP don't use arrays at all, some other scripts might be using them in future and I will have a problem with compatibility.

So I propose to discuss how arrays would be implemented best to the point of including this into base sfall (if Timeslip approves). I can do sfall coding/compiling myself (since I already have working implementation).
My current (not final) proposed course of actions:
1) create_array will always create "normal" array (not temp and not permanent)
2) temp_array will still create arrays that delete themselves at the end of the frame (I'm not sure about this, but at least you can expect something like this when it's called "temp")
3) (maybe) fix_array will turn normal array into permanent. This is easy to implement, but more logical thing to do would be to add functions like "set_sfall_array" and "get_sfall_array" that would work similarly to "get_sfall_global_int" and "set_sfall_global" (you don't "turn" array into permanent, you just do a one-time operation of storing normal array in the global sfall arrays storage; and when you "get_sfall_array" it will create a copy of stored array as a normal sfall array)
4) All existing mods/scripts that use arrays will have to be changed/recompiled. I know, this sounds terrible, but as I looked, there are not many mods that use them (RP doesn't use them, and from all mods I've seen only my and Jim's mod used them).
5) I can also add a ddraw.ini setting that will allow switching between old and new array behavior (for backward compatibility), but for all new mods it will be recommended to use new mode.

Additional ideas about other aspects of arrays:
1) Currently, we can store values of different types in arrays, but the downside is: you have to define number of bytes for all array elements. While it's normal thing for C programmer to do, we are talking high level language here and this is an implementation detail. In other words, scripter doesn't need to be bothered by how many bytes his value takes in memory (unless he wants to, to work with binary files for example).
2) While having array dereferencing operator is cool (shortcut to get_array/set_array), we also need new syntax to define constant arrays. Like this:

Code:

array1 := {5, 7, 8, 1, 2};

Could be translated like so:

Code:

array1 := create_array(5);
set_array(array1, 0, 5);
set_array(array1, 1, 7);
set_array(array1, 2, 8);
set_array(array1, 3, 1);
set_array(array1, 4, 2);

3) We really need some kind of maps, objects or associative arrays (with numeric or string keys). While this can be easily done even with current arrays, there can be no readable shortcut for this. Best we can do now is something like:

Code:

ASSOC(array1, "key", "value");

but what would be readable is something like this:

Code:

array1["key"] := "value";

I imagine it might be done by adding new function "create_map" which would create just another array but with some flag in it, telling sfall that this is an associative array. It will be stored differently and when you use "set_array" it will check, if it's a map, allow index of any type and value.

I will think more about this and when I'll see the whole picture, I will implement it.

 

Unrelated bug report. ScriptExtender.cpp, StringSplit() function.

Code:

id=TempArray(count, maxlen);

This should be:

Code:

id=TempArray(count, maxlen+1);

Because maxlen is maximum string length after split, but you need 1 additional byte for null-termination. This bug caused several unstable problems with scripts.

Workaround function:

Code:

/**
 * Workaround for string_split bug in sfall 3.3
 */
procedure string_split_safe(variable str, variable split) begin
   variable lst;
   str += split;
   lst := string_split(str, split);
   resize_array(lst, len_array(lst) - 1);
   return lst;
end

 

(sorry for third post in a row, but) I really want to contribute with new features, engine hacks, bug fixes. Can I do that somehow?

Timeslip doesn't seem to show up here. Is she the only project maintainer? Does anyone else has permission to commit something?

My plan:
1) Fix string_split
2) Rewrite arrays like in post above
3) Add several string-related functions (if I can)
4) Add function "set_explosion_radius"

 

Mash and Haenlomal AFAIK, but Haenlomal has been inactive for quite some time.
I'm curious about the explosion radius, does it really work?

 

It worked for me, check the other thread. I only found explosion radius for grenades and rockets though.

Another question, what tools people typically use here to map fallout2.exe? Does anyone have some kind of disassembly database for it with different subroutines/variables named?
I started to map out some subroutines using IDA, but there is a lot of code

 

I was hoping it is possible for explosions to light up an area for a second - just like real life...it would cool with night encounters.

 

Strange thing. Most misc prototypes has proper light intensity/radius defined (if you place rocket or explosion in Mapper at night - you will see it), but when actual explosion happens or rocket fly - no light
Need to understand how game creates projectiles and explosion effects. I have a hunch that they are not actual objects, but only graphics is taken from prototype and animated directly on the screen.

Tried to search for many misc prototype IDs (hardcoded in engine) using decompiler, found a couple but changing them had no effect (tried to replace explosion PID with emp explosion).

 

@phobos2077

The skill level of folks here on our forum varies widely, and many readers are beginners at programming. Could you give short description of what an array is and why a person might want to use such a thing, or perhaps a brief example of how it's a time-saver or problem solver?


@ sfall discussion

I can't get play_sfall_sound to work. I use Ruby and the command compiles fine with my script compiler, but during actual game-play any call to play_sfall_sound causes the game to crash (the sound files are in properly named, in .wav format, and in the directory specified in the script).

I read over here:

I don't understand that-- what is one supposed to do then with the resulting variable (in this case, "tmp") in order to get the sound to play? Some other step would be necessary.

Thanks

 

Well basically an array is a bunch of values stored in single variable (pointer), you can have as many values in one array as you want. In many cases, when you need to work with many of the same (anything), arrays are great.
My English is not very good, so I'll just write some real examples.

1) Let's say we have a mod that delete items from inventory of dying critters (like stimpaks) to make game harder. And you want this configurable, so you add an INI config directive like this:

Code:

...
removePids=40,77,124

And checking if certain item would qualify for removal would be simple:

Code:

// do this once on script init:
// this is just a function that splits string and then converts all values to integers
pidsList := string_split_ints(get_ini_string(....), ","); 

....

// use this in hs_ondeath to check if item qualify:
if (is_in_array(obj_pid(item), pidsList)) then ...

Without arrays we would've needed some dirty solutions like in party orders addon (where NPCs PIDs are defined on different lines, and you have to iterate over INI sections every time - this is a huge disk performance issue), or using some hard-to-understand functions to iterate over string of tokens using "tokenize" function.

2) Let's say we need to implement placable traps. Currently the engine not allows you to add spatial scripts dynamically (only through the mapper), so you need to use "list_begin, list_next" functions to iterate over all critters and check if someone should trigger the trap. How do you store the trap? You can use sfall globals, but since one global is just one value, you will have to limit maximum number of active traps to something like 4 or 5 and your script will have the same code repeated 5 times (repeated code is always HORRIBLE, as it leads to great many bugs and difficult to modify).
Arrays saves the day, as you can place as many of them (information about each trap - tile, elevation, status, trap type, etc.) in the array as you want, and easily save it to the savegame.
When you need to check if trap is triggered, you just iterate over all traps using "foreach" or "while" loop.

Benefits: Unlimited number of traps, less code (easy to modify).

Having arrays that work in expected way and safe to use, you will surely find many applications for them in your daily scripting (store NPCs PIDs to iterate over party members with ease, store object pointers with additional information to implement some new game mechanics, etc.).

What I also want to have is an associative arrays. Normal arrays have specific length (number of values) and to access/change those values you use numeric indexes starting from 0. Those are called "lists" or "vectors". This behavior is good in many applications.
But, let's say you need to make it so some item PIDs may have some information associated with them (like price modifiers). Example:

Code:

; Klamath: Gecko Pelts cost much more, stimpacks less, leather armor less, scorpion tails more (illogical, but balanced)
2=276*2.5|277*2.5|623*2.0|40*0.7|3*0.8|74*0.8|92*2.0|591*2.0

In this case I specify that some items should cost less/more in Klamath. So how do I use this string? Associative arrays come in handy here:

Code:

pidList := some_parse_func(get_ini_string(...)); // this parses the string using string_split and creates array with 8 key => value pairs, where each listed item PID is associated with float price modifier

...

// modify prices like this (simplified pseude-code):
foreach ( pid in all_item_PIDs) begin
   mod := get_assoc_array(pidList, pid);
   if (mod) then
      set_proto_data(pid, PRODATA_IT_COST, cost * mod);
end

I actually have function called "get_assoc_array", but what I want is to be able to use square brackets for this:

Code:

mod := pidList[pid];

Which is currently not possible, because the compiler will simple translate this to:

Code:

mod := get_array(pidList, pid);

And get_array only works with normal arrays (lists), in which we have indexes 0, 1, 2 .... (and not 276, 277, ...).

I hope this is enough to understand that they are must-have in every script language

 

Howdy Smoothskins,

Right, I have a problem with recruiting Marcus, the option to tell him ive found the bodies does not show up. after re-loading and trying the quest a couple ways.. no joy. So, i figured why not change the script for marcus to manually add him to my party (without 'completing' this quest) I'm close to trying doing it myself but if I mess up and have to start again.. i just wont.

so if anyone knows how to, or even possibly post a a file replacing Marcus' script with one where he's in my party, that wound be super super cool.

p.s Ive found the bodies, got the note , talked to Zauis, talked to Dan (whom i cant show the note to, tried even placing it on his person..) talked to francis and made him leave (thus making the note disappear,probably not important) but still cant tell Marcus what ive discovered. Running multiple mods this is my own fault.

also apologies if this should be on another board.

thanks.

 

After many work, finally managed to implement this. Not perfect though (after explosion light is gone, some of it remains... you can think of this as a grass on fire ). Will post code later with all other additions.

 

Sorry, I went and got married, so my time here has dropped from the odd evening down to zero for the foreseeable future. If you want svn access, pm me your sourceforge name and I'll hook you up.

On a completely unrelated note, more weddings should involve ball pits.

 

Congratulation Timeslip, I hope we don't lose you completely...I didn't realise ball pits and weddings were compatible. Thinking about it, why not...

 

Is there an INI option to change when each dream occurs? I couldn't find it.
Because RP adds a lot of new content to explore, it would be wise to move those dreams a little (suggestion by lujo).

 

No option for it AFAIK. Maybe they're triggered by hitting certain numbers of game ticks that equal to those specific dates?

 

@ arrays

Just curious...

- there's a X amount of "special" doors in a game,

- with default commands/functions/whatever I'd have to create GVAR/MVAR/LVAR per door to make NPC remember that that something happened around the door:

Code:

if (condition) then begin
    // remember that something happend around this particular door
    set_map_var(MVAR_DOOR_X, 1);
    set_local_var(LVAR_TEMP_DOOR_POINTER, 0);
end

and later read those saved variables like this:

Code:

// check is something happened
if (map_var(MVAR_DOOR_X) == 1) or (map_var(MVAR_DOOR_Y) == 1) or (map_var(MVAR_DOOR_Z) == 1) then begin
    do_some_shit;
end

(I'd have to check each door LVAR/MVAR/GVAR! )

- now, can you make something like this with sfall's array:

Code:

// save object to array
if (condition) then begin
    save_to_array(MY_AWESOME_ARRAY, local_var(LVAR_TEMP_DOOR_POINTER));
    set_local_var(LVAR_TEMP_DOOR_POINTER, 0);
end

// check if object is in array
if (is_object_in(MY_AWESOME_ARRAY)) then begin
    do_some_shit;
end

no pre-defined .inis, and shit - save/read pointers on the fly, just like in case other variables.



---------------------EDIT

Of course, ignore "syntax", or lack of any definition of my array example it just to show what do I mean - I predict I'd have to define max length of array + do some other shits in the first place to make it working. At least this how it was looking with one-dimensional array of objects I've been playing around in some other game. But this was pre-defined, read-only type of thingy.

 

Yeah, this is exactly the thing you can do with arrays. Need to think on how to make them more user-friendly though. This is exactly what I'm going to do: change how arrays work, add some new functions for them, maybe adjust compiler a bit and then do something with backward compatibility...
I would've added INI parameter to switch between new/old arrays behavior but it may be excess. For example I think the second parameter to "create_array" should not exist at all (how many bytes to allocate in memory for each element... not something that regular scripter should worry about).

PS: I don't know if someone already shared something like this or not. While I'm studying both Mapper2.exe and Fallout2.exe in IDA Pro, I'm filling the database with properly named functions, variables, some structures. If anyone wants it, I could share it.

 

Ok, so I've made a few commits. This is a part of the things I want to do apart from array changes. Here's what's been done:

- splitted ScriptExtender.cpp into several .hpp files with all asm/c code related to new operators;
- added several new operators related to strings: 'substr' (get part of string), 'strlen' (length of a string), 'sprintf' (format any variable using format string);
- added 'typeof' operator: returns type of value (1 - integer, 2 - float, 3 - string);
- modified 'atoi' operator: now it will parse hex&octal strings properly into integers;
- Hakunin movie timers are now configurable through INI;
- added INI options to change explosives damage (dynamite and plastic explosive);
- added ComputeSpray mod which allows you to tweak bullet distribution in bursts;
- added ExplosionLighting mod: explosions and projectiles emit light (currently bugged, only enable for testing);

Movie timers and explosive damage are both quick dirty hacks. But it allows to mod those thing for Fallout 2. It may be replaced with separate ini files, etc, in future.

If anyone wants to test new scripting functions, I can post updated compile.exe & ddraw.dll.
In particular, I'm thinking how to better name new utility functions (like strlen). They all named by their C language counterparts (like Timeslip did with atoi, atof, etc.). But maybe they're not very readable for people who aren't familiar with C? They are short and distinctive though.

PS: I hope you guys won't kill me for code refactoring. Several thousand lines of code in single file was too much... It should be a bit more organized now.

 

@ Everyone, NovaRain is now the official sfall file releaser. I won't be doing the sourceforge releases any more.

@ Phobos, might as well move the PM conversation over here. Things like source control and array changes affect everyone, not just me.

The byte parameter was just a quick and dirty hack to let people store strings in the arrays. If you have a better solution for storing strings, go for it. But I think there is a real use case for arrays saved into save games, and that functionality should be preserved, whether as a separate opcode or a parameter to create_array.

I have no particular objection to changing the semantics of create_array, as long as binary script compatibility is preserved. (In sslc, change the create_array token to create_array_depricated or similar, and introduct a new create_array token with a new opcode.) I would prefer source compatibility to be maintained though, unless you have a very good reason not to.

Someone who actually uses arrays in mods should probably weigh in here.

Short answer is I don't know. I don't believe we have permission to put either in the repository.

On the personal side, I've never used git before and have none of the tools installed, so I'd rather not have to learn it in order to work on sfall in the future.

On the non-personal side, the differences between source control systems matter very very little for small projects like sfall where there's no contention over files. What concrete advantages would git have over svn assuming no merges, etc are required?

The only reason to switch I can see is if there are other more active contributors in the opposite situation to me. e.g. Glovz sends me patches and I commit them, but if git let him commit himself then that would be a pretty good advantage.

I've thought about doings that a few times. Never quite got around to it though; everything I tried I wasn't quite happy with.

All of sfall is one big dirty hack.

 

Of course, possibility to store arrays in savegames will be preserved. However, it shouldn't be default (and forced) behavior. To store array in save, I'm thinking of function like this:

Code:

arr := [1,6,85,"Cool string", 0.3343];
save_array("my_cool_array", arr);

To load it, you will have to call "load_array". This will basically be the same as (and consistent with) "set/get_sfall_global" functions, but for arrays. And it will be safer, since you are using pre-defined array ID.
Other option might be to turn array into "permanent mode" with some function. But that way it won't be clear what arrays are permanent and what are not. So I would stick with "save/load_array".

So here's a sum of what I'm trying to achieve:
1) Arrays won't be saved automatically with auto-generated IDs.
2) You won't have to specify number of bytes for array creation (second argument to create_array will probably be ignored).
3) Easy to use associated arrays with keys of any type.
4) I think set/get_sfall_global should accept any value as key, not just 8-byte string (it will basically be one "main" associated array).

I wonder why we have different functions get_sfall_global_int and get_sfall_global_float ? AFAIK, SSL operator can have returned value of any type.

Well I asked Nirran, he's okay with putting int2ssl out (though he's not an original author). As I said before, having all 3 in 1 repo would be of great benefit. I will wait for your permission on this (since there are no license file with sslc and original devs are most likely hard to find/contact with).

Well basically git (or mercurial) are better for projects with several active developers (branching, merging, that sort of stuff is a lot clearer). Also it allows you to commit several times locally before pushing to remote server.

I guess we stick with SVN for now, unless there will be more active developers.

Edit:
Progress on arrays is slow. Would've done by now if not for backward compatibility issues
After playing with sslc code, I've come up with a relatively simple solution how to implement array expressions!

So for example, instead of this:

Code:

if (weaponPid == PID_KNIFE or weaponPid== PID_COMBAT_KNIFE or weaponPid== PID_LITTLE_JESUS)

you could do this:

Code:

if (in_array(weaponPid, [PID_KNIFE, PID_COMBAT_KNIFE, PID_LITTLE_JESUS]))

Or if you want to initialize some array:

Code:

myArray := [123, 0.232, "Good stuff"];

"Proper" solution for arrays would be to adjust script interpreter itself to account for array type, create some opcode with variable number of arguments.. But as a quick hack, we can tell compiler to translate our array expression in something like this:

Code:

myArray := (arraystack_new+arraystack(123)+arraystack(0.232)+arraystack("Good stuff"));

This is still valid SSL code. Internally, arraystack_new will create temp array and return it's ID (just like temp_array), and arraystack(x) pushes x to the end of "current" array and returns 0. In the end we get ID of fully prepared array with just one expression