J.E. Sawyer on FO3/NV save game-related slowdowns

Discussion in 'NMA News and Information' started by WorstUsernameEver, Dec 5, 2011.

  1. WorstUsernameEver

    WorstUsernameEver But best title ever!

    May 28, 2010
    Should you have played a game from Bethesda this generation you'll probably be familiar with slowdowns related to the increasing size of save-game files, a problem which wasn't fixed even in the recently published Skyrim. Since the problem was present in New Vegas too, project director J.E. Sawyer from Obsidian took some time to answer to some questions about it on his Formspring account, and explained why the bug couldn't be fixed in a patch:<blockquote>Question from one of lesser knowledge... Is there a save game size that you would call "normal" for F:NV? Mine is 14mb, and i'm having intense lag in multiple sections of the Mojave/ all DLC lands. Just curious if that is/could be an issue!

    That can easily be a big problem, especially if you're on the PS3. The longer you play a character, the more bit differences on objects (characters, pencils on tables, containers, etc.) get saved off and carried around in memory. I think we've seen save games that are pushing 19 megs, which can be really crippling in some areas.

    ...

    If it's an issue why hasn't it been resolved? The pace at which game sellers basically dish an alpha or beta game to the marketplace is sickening. New V stopped being "beta" in mid-2011. I know as a dev my workplace wasn't AAA dishing but we had integrity.

    Since you're a developer, you should understand the implication of what I wrote. It's an engine-level issue with how the save game data is stored off as bit flag differences compared to the placed instances in the main .esm + DLC .esms. As the game modifies any placed instance of an object, those changes are stored off into what is essentially another .esm. When you load the save game, you're loading all of those differences into resident memory.

    It's not like someone wrote a function and put a decimal point in the wrong place or declared something as a float when it should have been an int. We're talking about how the engine fundamentally saves off and references data at run time. Restructuring how that works would require a large time commitment. Obsidian also only had that engine for a total of 18 months prior to F:NV being released, which is a relatively short time to understand all of the details of how the technology works.

    ...

    Hey Josh, different user, but same kind of 'lesser knowledge' bloke. What's a bit difference and how does it determine a save file?

    http://en.wikipedia.org/wiki/Bit_field

    It's a compact way to store data. The bit differences in this case are just flags set up to mark what data has changed (i.e., are different) from what's in the core .esm.

    Let's say that I, as a designer, set up a creature in an area. I set all of the character's statistics and gear and save it in the master FalloutNV.esm file that gets loaded into the game. You, the player, run through the area and shoot that dude. You loot him of his gear and put a shovel in his inventory because you are wacky.

    The game needs a way to mark that his a) position B) health c) inventory d) some other stuff has changed on him. It does that by marking what fields have changed (by setting individual bits) and then indexing the individual (changed) values for reference later.

    When you load the save game, it loads up all of the bit fields marking changes in your save game. When the individual objects load, it applies the indexed changes to those objects. That way, when you come back to the area you left two nights ago, the character is still sprawled out where you left him, naked, with a shovel in his inventory.

    Individual bits of data are tiny, but there are thousands upon thousands of objects in F:NV, each one containing numerous data fields that could potentially be changed in your save game. Over time, it adds up.

    ...

    Is the inflating save file just an issue for the PS3 (I've seen lots of lag/crash complaints from PS3 users) or does it happen on all platforms? I'm just wondering if other platforms handle it better than the PS3.

    As with Fallout 3 and Skyrim, the problems are most pronounced on the PS3 because the PS3 has a divided memory pool.

    ...

    (Same fellow you just answered) So....basically, every time I manipulate an object, it fluctuates the save file up or down? I've noticed files can get rather huge, is there no way this could have been greatly diminished in a game as big as New Vegas?

    It almost always goes up. Some areas will reset contents after three (game) days, but a lot of stuff lingers. Additionally, we also have to deal with "persistent references". These are objects that are immediately loaded with the game because we need to be able to reference them anywhere/everywhere in the world -- even if the player is nowhere near the object. Characters are the most common example. All of the companions need to be able to move around the world even when they are not in your current area, so they are all persistent references.

    All object data (excluding art assets like .nifs and audio assets [VO]) for persistent references is loaded at all times, so that's more-or-less a permanent chunk of resident memory. The number of persistent references invariably goes up with each DLC, so as the number of DLCs increases, the system has less and less memory available. Of course, the player's save game file only gets bigger and bigger, since he or she is going through more or more areas manipulating an increasingly large number of objects.

    This is why some of our later patches actually removed content from the core game (e.g. Primm). Even though we had balanced the memory footprint for the core game, DLC content was pushing down the available resources.

    ...

    "divided memory pool"?

    The Xbox 360 has a unified memory pool: 512 megs of RAM usable as system memory or graphics memory. The PS3 has a divided memory pool: 256 megs for system, 256 for graphics. It's the same total amount of memory, but not as flexible for a developer to make use of.

    ...

    But why did the patch remove content from the PC? Most PCs nowadays have 2-4GB of RAM, plus 500MB-1GB of GPU memory. So I doubt the DLCs negatively impacted most PCs to justify content removal...

    If we had generated .esms per-platform, that would have been a crazy nightmare for a lot of reasons. A slightly less risky approach could have been to script the removal of assets using the IsPC/IsXbox/IsPS3 functions, but that also introduces its own host of potential problems, especially if objects are attempting to reference something as the script removes it.

    We ran into a small but non-trivial number of crashes in F:NV involving persistent references attempting to interact with an object as the player transitions out of his or her current area. E.g. Chief Hanlon attempts to sit in a chair. The player leaves the area, the chair Hanlon wants to sit in is unloaded, and the game crashes.

    ...

    Hypothetically, if I were to play through FNV without opening any non-necessary containers (e.g. never opening an ammo case) or touching any non-necessary objects (e.g. never moving a pencil) would the game be more stable?

    By some margin, yes, but randomized loot I believe is generated on the area's first load, which also applies to the equipment in many characters' (e.g. Fiends') inventories.</blockquote>Thanks GameBanshee.
     
  2. 4too

    4too Vault Senior Citizen

    Apr 30, 2003
    Techno Cul De Sac Retorical Amusment

    Techno Cul De Sac Rhetorical Amusement



    What other games are hitting this memory allocation ceiling?

    Will the short term answer mean less interactive STUFF, (less environmental targets, less EXPLODING HEADS), could this start an interactive Stuff techno war between AAA title pushers?
    Could collecting pencils or exploding heads became console achievements?
    How many collected rocks in the FO's? ;)

    So consoles don't have the TECHNOLOGY for grabbing all the present day video game FAT LOOT without maxing out the hardware / software?

    Could this trauma of save game size truncating the cornacopia of FAT LOOT, and or crash, resonate into the next marketing frenzy for console hardware?

    Imagine, PC players fingering the console half gig of ram choke point as " ' this is why we can't have nice things! ' " :(

    Which console purveyor will be the first to close the FAT LOOT lack gap? ;)




    4too
     
  3. UncannyGarlic

    UncannyGarlic Sonny, I Watched the Vault Bein' Built!

    Feb 6, 2008
    It looks like filling your game with junk not only negatively impacts gameplay but the stability and efficiency of the game. If the item doesn't have any purpose then don't allow the player to interact with it.
     
  4. gumbarrel

    gumbarrel It Wandered In From the Wastes

    159
    Oct 18, 2010
    Tell that shit to the Deus Ex fans.
     
  5. Alphadrop

    Alphadrop A right proper chap.

    Aug 21, 2008
    Deus Ex can get away with it though as it has a smaller amount of areas and less loot.

    Rage is a modern example where pretty much nothing apart from actors and vehicles moves and those are loaded with the area and scripts rather than kept as a reference (seen by how everything respawns when you change area) which is why the game runs so smooth on consoles even if it was criticised that you couldn't knock over a can.

    256 (well sort of) is a horribly low amount of ram to try something as big as a persistant freeroam game on, no wonder the ps3 versions always have it bad.
     
  6. 4too

    4too Vault Senior Citizen

    Apr 30, 2003
    Pardon My Mime

    Pardon My Mime








    The prevailing business model for marketing to the consumer culture requires ...




    :drunk: :drunk: :drunk:



    4too
     
  7. Richwizard

    Richwizard So Old I'm Losing Radiation Signs
    Orderite

    Jan 18, 2010
    Sawyer's answers make sense. I have 6GB of RAM in my PC, but my video card only has 768mb. I do have an occasional graphics glitch because of this and the fact that my graphics settings are maxed out. As the game progresses and the save files get bigger, it can get worse. The simple solution to this problem is to have enough memory to process the files efficiently. On a console, this is more of a problem.

    @Sabirah - Don't feed the trolls. :P
     
  8. NVExpert

    NVExpert First time out of the vault

    10
    Dec 6, 2011
    let us get this straighted out one more time.

    the game that ships with the vanilla esps/esms will have fixed positions you want ot keep for when the user wants to start a new game.

    but for the save file we are still looking at a 3d vector, 3 floats for every object static or animated static or kick around /drag debris.

    having less items that can be moved around does mean less needs ot technically be saved. but that isn't a good excuse for why the game would be slow as you are playing it. you save the most current position regardless of where it is in its animation. only things acted upon in masses causes potential slow downs in RT.

    the game is not littered enough to cause size issues in the case of load/save or speed issues in the case of just walking around and shit.
     
  9. Lexx

    Lexx Background Radiant
    Moderator Modder

    Apr 24, 2005
    Well, Sawyer never said that it's the only problem, hm? I don't get why you are jumping so up and down on it. Can't be changed now anyway.
     
  10. DemonNick

    DemonNick It Wandered In From the Wastes

    141
    Feb 17, 2010
    Supposedly the engine's really retarded about how it handles memory in general though, hence the need for the LAA Executables.

    Also IIRC the engine is missing some pretty basic features that were finally added in Skyrim (occlusion culling, for instance) which puts way more pressure on the game. I might be wrong about this. But I'm pretty sure all the LOD Terrain is loaded all the time.
     
  11. TychoXI

    TychoXI Still Mildly Glowing

    246
    Oct 20, 2008
    Well the original Fallouts had savegames that would get huge (humongous!) I remember loading times of an eternity...

    Of course, as old as the engine was at least it didn't move objects around (sometimes actually flying around) every step you took through the gameworld, and save their new positions.
     
  12. Brother None

    Brother None This ghoul has seen it all
    Staff Member Admin Orderite

    Apr 3, 2003
    I used to jump on top of tables and run around knocking tableware around, thinking it funny. Now I realize I was breaking the game. I'm so sorry, game!
     
  13. TheWesDude

    TheWesDude Sonny, I Watched the Vault Bein' Built!

    Feb 25, 2005
    and finally the truth comes out.


    beth really sucks at making games and never "pushed the envelope" of technology.


    their engine really limited what they could do. not to mention that consoles are the biggest problem.


    and seriously sony... 256 megs of ram???? wtf is up with that.
     
  14. Stanislao Moulinsky

    Stanislao Moulinsky Vault Fossil

    Jul 16, 2009
    Low RAM and shitty costant streaming from the HD from what I gathered. Apparently running the game with a SSD reduces dramatically stuttering.
     
  15. valcik

    valcik So Old I'm Losing Radiation Signs

    Dec 20, 2008
    Yup. Mr. Howard has deserved a Master Recyclator achievement.
     
  16. Black

    Black Vault Senior Citizen

    Jun 21, 2007
    How amusing, doesn't/didn't skyrim have the same problem on ps3, too?
    Weird, considering that it's a brand new engine!
     
  17. WorstUsernameEver

    WorstUsernameEver But best title ever!

    May 28, 2010
    Pete Hines denied that the Skyrim issue is the same on Twitter:
    Not flattering that they basically have a similar issue in their all-new engine though (and I'm giving him the benefit of doubt and assuming he's saying the truth and that in Skyrim the issue is a different one and not a legacy of Morrowind/Oblivion/Fallout 3/Fallout: New Vegas).
     
  18. Nark

    Nark Sonny, I Watched the Vault Bein' Built!

    Dec 6, 2008
    I wouldn't be surprised if it turned out he was lying, knowing Bethesda's reputation with telling the truth. The problem sounds very similar to the one people are getting in Skyrim.
     
  19. Walpknut

    Walpknut This ghoul has seen it all

    Dec 30, 2010
    So it seems getting one of those "Clean up the wasteland" mods on your game is a very good idea, all that junk not only is it useless it apparently also hurts perfomance.
     
  20. Alphadrop

    Alphadrop A right proper chap.

    Aug 21, 2008
    Well apart from the problems mostly being on consoles, PC's have enough RAM to get away with it.