User:QQuix/On Dynamic Items and Savegame Bloating

Foreword

This is the result of some tests I am doing to determine/review the behavior of Dynamic and Non-Dynamic items.

For definitions, read Modding Terminology, particularly the topics “Dynamic Content” and “Dynamic Items”.

Most of what is here is not new, and is written one way or another across the CS WIKI.

Feel free to add comments/suggestions either here or to the Talk page. I suppose any specifics would fit nicely under the text you are commenting on. General comments/suggestions may fit better in the Talk page. Either way, all are welcome.

Be aware that this is a Work In Progress and the contents may change as a result of additional tests and contributions.

Introduction

The main objective of the tests was to get a firsthand, comprehensive experience with dynamic items, motivated by the above mentioned article and the PlaceAtMe bloat potential. And, while doing it, keep an eye for fine details, exceptions and odd results that might show up.

The tests were focused on dynamic items and savegame bloating.

NOTE: Most probably these kinds of tests have been done by other modders. Any info on equivalent tests would be appreciated.

To do

• Scripted items
• DuplicateAllItems
• Inter-container item movement

Conclusions

(I am placing the conclusions up front for the benefit of those that don’t care to read the whole text)

Dynamic items

• When a dynamic item is moved to a container, a Ref Variable containing its FormID is not invalidated immediately. It remains intact for the rest of the frame. It turns to Null in subsequent frames.

TODO: check if it turns to Null right in the next frame.

There will be a new function in v16, GetCurrentFrameID, that will make this test easier and more accurate.

--Haama 11:05, 17 September 2008 (EDT)

• Functions using the null Ref Variable either are ignored or produce undesirable results. Didn’t experience any CDT with the few functions tested.

TODO: Test more functions. Agree. These are separate tests.

It might be better to think of these as separate tests - if the reference variable is 0, then it's 0, and it doesn't matter if it hasn't been initialized yet or if it just turned 0.

If the reference remains non-0, however, that's a different story. In general, OBSE functions are very good about ignoring bad references while vanilla functions crash and burn.

--Haama 11:05, 17 September 2008 (EDT)

• A free FormID is reused only if it is after/higher than the last used one. Free FormIDs in the middle of the series are not reused. Example: Drop 3 items. Lets say they receive FormIDs 11,12 and 13. If you pick #13 up and drop it again, it will be 13 again, one higher than the last used (which is now #12). If you pick #12 up and drop it again, it will be FormID 14, one higher than the last used (#13).

Non-Dynamic items

• Items placed in the CS also add to the savegame size once, when they are moved from its original position for the first time. Although not obvious, this makes sense: while the item stays in its original position, their data is loaded from the esp/esm. Once moved, the game must start keeping track of its position and saves the data in the savegame. The increase is around 70 bytes per item.

Would SetAtStart get rid of this bloat? Not really worth it, but there are some very picky people out there.

--Haama 11:07, 17 September 2008 (EDT)

SetAtStart does not seem to work (as its page says). I would guess that the original idea was what you mean: not saving any data on the item, at the next load it would stay in its start location. But to move it immediately, I wonder if the game keeps its start location in memory. It does.

QQuix 21:27, 17 September 2008 (EDT)

Bloating tests

(I am using the term ‘Bloating’ in its negative sense: when the savegame size increases due to apparently useless data)

• Once placed in the world, the effect on bloating is the same, regardless of how the item was created. Meaning: dropping 1000 individual quills from a container has the same effect as 1000 individual quills created with PlaceAtMe.

I assume you mean 1000 stacks of 1 quill?

--Haama 11:23, 17 September 2008 (EDT)

Yes. Text adjusted. Feel free to rephrase it if you like. QQuix 21:27, 17 September 2008 (EDT)

• Bloating is the same for different items (tests used Flawless Emeralds, Quills and Paintbrushes).

Note that all of those items are misc. items. (Probably won't matter, though)

--Haama 11:23, 17 September 2008 (EDT)

• Each item placed in the world added ~78 bytes to the savegame.

There are a few more variables you could try - the owner of the items (PlaceAtMe leaves it at 0), the script variables, etc. Generally this is refered to as extra data

--Haama 11:23, 17 September 2008 (EDT)

Good point. Do you know if there is any documentation on these fields. UESP has a lot, but not at this level of detail. QQuix 21:27, 17 September 2008 (EDT)

• Each item moved to a container reduced savegame size by ~23 bytes immediately (before cell reset).

• Waiting outside for cell reset, additionally reduced savegame size, but this reduction does not seem to be related to cleaning up leftovers from the test scenario: is was always about the same (~20K), no matter how many items were used

Are you using the original Oblivion.esm? It might be better to start with a blank .esm file, if you can find/make one. I remember that there are a few particular things needed in the .esm, but I don't remember which more than that.

Otherwise, you might be able to narrow it down by either waiting for another cell reset, or waiting for the first cell reset and then testing the save file.

--Haama 11:23, 17 September 2008 (EDT)

Never heard of a blank esm. Is it possible to load the game without the Oblivion.esm? QQuix 21:27, 17 September 2008 (EDT)

• Bloating also occur if a dynamic item is repeatedly added/dropped to/from a container. Meaning: a single item added/dropped to/from a container a thousand times has the same bloating effect as 1000 items doing the same once.

NOTE: Will the remaining bytes stay forever in the savegame?

• RemoveAllItems has no effect in savegame size when removing items placed in the container in the CS.

• RemoveAllItems reduces ~13 bytes per ItemType when removing items added dynamically.

• RemoveAllItems HAS NO EFECT in savegame size. (Surprise! Surprise!)

Umm... that is surprising. If an actor has 100s of items they would need a linked list of each item's reference and quantity. Each reference is at least 32-bit, IIRC, so... yeah, that would be surprising...

My only guess - the linked list never really goes away. So, if you add an Apple to an NPC, use RemoveAllItems, then when you add a second Apple it won't cause any more bloat. (BTW, annoying as all *#!@%)

--Haama 11:23, 17 September 2008 (EDT)

You are right. It seems the linked list entry is created when a new ItemType is added to a container and is removed by RemoveAllItems. So, the net result is zero. Meaning: adding and removing items to containers don’t cause bloating. QQuix 21:27, 17 September 2008 (EDT)

TODO: Re-test RemoveItem

TODO: Test with RemoveMe.

• AddItem increases in savegame size by ~13 bytes per ItemType. These bytes are removed when the ItemType is removed from the container.

See above, I believe you're adding the same item. If you add a different item the savegame size will change. Where else would the information be stored?

--Haama 11:23, 17 September 2008 (EDT)

You are right in both cases. As I was testing with 1000 x same item, the difference was so small to relate it to the functions. Further tests with 222 different items showed a better figure. Text fixed. Conclusion added.

QQuix 21:27, 17 September 2008 (EDT)

• If items are picked up by the Player Character, the savegame size returns to its original size at cell reset.

• Adding and removing items to/from containers don’t cause bloating.

• Adding and removing items to/from the game world DO cause bloating. Considering previous conclusions, the current results of the tests suggests that the bloating is caused by the Activate function used to transfer items to containers. It looks like this function is not flagging the items correctly, so the cell reset routines is failing to eliminate some unnecessary information.

Observations and Comments

1) All byte increases/decreases mentioned above are averages that may be affected by causes other then the test scenario. Therefore, consider them as good approximations rather than exact values.

2) The leftover bytes may be related to the following info on the Modding Terminology page:

However, under some circumstances the reference will not be removed from the save game. (Non-removal seems to be associated with having script record variables either attached to the reference and/or pointing at the reference.)

There is no way to do massive tests with thousands of items without “having script record variables pointing at the reference”. Unless, of course, having the PC pick up all those items one by one.

Therefore the leftover bytes may be related to this effect.

3) PlaceAtMe presented two different behaviors during the tests:

• Sometimes it behaved as mentioned above: added ~78 bytes when used and reduced ~23 when moved to a container
• Other times it added ~147 bytes when placed, reduced ~23 when moved to a container and reduced an additional ~70 bytes at cell reset

Although different, the net result was about the same.

4) Exit+Restart+Load+Save didn’t reduce savegame size. Since I have not tested this extensively, it is mentioned as an observation, rather than a conclusion.

5) Some observations about havok (gravity):

• In a few runs, 1000 items were dropped in a single frame loop. At the end of the frame, all were in the same spot in mid air. The next frame some items start falling, but it takes ~100 frames to the last one start falling. Eg. If the item is a Ruby, it forms a cascade of rubies pouring for 10-15 seconds from their original position.
• In the same scenario, if the items were spread in the air, still in the first frame (drop all and GetNextRef+SetPos), they don’t fall at the same time. Instead their fall forms like a wave, as if only a few items started falling on each subsequent frame.
• Then, if all the items now in the floor are moved to a certain height in a single frame, the next frame all of them start falling at the same time.

I don’t know if there are any conclusions to take from the fact that items placed in the air after being dropped don’t fall all at once as items already present in the world.

Another observation from the same scenario is that droping an item adds ~48 bytes to the savegame immediately (same frame). The remaining ~30 bytes are added on following frames, during/after their fall.

Common knowledge / Backgound

Non-Dynamic items

• Non-Dynamic items are items placed in the world in the CS.
• Formid starts with Mod# (NNxxxxxx).
• FormIDs are stable after adding to and dropping from a container, therefore FormIDs saved in Ref variables may be reused over time.
• At game load, the engine loads data from esps/esms first. Then, it loads data from the savegame, updating the data in memory with whatever has been changed during the course of play.

Dynamic items

• Dynamic items are items placed in the world during the course of the gameplay.
• Dynamic items may be created by dropping an item from a container (either placed in the container in the CS or added on the fly by AddItem) or may be created directly in the world by PlaceAtMe.
• Formid starts with FF (FFxxxxxx).
• FormIDs are dynamic and may change after adding to and dropping from a container, therefore FormIDs saved in Ref variables should not be reused over time.
• The ‘holes’ left in the FormID sequence will be reused when the count reaches FFFFFFFF. UESPWiki Tes4Mod:Formid Advanced Questions

General Methodology

The tests were done with Misc Items only.

FormID tests

Create an item, place it in the world, locate and save its FormID, remove the item to a container and play with the saved formID to see what happens.

Bloating tests

Create 2000 items, place them in the world, remove them to a container and destroy them, saving the game along the way to measure bloating.

Variations

Item Creation

For the test, items were created and placed in the world four different ways:

• Placed in the world in the CS.
• Placed in a container in CS and dropped in the world.
• Dynamically created in a container with AddItem and droped in the world.
• Dynamically created in the world with PlaceAtMe.

Item Destruction

Items were removed from the game 3 ways

• RemoveAllItems
• RemoveItem
• RemoveMe (not done yet)

Other variations

• Scripted (not done yet) and non-scripted items.

Test Results

Invalid FormIDs

By Invalid FormIDs, I mean a dynamic formID after the item is moved to a container

• Item added to an NPC in CS.
• Dropped in game (gets FormID=FF000001).
• FormID saved in a Ref variable >> Set xItem to [item found by GetFirst/NextRef loop].
• At this point xItem contains FormID FF000001 (valid).
• Added to an NPC and dropped again (gets FormID=FF000002).
• At this point xItem still contains FormID FF000001 (now invalid: does not represent an existing item).

Using a reference to the now inexistent item, IN THE SAME FRAME, does not cause any visible problem.

• Normal, correct returns from PrintToConsole, GetBaseObject, GetPos, SetPos, MoveToMarker:

• PrintToConsole “%i %n” xItem xItem - prints FF000001 and the correct item name.
• xItem.GetBaseObject – returns the correct Base Object.
• SetPos – Sets new XYZ positions as verified by subsequent GetPos.
• MoveToMarker – Sets new XYZ positions as verified by subsequent GetPos.

Using a reference to the now inexistent item, in following frames:

• xItem becomes a Null reference.
• PrintToConsole “%i %n” xItem xItem - prints 0 NULL.
• xItem.GetBaseObject – returns Null.
• xItem.SetPos – Sets new XYZ positions to the object where the script is running (as xItem==Null, assumes Self??).
• xItem.MoveToMarker – does not seem to do anything.

Conclusions

It seems that after an item is removed from the world as described above, all its data remains intact and can be manipulated by scripts in the same frame they are removed.

Using a reference to the now inexistent item, in following frames:

• The functions tested did not crash the game.
• Most of the time, it seems the function does nothing.
• But sometimes, have undesired results (as in the SetPos).

Bloating

Test scenario

All runs started from the same clean save.

The tables show the variation (in bytes) in savegame size after each step of the tests.

Tests where done in an additional Divine Elegance basement. The “Going to exterior” step means leaving the basement, crossing DE cell and activating the exit door. Saves where made immediately after the exterior cell was loaded.

All bytes increases/decreases mentioned in the Conclusions were calculated dividing the difference in savegame size by the number of items used in that run (typically 1000). Since savegame size is affected by other causes, the values collected may not be exclusively due to the test environment, but, probably, are good approximations.

RemoveAllItems Tests

The column “With RemoveAllItems” shows the results of the test as initially intended.

Since the RemoveAllItems didn’t seem to reduce savegame, I re-ran all tests without using it. The results are in the last column

	With	Without
	RemoveAllItems	REmoveAllItems
After adding 1000 items	+78.373
After adding 2000 items	+78.416
After moving 1000 items to container	-22.670	-22.959
After moving 2000 items to container	-23.014	-23.014
After RemoveAllItems	-18
After going to exterior	+13.801	+14.316
After Waiting 4 days	-21.218	-24.561
After returning to cel	-5.265	-2.968
Net result	+98.405	+97.604

Tests were run 6 times, two times for each of the creation method mentioned earlier.

Individual runs did’t vary much from each other, so they are not included here.

The initial size of the savegame is 2.983.546 bytes .

For each of the three types of dynamic items considered, the test consisted of:

• Placing 1.000 items in the world (either by dropping from a container or by PlaceAtMe, one item per frame)
• Placing 1.000 more items in the world (same)
• Moving 1.000 items to an NPC (Single frame GetNextRef loop to identify the item and then NPC.Activate Item)
• Moving 1.000 items to an NPC (same frame as above)
• Using RemoveAllItems on the NPC
• Going to an exterior cell
• Waiting 4 days
• Returning to the test cell
• Saving the game at the beginning and after each of the steps above

NOTE: 2000 items seems to be the limit for my hardware to handle (down to about 3-5 FPS)

Multiple Drop+PickUp Tests

1.000 items were created four different ways (see below)

Then they were added/dropped to/from an NPC a number of times.

The game was saved after each operation using the con_Save function

The column acronyms mean:

• CS-W – Items created in the CS – placed in the cell
• CS-C – Items created in the CS – placed in a container
• AI – Items created ingame by AddItem function
• PAM – Items created ingame by PlaceAtMe function

	CS-W	CS-C	AI	PAM
	(Non-dynamic)	(Dynamic)	(Dynamic)	(Dynamic)
After moving	+70.518	---	---	---
After PickUp	+513	---	---	---
After AddItem	---	---	+601	---
After PlaceAtMe	---	---	---	+147.429
After Droping (1)	+19.003	+77.770	+79.060	---
After PickUp (1)	-9.022	-22.902	-22.878	-21.862
After Droping (2)	+9.991	+78.906	+79.910	+55.368
After PickUp (2)	-9.968	-22.947	-22.942	-22.929
After Droping (3)	+10.299	+80.040	+78.952	+79.101
After PickUp (3)	-9.977	-22.958	-22.937	-22.934
After Droping (4)	+9.925	+78.948	+79.099	+78.939
After PickUp (4)	-9.968	-22.953	-22.942	-22.934
After Droping (5)	+10.220	+68.855	+78.947	+79.270
After PickUp (5)	-9.975	-22.966	-22.942	-22.934
After going to exterior	+14.960	+12.741	+13.119	+14.980
After Waiting 4 days	-21.676	-19.482	-19.227	-68.350
After returning to cell	-5.918	-5.137	-4.707	-4.196
Net result	68.925	+257.915	+271.113	+268.948

Each column shows the result of just one run.

As can be noticed, non-dynamic items caused an initial increase in savegame size (as explained in the Conclusions section) and, after that, they didn’t affect it much.

Dynamic items, on the other hand, consistently increased savegame size with each drop+pick.

Tests with multiple items (5x200 items instead of 1000x1 item) got similar results.