User:QQuix/On Dynamic Items and Savegame Bloating

Foreword

This is the result of some tests I am doing to determine/review the behavior of Dynamic and Non-Dynamic items.

I have reviewed, reformatted and split the original text into two more focused pages:

• Information related to savegame size remained here (tests results and conclusions)
• Other contents, related to behaviors of Dynamic and Non-dynamic items, were moved to [User:QQuix/ Dynamic and Non-dynamic references]

Contributions

• Please, use the talk page to suggest new info about the subject and/or propose bloat test scenarios.

Some disclaimers

• Be aware that this is a Work In Progress and the contents may change as a result of additional tests and contributions.
• Much of what is here is not new, and is written one way or another across the CS WIKI.

What is new?

Apr/2010 - Bloat tests with scripted items

Introduction

For definitions, read Modding Terminology, particularly the topics Dynamic Content and Dynamic Items.

The main objective of the tests was to get a firsthand, comprehensive experience with dynamic items, motivated by the above mentioned article and the PlaceAtMe bloat potential. And, while doing it, keep an eye for fine details, exceptions and odd results that might show up.

The tests were focused on dynamic items and savegame bloating.

The savegame file size increases naturally as the player advances thru the game, because the game engine has to keep an increasing amount of information about the game state.

• Some of this information is vital for the game play, e.g. player inventory, main quest state, NPCs that have died, etc.
• Some are important to keep, e.g. objects that the PC has placed in her home.
• Some are not that important: a sword the PC has dropped in a far cave that will never be visited again (quick tests lead to the conclusion that items dropped by the player are not removed by cell reset)
• Some are useless data due to bad/inadvertent modding practices, like PlaceAtMe + Disable + Forgotten
• Some are useless data due to engine bugs

I am using the term ‘bloating’ in its negative sense: when the savegame size increases due to apparently useless data.

To do

• DuplicateAllItems
• Inter-container item movement

Conclusions

(I am placing the conclusions up front for the benefit of those that don’t care to read the whole text)

Scripted items

WIP

Unscripted items

Adding/Removing items to/from containers

• AddItem increases in savegame size by ~13 bytes per ItemType. These bytes are removed when the ItemType is removed from the container.

• RemoveAllItems reduces ~13 bytes per ItemType when removing items added dynamically.

• RemoveAllItems has no effect in savegame size when removing items placed in the container in the CS. The information about the removal has to remain in the savegame so, the next time the esp is loaded (with the items in the container), the engine knows it has been removed.

• Bottom line: Adding and removing items to/from containers don’t cause bloating.

Adding/Removing items to/from the world

• Once placed in the world, the effect on savegame size is the same, regardless of how the item was created. Meaning: dropping 1000 individual quills from a container has the same effect as 1000 individual quills created with PlaceAtMe.

• Each item placed in the world added ~76 bytes to the savegame. Additional attributes, like ownership and health, may increase this value.

• Bloating may occur, depending on how the item is removed from the game world

• If items are picked up by the Player Character, the savegame size returns to its original size at cell reset.

• If items are moved to a container (NPC) by “MyItem.Activate MyNPC 1”, savegame is reduced correctly and immediately (before cell reset), and savegame bloating does not occur.

• If items are moved to a container (NPC) by “MyItem.Activate MyNPC 0”, savegame size is reduced by ~22 bytes immediately (before cell reset). The remaining 50+ bytes seem to remain in the savegame forever.

• The effects mentioned above are true even if a dynamic item is repeatedly added/dropped to/from a container. Meaning: a single item added/dropped to/from a container a thousand times has the same effect as 1000 individual items doing the same once.

• Bloating is the same for different items (tests used Flawless Emeralds, Quills and Paintbrushes).

• Waiting outside for cell reset, additionally reduced savegame size, but this reduction does not seem to be related to cleaning up leftovers from the test scenario: is was always about the same (~20K), no matter how many items were used

Are you using the original Oblivion.esm? It might be better to start with a blank .esm file, if you can find/make one. I remember that there are a few particular things needed in the .esm, but I don't remember which more than that.

Otherwise, you might be able to narrow it down by either waiting for another cell reset, or waiting for the first cell reset and then testing the save file.

--Haama 11:23, 17 September 2008 (EDT)

Never heard of a blank esm. Is it possible to load the game without the Oblivion.esm? QQuix 21:27, 17 September 2008 (EDT)

• Bottom line: Adding and removing items to/from the game world DO cause bloating if items are removed from the world with “MyItem.Activate MyNPC 0” or “MyItem.Activate MyNPC”. It seems that, when the RunOnActivateBlock Flag set to 0, the Activate function does not remove all the extra data created when the item is placed in the world.

Miscellaneous

1) All byte increases/decreases mentioned above are averages that may be affected by causes other then the test scenario. Therefore, consider them as good approximations rather than exact values.

2) The leftover bytes may be related to the following info on the Modding Terminology page:

However, under some circumstances the reference will not be removed from the save game. (Non-removal seems to be associated with having script record variables either attached to the reference and/or pointing at the reference.)

There is no way to do massive tests with thousands of items without “having script record variables pointing at the reference”. Unless, of course, having the PC pick up all those items one by one.

Therefore the leftover bytes may be related to this effect.

3) PlaceAtMe presented two different behaviors during the tests:

• Sometimes it behaved as mentioned above: added ~78 bytes when used and reduced ~23 when moved to a container
• Other times it added ~147 bytes when placed, reduced ~23 when moved to a container and reduced an additional ~70 bytes at cell reset

Although different, the net result was about the same.

4) Exit+Restart+Load+Save didn’t reduce savegame size. Since I have not tested this extensively, it is mentioned as an observation, rather than a conclusion.

5) Some observations about havok (gravity):

• In a few runs, 1000 items were dropped in a single frame loop. At the end of the frame, all were in the same spot in mid air. The next frame some items start falling, but it takes ~100 frames to the last one start falling. Eg. If the item is a Ruby, it forms a cascade of rubies pouring for 10-15 seconds from their original position.
• In the same scenario, if the items were spread in the air, still in the first frame (drop all and GetNextRef+SetPos), they don’t fall at the same time. Instead their fall forms like a wave, as if only a few items started falling on each subsequent frame.
• Then, if all the items now in the floor are moved to a certain height in a single frame, the next frame all of them start falling at the same time.

I don’t know if there are any conclusions to take from the fact that items placed in the air after being dropped don’t fall all at once as items already present in the world.

Another observation from the same scenario is that droping an item adds ~48 bytes to the savegame immediately (same frame). The remaining ~30 bytes are added on following frames, during/after their fall.

Scripted items tests [NEW]

General Methodology

• Two scripted MiscItems Base Objects - A and B - running the same script.
• Each instance of the script has 50 variables initialized with different values: 20 floats, 10 longs, 10 shorts and 10 refs (plus a few variables used by the script). Supposing 4 bytes per variable, there are, at least, 200 bytes worth of variables.

• Each instance of the script also has an array and 5 string vars. The array has 50 entries, each entry value being a string about 15 characters long. Each of the 5 string vars is about 40 characters long..

• The script has about 200 lines of code.

• Each item assign itself a unique ID when its script runs for the first time, so the item can 'sign' its log messages, even when in an inventory and does not have a FormID.

• The item behavior varies based upon 'global' variables in a quest script.

• Most tests were done by repeating some kind of action, typically the creation and destruction of a scripted item, over and over, thousands of times (whenever possible), saving the game at certain milestones and comparing savegame sizes.

• All tests started from the same save file, which is about 3Mb in size.

Test 1 = Flip flop

These tests involve adding and removing scripted items to/from the player inventory.

Each item, when added to the inventory by AddItem, waits a random number of frames, adds the other item and destroys itself.

There were four different test scenarios with four different ways the item destroys itself:

• Scenario 1 - Item script executes a RemoveMe function.
• Scenario 2 - Item removes itself to a persistent container nearby. Container is emptied every frame (RemoveAllItems)
• Scenario 3 - Item removes itself to a persistent container in a hidden cell. Container is emptied every frame (RemoveAllItems)
• Scenario 4 - Item removes itself to a dynamic container created with PlaceAtMe. Container is deleted at the end of the test (DeleteReference) with the scripted items still inside. This test had to be limited to 100 items as the game CTDs on save if there is more than 100 items in the container.

In scenarios 1, 2 and 3, there were around 20 items in the player inventory at any given time, as the items remove themselves after creating a new one.

In scenario 4 there was an increasing number of items and all items were deleted at the end.

Conclusions

• Adding and removing scripted items from inventory does not cause savegame bloating. In all scenarios, the savegame size returned to the initial values.

• Even when a container with scripted items inside is deleted, the game correctly removes all vanilla variables from those items from the savegame. OBSE arrays are also removed. But not OBSE strings, as OBSE strings must be explicitly destroyed.

• If the script uses OBSE arrays and/or strings, there is a possibility that the contents of these variables don't get removed from the OBSE co-save when there are more than one of that particular scripted item and one of them is removed. In this scenario there is no way to determine which one will be removed and the removed item script won't have a chance to clear its variables. Better avoid using these kinds of variables in carriable items scripts or devise some way to make sure the variables are cleared in every possible situation.

Test numbers

Each line represents a save game created at specific milestones

The numbers are the file size difference from the previous save, in bytes.

Num Items	Test 1	Test 2	Test 3	Test 4
After 20 items	+11,868	+11,981	+12,359	+12,864
After 30 items	-876	-638	-796	+6,448
After 40 items	-36	-12	-84	+5,298
After 50 items	+588	+604	+628	+6,612
After 100 items	-768	-856	-664	+29,305
After 200 items	+660	+796	+772
After 300 items	-552	-604	-640
After 400 items	+12	+48	-60
After 500 items	-84	-132	+36
After 1000 items	+132	+97	+24
After all items removed	-10,839	-11,193	-11,192	-59,636
Net result	+105	+91	+383	+891

Unscripted items tests

General Methodology

The tests were done with Misc Items only.

FormID tests

Create an item, place it in the world, locate and save its FormID, remove the item to a container and play with the saved formID to see what happens.

Bloating tests

Create 2000 items, place them in the world, remove them to a container and destroy them, saving the game along the way to measure bloating.

Variations

Item Creation

For the test, items were created and placed in the world four different ways:

• Placed in the world in the CS.
• Placed in a container in CS and dropped in the world.
• Dynamically created in a container with AddItem and droped in the world.
• Dynamically created in the world with PlaceAtMe.

Item Destruction

Items were removed from the game 3 ways

• RemoveAllItems
• RemoveItem
• RemoveMe (not done yet)

Other variations

• Scripted (not done yet) and non-scripted items.

Test Results

Invalid FormIDs

By Invalid FormIDs, I mean a dynamic formID after the item is moved to a container

• Item added to an NPC in CS.
• Dropped in game (gets FormID=FF000001).
• FormID saved in a Ref variable >> Set xItem to [item found by GetFirst/NextRef loop].
• At this point xItem contains FormID FF000001 (valid).
• Added to an NPC and dropped again (gets FormID=FF000002).
• At this point xItem still contains FormID FF000001 (now invalid: does not represent an existing item).

Using a reference to the now inexistent item, IN THE SAME FRAME, does not cause any visible problem.

• Normal, correct returns from PrintToConsole, GetBaseObject, GetPos, SetPos, MoveToMarker:

• PrintToConsole “%i %n” xItem xItem - prints FF000001 and the correct item name.
• xItem.GetBaseObject – returns the correct Base Object.
• SetPos – Sets new XYZ positions as verified by subsequent GetPos.
• MoveToMarker – Sets new XYZ positions as verified by subsequent GetPos.

Using a reference to the now inexistent item, in following frames:

• xItem becomes a Null reference.
• PrintToConsole “%i %n” xItem xItem - prints 0 NULL.
• xItem.GetBaseObject – returns Null.
• xItem.SetPos – Sets new XYZ positions to the object where the script is running (as xItem==Null, assumes Self??).
• xItem.MoveToMarker – does not seem to do anything.

Conclusions

It seems that after an item is removed from the world as described above, all its data remains intact and can be manipulated by scripts in the same frame they are removed.

Using a reference to the now inexistent item, in following frames:

• The functions tested did not crash the game.
• Most of the time, it seems the function does nothing.
• But sometimes, have undesired results (as in the SetPos).

Bloating

Test scenario

All runs started from the same clean save.

The tables show the variation (in bytes) in savegame size after each step of the tests.

Tests where done in an additional Divine Elegance basement. The “Going to exterior” step means leaving the basement, crossing DE cell and activating the exit door. Saves where made immediately after the exterior cell was loaded.

All bytes increases/decreases mentioned in the Conclusions were calculated dividing the difference in savegame size by the number of items used in that run (typically 1000). Since savegame size is affected by other causes, the values collected may not be exclusively due to the test environment, but, probably, are good approximations.

RemoveAllItems Tests

The column “With RemoveAllItems” shows the results of the test as initially intended.

Since the RemoveAllItems didn’t seem to reduce savegame, I re-ran all tests without using it. The results are in the last column

Items were moved to the container (NPC) using “xItem.Activate xNPC 0”

	With	Without
	RemoveAllItems	RemoveAllItems
After adding 1000 items	+78.373
After adding 2000 items	+78.416
After moving 1000 items to container	-22.670	-22.959
After moving 2000 items to container	-23.014	-23.014
After RemoveAllItems	-18
After going to exterior	+13.801	+14.316
After Waiting 4 days	-21.218	-24.561
After returning to cel	-5.265	-2.968
Net result	+98.405	+97.604

Tests were run 6 times, two times for each of the creation method mentioned earlier.

Individual runs did’t vary much from each other, so they are not included here.

The initial size of the savegame is 2.983.546 bytes .

For each of the three types of dynamic items considered, the test consisted of:

• Placing 1.000 items in the world (either by dropping from a container or by PlaceAtMe, one item per frame)
• Placing 1.000 more items in the world (same)
• Moving 1.000 items to an NPC (Single frame GetNextRef loop to identify the item and then NPC.Activate Item)
• Moving 1.000 items to an NPC (same frame as above)
• Using RemoveAllItems on the NPC
• Going to an exterior cell
• Waiting 4 days
• Returning to the test cell
• Saving the game at the beginning and after each of the steps above

NOTE: 2000 items seems to be the limit for my hardware to handle (down to about 3-5 FPS)

Multiple Drop+PickUp Tests

1.000 items were created four different ways (see below)

Then they were added/dropped to/from an NPC a number of times.

The game was saved after each operation using the con_Save function

Items were moved to the container (NPC) using “xItem.Activate xNPC 0”

The column acronyms mean:

• CS-W – Items created in the CS – placed in the cell
• CS-C – Items created in the CS – placed in a container
• AI – Items created ingame by AddItem function
• PAM – Items created ingame by PlaceAtMe function

	CS-W	CS-C	AI	PAM
	(Non-dynamic)	(Dynamic)	(Dynamic)	(Dynamic)
After moving	+70.518	---	---	---
After PickUp	+513	---	---	---
After AddItem	---	---	+601	---
After PlaceAtMe	---	---	---	+147.429
After Droping (1)	+19.003	+77.770	+79.060	---
After PickUp (1)	-9.022	-22.902	-22.878	-21.862
After Droping (2)	+9.991	+78.906	+79.910	+55.368
After PickUp (2)	-9.968	-22.947	-22.942	-22.929
After Droping (3)	+10.299	+80.040	+78.952	+79.101
After PickUp (3)	-9.977	-22.958	-22.937	-22.934
After Droping (4)	+9.925	+78.948	+79.099	+78.939
After PickUp (4)	-9.968	-22.953	-22.942	-22.934
After Droping (5)	+10.220	+68.855	+78.947	+79.270
After PickUp (5)	-9.975	-22.966	-22.942	-22.934
After going to exterior	+14.960	+12.741	+13.119	+14.980
After Waiting 4 days	-21.676	-19.482	-19.227	-68.350
After returning to cell	-5.918	-5.137	-4.707	-4.196
Net result	68.925	+257.915	+271.113	+268.948

Each column shows the result of just one run.

As can be noticed, non-dynamic items caused an initial increase in savegame size (as explained in the Conclusions section) and, after that, they didn’t affect it much.

Dynamic items, on the other hand, consistently increased savegame size with each drop+pick.

Tests with multiple items (5x200 items instead of 1000x1 item) got similar results.

Tests with RunOnActivateBlock Flag set to 0 and set to 1

1.000 items (5 x 200 ItemTypes) were created with AddItem

Then they were dropped and added from/to an NPC a number of times.

The game was saved after each operation using the con_Save function.

Items were moved to the container (NPC) using “xItem.Activate xNPC 0” (first column) and “xItem.Activate xNPC 1” (second column).

-	With	With
	Flag = 0	Flag = 1
After AddItem	+2.914	+2.966
After Droping (1)	+77.174	+73.781
After PickUp (1)	-20.887	-73.599
After Droping (2)	+73.500	+76.932
After PickUp (2)	-20.891	-76.887
After Droping (3)	+76.743	+76.909
After PickUp (3)	-20.781	-76.855
After Droping (4)	+77.171	+76.978
After PickUp (4)	-20.517	-77.004
After Droping (5)	+73.759	+76.933
After PickUp (5)	-20.748	-76.917
After RemoveAllItems	-2.690	-2.682
After leaving cell	-577	-33
After Cell Reset	+145	-62
After returning to cell	+680	+728
Net result	+ 274.996	+1.189
Average drop	+75.670	+76.307
Average pick up	-20.765	-76.252

As can be noticed, using “xItem.Activate xNPC 0” caused an increase in savegame size by not removing the data added by the Drop function. Using “xItem.Activate xNPC 1” correctly removed the added data.

Each column shows the average of three runs.

Individual runs did’t vary much from each other, so they are not included here.

TO DO: Re-test RemoveItem

TO DO: Test with RemoveMe.

TO DO: Other item types