Editing User:QQuix/On Dynamic Items and Savegame Bloating

=Foreword=
This is the result of some tests I am doing to determine/review the behavior of Dynamic and Non-Dynamic items.

For definitions, read [[Modding Terminology]], particularly the topics “Dynamic Content” and “Dynamic Items”.

Most of what is here is not new, and is written one way or another across the CS WIKI.

Feel free to add comments/suggestions either here or to the Talk page. I suppose any specifics would fit nicely under the text you are commenting on. General comments/suggestions may fit better in the Talk page. Either way, all are welcome.

Be aware that this is a Work In Progress and the contents may change as a result of additional tests and contributions.

=Introduction=
The main objective of the tests was to get a firsthand, comprehensive experience with dynamic items, motivated by the above mentioned article and the PlaceAtMe bloat potential. And, while doing it, keep an eye for fine details, exceptions and odd results that might show up.

The tests were focused on dynamic items and savegame bloating.

::NOTE: Most probably these kinds of tests have been done by other modders. Any info on equivalent tests would be appreciated.

====To do====
*Scripted items
* DuplicateAllItems
*Inter-container item movement
=Conclusions=
(I am placing the conclusions up front for the benefit of those that don’t care to read the whole text)

==Dynamic items==
*When a dynamic item is moved to a container, a Ref Variable containing its FormID is not invalidated immediately. It remains intact for the rest of the frame. It turns to Null in subsequent frames.
::TODO: check if it turns to Null right in the next frame.
:::There will be a new function in v16, '''GetCurrentFrameID''', that will make this test easier and more accurate.
:::--[[User:Haama|Haama]] 11:05, 17 September 2008 (EDT)

*Functions using the null Ref Variable either are ignored or produce undesirable results. Didn’t experience any CDT with the few functions tested. 
::<s>TODO: Test more functions.</s> Agree. These are separate tests.
:::It might be better to think of these as separate tests - if the reference variable is 0, then it's 0, and it doesn't matter if it hasn't been initialized yet or if it just turned 0.
:::If the reference remains non-0, however, that's a different story. In general, OBSE functions are very good about ignoring bad references while vanilla functions crash and burn.
:::--[[User:Haama|Haama]] 11:05, 17 September 2008 (EDT)

*A free FormID is reused only if it is after/higher than the last used one. Free FormIDs in the middle of the series are not reused. Example: Drop 3 items. Lets say they receive FormIDs 11,12 and 13. If you pick #13 up and drop it again, it will be 13 again, one higher than the last used (which is now #12). If you pick #12 up and drop it again, it will be FormID 14, one higher than the last used (#13).

==Non-Dynamic items==
*Items placed in the CS also add to the savegame size once, when they are moved from its original position for the first time. Although not obvious, this makes sense: while the item stays in its original position, their data is loaded from the esp/esm. Once moved, the game must start keeping track of its position and saves the data in the savegame. The increase is around 70 bytes per item.
:::Would [[SetAtStart]] get rid of this bloat? Not really worth it, but there are some very picky people out there.
:::--[[User:Haama|Haama]] 11:07, 17 September 2008 (EDT)
::::SetAtStart does not seem to work (as its page says). I would guess that the original idea was what you mean: not saving any data on the item, at the next load it would stay in its start location. <s>But to move it immediately, I wonder if the game keeps its start location in memory.</s> It does.
::::[[User:QQuix|QQuix]] 21:27, 17 September 2008 (EDT)

==Bloating tests==
(I am using the term ‘Bloating’ in its negative sense: when the savegame size increases due to apparently useless data)

====Adding/Removing items to/from containers====
*AddItem increases in savegame size by ~13 bytes <u>per ItemType</u>. These bytes are removed when the ItemType is removed from the container.
:::See above, I believe you're adding the same item. If you add a different item the savegame size will change. Where else would the information be stored?
:::--[[User:Haama|Haama]] 11:23, 17 September 2008 (EDT)
::::You are right in both cases. As I was testing with 1000 x same item, the difference was so small to relate it to the functions. Further tests with 222 different items showed a better figure. Text fixed. Conclusion added.
:::: [[User:QQuix|QQuix]] 21:27, 17 September 2008 (EDT)

*RemoveAllItems has no effect in savegame size when removing items placed in the container in the CS. The information about the removal has to remain in the savegame so, the next time the esp is loaded (with the items in the container), the engine knows it has been removed. 

*RemoveAllItems reduces ~13 bytes <u>per ItemType</u> when removing items added dynamically.  

*<s>RemoveAllItems HAS NO EFECT in savegame size. (Surprise! Surprise!)</s>
:::Umm... that is surprising. If an actor has 100s of items they would need a linked list of each item's reference and quantity. Each reference is at least 32-bit, IIRC, so... yeah, that would be surprising...
:::My only guess - the linked list never really goes away. So, if you add an Apple to an NPC, use RemoveAllItems, then when you add a second Apple it won't cause any more bloat. (BTW, annoying as all *#!@%)
:::--[[User:Haama|Haama]] 11:23, 17 September 2008 (EDT)
::::You are right. It seems the linked list entry is created when a new ItemType is added to a container and is removed by RemoveAllItems. So, the net result is zero. Meaning: adding and removing items to containers don’t cause bloating. [[User:QQuix|QQuix]] 21:27, 17 September 2008 (EDT)

::TODO: Re-test RemoveItem 
::TODO: Test with RemoveMe.

*Adding and removing items to/from containers don’t cause bloating.


====Adding/Removing items to/from the world====

*Once placed in the world, the effect on savegame size is the same, regardless of how the item was created. Meaning: dropping 1000 individual quills from a container has the same effect as 1000 individual quills created with PlaceAtMe.
:::I assume you mean 1000 stacks of 1 quill?
:::--[[User:Haama|Haama]] 11:23, 17 September 2008 (EDT)
::::Yes. Text adjusted. Feel free to rephrase it if you like. [[User:QQuix|QQuix]] 21:27, 17 September 2008 (EDT)

*Each item placed in the world added ~78 bytes to the savegame.
:::There are a few more variables you could try - the owner of the items (PlaceAtMe leaves it at 0), the script variables, etc. Generally this is refered to as ''extra data''
:::--[[User:Haama|Haama]] 11:23, 17 September 2008 (EDT)
::::Good point. Do you know if there is any documentation on these fields. UESP has a lot, but not at this level of detail. [[User:QQuix|QQuix]] 21:27, 17 September 2008 (EDT)

*Bloating may occur, depending on how the item is removed from the game world

:*If items are picked up by the Player Character, the savegame size returns to its original size at cell reset.

:*If items are moved to a container (NPC) by “xItem.Activate xNPC 1”, savegame is reduced correctly and immediately (before cell reset), and savegame bloating does not occur. 

:*If items are moved to a container (NPC) by “xItem.Activate xNPC 0”, savegame size is reduced by ~23 bytes immediately (before cell reset). The remaining 50+ bytes seem to remain in the savegame forever.

*The effects mentioned above are true even if a dynamic item is repeatedly added/dropped to/from a container. Meaning: a single item added/dropped to/from a container a thousand times has the same effect as 1000 items doing the same once.

*Bloating is the same for different items (tests used Flawless Emeralds, Quills and Paintbrushes).
:::Note that all of those items are misc. items. (Probably won't matter, though)
:::--[[User:Haama|Haama]] 11:23, 17 September 2008 (EDT)

*Waiting outside for cell reset, additionally reduced savegame size, but this reduction does not seem to be related to cleaning up leftovers from the test scenario: is was always about the same (~20K), no matter how many items were used
:::Are you using the original Oblivion.esm? It might be better to start with a blank .esm file, if you can find/make one. I remember that there are a few particular things needed in the .esm, but I don't remember which more than that.
:::Otherwise, you might be able to narrow it down by either waiting for another cell reset, or waiting for the first cell reset and then testing the save file.
:::--[[User:Haama|Haama]] 11:23, 17 September 2008 (EDT)
::::Never heard of a blank esm. Is it possible to load the game without the Oblivion.esm? [[User:QQuix|QQuix]] 21:27, 17 September 2008 (EDT)

*Adding and removing items to/from the game world DO cause bloating if items are removed from the world with “xItem.Activate xNPC 0” or “xItem.Activate xNPC”. It seems that, when the RunOnActivateBlock Flag set to 0, the Activate function does not remove all the extra data created when the item is placed in the world. 


===Observations and Comments=== 
1)	All byte increases/decreases mentioned above are averages that may be affected by causes other then the test scenario. Therefore, consider them as good approximations rather than exact values.


2)	The leftover bytes may be related to the following info on the [[Modding Terminology]] page:
<blockquote>
''However, under some circumstances the reference will not be removed from the save game. (Non-removal seems to be associated with having script record variables either attached to the reference and/or pointing at the reference.)''
</blockquote>

:There is no way to do massive tests with thousands of items without “having script record variables pointing at the reference”. Unless, of course, having the PC pick up all those items one by one.

:Therefore the leftover bytes may be related to this effect.


3) PlaceAtMe presented two different behaviors during the tests:
:*Sometimes it behaved as mentioned above: added ~78 bytes when used and reduced ~23 when moved to a container
:*Other times it added ~147 bytes when placed, reduced ~23 when moved to a container and reduced an additional ~70 bytes at cell reset

:Although different, the net result was about the same.


4) Exit+Restart+Load+Save didn’t reduce savegame size. Since I have not tested this extensively, it is mentioned as an observation, rather than a conclusion.


5) Some observations about havok (gravity):

:*In a few runs, 1000 items were dropped in a single frame loop. At the end of the frame, all were in the same spot in mid air. The next frame some items start falling, but it takes ~100 frames to the last one start falling. Eg. If the item is a Ruby, it forms a cascade of rubies pouring for 10-15 seconds from their original position. 
:*In the same scenario, if the items were spread in the air, still in the first frame  (drop all and GetNextRef+SetPos), they don’t fall at the same time. Instead their fall forms like a wave, as if only a few items started falling on each subsequent frame. 
:*Then, if all the items now in the floor are moved to a certain height in a single frame, the next frame all of them start falling at the same time.

:I don’t know if there are any conclusions to take from the fact that items placed in the air after being dropped don’t fall all at once as items already present in the world.

:Another observation from the same scenario is that droping an item adds ~48 bytes to the savegame immediately (same frame). The remaining ~30 bytes are added on following frames, during/after their fall.

=Common knowledge / Backgound=

====Non-Dynamic items====
*Non-Dynamic items are items placed in the world in the CS.
*Formid starts with Mod# (NNxxxxxx).
*FormIDs are stable after adding to and dropping from a container, therefore FormIDs saved in Ref variables may be reused over time.
*At game load, the engine loads data from esps/esms first. Then, it loads data from the savegame, updating the data in memory with whatever has been changed during the course of play.

====Dynamic items====
*Dynamic items are items placed in the world during the course of the gameplay.
*Dynamic items may be created by dropping an item from a container (either placed in the container in the CS or added on the fly by AddItem) or may be created directly in the world by PlaceAtMe.
*Formid starts with FF (FFxxxxxx).
*FormIDs are dynamic and may change after adding to and dropping from a container, therefore FormIDs saved in Ref variables should not be reused over time.	
*The ‘holes’ left in the FormID sequence will be reused when the count reaches FFFFFFFF. [http://www.uesp.net/wiki/Tes4Mod:Formid UESPWiki Tes4Mod:Formid Advanced Questions]

=General Methodology=
The tests were done with Misc Items only.
===FormID tests===
Create an item, place it in the world, locate and save its FormID, remove the item to a container and play with the saved formID to see what happens.
===Bloating tests===
Create 2000 items, place them in the world, remove them to a container and destroy them, saving the game along the way to measure bloating.
===Variations===

====Item Creation====
For the test, items were created and placed in the world four different ways:
:*Placed in the world in the CS.
:*Placed in a container in CS and dropped in the world.
:*Dynamically created in a container with AddItem and droped in the world.
:*Dynamically created in the world with PlaceAtMe.

====Item Destruction====
Items were removed from the game 3 ways
:*RemoveAllItems 
:*RemoveItem 
:*RemoveMe (not done yet)

====Other variations====
:*Scripted (not done yet) and non-scripted items.

=Test Results=

== Invalid FormIDs ==
By Invalid FormIDs, I mean a dynamic formID after the item is moved to a container
:*Item added to an NPC in CS.
:*Dropped in game (gets FormID=FF000001).
:*FormID saved in a Ref variable >> Set xItem to [item found by GetFirst/NextRef loop].
:*At this point xItem contains FormID FF000001 (valid).
:*Added to an NPC and dropped again (gets FormID=FF000002).
:*At this point xItem still contains FormID FF000001 (now invalid: does not represent an existing item).
Using a reference to the now inexistent item, IN THE SAME FRAME, does not cause any visible problem.
:*Normal, correct returns from PrintToConsole, GetBaseObject, GetPos, SetPos, MoveToMarker:
::*PrintToConsole “%i %n” xItem xItem - prints FF000001 and the correct item name.
::*xItem.GetBaseObject – returns the correct Base Object.
::*SetPos – Sets new XYZ positions as verified by subsequent GetPos.
::*MoveToMarker – Sets new XYZ positions as verified by subsequent GetPos.
Using a reference to the now inexistent item, in following frames:
:*xItem becomes a Null reference.
:*PrintToConsole “%i %n” xItem xItem - prints 0 NULL.
:*xItem.GetBaseObject – returns Null.
:*xItem.SetPos – Sets new XYZ positions to the object where the script is running (as xItem==Null, assumes Self??).
:*xItem.MoveToMarker – does not seem to do anything.

===Conclusions===
It seems that after an item is removed from the world as described above, all its data remains intact and can be manipulated by scripts in the same frame they are removed.

Using a reference to the now inexistent item, in following frames:
:*The functions tested did not crash the game.
:*Most of the time, it seems the function does nothing.
:*But sometimes, have undesired results (as in the SetPos).

==Bloating==

====Test scenario====
All runs started from the same clean save.

The tables show the variation (in bytes) in savegame size after each step of the tests.

Tests where done in an additional Divine Elegance basement. The “Going to exterior” step means leaving the basement, crossing DE cell and activating the exit door. Saves where made immediately after the exterior cell was loaded.

All bytes increases/decreases mentioned in the Conclusions were calculated dividing the difference in savegame size by the number of items used in that run (typically 1000). Since savegame size is affected by other causes, the values collected may not be exclusively due to the test environment, but, probably, are good approximations.

===RemoveAllItems Tests===
The column “With RemoveAllItems” shows the results of the test as initially intended.

Since the RemoveAllItems didn’t seem to reduce savegame, I re-ran all tests without using it. The results are in the last column 

<dl><dl><dl>
{| class="wikitable"
|-
|
|align="center" |With
|align="center" |Without
|-
|
|align="center" |RemoveAllItems
|align="center" |REmoveAllItems
|-
|After adding 1000 items
|align="center" |+78.373	
|-
|After adding 2000 items
|align="center" |+78.416	
|-
|After moving 1000 items to container
|align="center" |-22.670
|align="center" |-22.959
|-
|After moving 2000 items to container
|align="center" |-23.014
|align="center" |-23.014
|-
|After RemoveAllItems
|align="center" |-18	
|-
|After going to exterior
|align="center" |+13.801
|align="center" |+14.316
|-
|After Waiting 4 days
|align="center" |-21.218
|align="center" |-24.561
|-
|After returning to cel
|align="center" |-5.265
|align="center" |-2.968
|-
|Net result
|align="center" |+98.405	
|align="center" |+97.604
|}
</dl></dl></dl>

Tests were run 6 times, two times for each of the creation method mentioned earlier.

Individual runs did’t vary much from each other, so they are not included here.

The initial size of the savegame is 2.983.546 bytes .

For each of the three types of dynamic items considered, the test consisted of:
:*Placing 1.000 items in the world (either by dropping from a container or by PlaceAtMe, one item per frame)
:*Placing 1.000 more items in the world (same)
:*Moving 1.000 items to an NPC (Single frame GetNextRef loop to identify the item and then NPC.Activate Item) 
:*Moving 1.000 items to an NPC (same frame as above)
:*Using RemoveAllItems on the NPC
:*Going to an exterior cell
:*Waiting 4 days
:*Returning to the test cell
:*Saving the game at the beginning and after each of the steps above
::NOTE: 2000 items seems to be the limit for my hardware to handle (down to about 3-5 FPS)



===Multiple Drop+PickUp Tests===
1.000 items were created four different ways (see below)

Then they were added/dropped to/from an NPC a number of times.

The game was saved after each operation using the con_Save function

The column acronyms mean:
*CS-W  – Items created in the CS – placed in the cell
*CS-C – Items created in the CS – placed in a container
*AI – Items created ingame by AddItem function
*PAM – Items created ingame by PlaceAtMe function

<dl><dl><dl>

{| class="wikitable" style="width:75%; text-align:right"
!|
!align="center" | CS-W
!align="center" | CS-C
!align="center" | AI
!align="center" | PAM
|-
| || (Non-dynamic)|| (Dynamic)|| (Dynamic)|| (Dynamic)
|-
| align="left" |After moving
| +70.518|| ---|| ---|| ---
|-
|align="left" | After PickUp
| +513|| ---|| ---|| ---
|-
|align="left" | After AddItem
| ---|| ---||+601|| ---
|-
|align="left" | After PlaceAtMe
| ---|| ---|| ---||+147.429
|-
|align="left" | After Droping (1)
| +19.003||+77.770||+79.060|| ---
|-
|align="left" | After PickUp (1)
| -9.022||-22.902||-22.878||-21.862
|-
|align="left" | After Droping (2)
| +9.991||+78.906||+79.910||+55.368
|-
|align="left" | After PickUp (2)
| -9.968||-22.947||-22.942||-22.929
|-
|align="left" | After Droping (3)
| +10.299||+80.040||+78.952||+79.101
|-
|align="left" | After PickUp (3)
| -9.977||-22.958||-22.937||-22.934
|-
|align="left" | After Droping (4)
| +9.925||+78.948||+79.099||+78.939
|-
|align="left" | After PickUp (4)
| -9.968||-22.953||-22.942||-22.934
|-
|align="left" | After Droping (5)
| +10.220||+68.855||+78.947||+79.270
|-
|align="left" | After PickUp (5)
| -9.975||-22.966||-22.942||-22.934
|-
|align="left" | After going to exterior
| +14.960||+12.741||+13.119||+14.980
|-
|align="left" | After Waiting 4 days
| -21.676||-19.482||-19.227||-68.350
|-
|align="left" | After returning to cell
| -5.918||-5.137||-4.707||-4.196
|-
| ||||||||
|-
|align="left" | Net result||68.925
| +257.915||+271.113||+268.948
|-
| 
|}
</dl></dl></dl>


Each column shows the result of just one run. 

As can be noticed, non-dynamic items caused an initial increase in savegame size (as explained in the Conclusions section) and, after that, they didn’t affect it much.

Dynamic items, on the other hand, consistently increased savegame size with each drop+pick.

Tests with multiple items (5x200 items instead of 1000x1 item) got similar results.