RunUO Community

This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

Quicker Save Times (Discussion/Etc)

Armageddon

Wanderer
I had originally posted this over at the Ultima Online Second Age forums but someone suggested I try it here since it more globally affects all RunUO shards.

This topic is meant as a discussion and/or brainstorming session and, maybe, eventually a step into changing the code to improve save times.

What I know:
1. All items/mobiles are serialized and saved as a binary world save.
2. Save times can be lengthy if there are lots of items stored/hoarded in the world.
3. The world has lots of items that sit in containers and never get touched or moved.
4. The server is restarted every day (I think, this may apply only to UOSA).

My thought is to do the following is to create a system where unchanged items are not repeatedly saved. If only modified items are saved, then world saves would be far smaller...

- At server startup: All containers in the world are flagged as 'untouched'. The world save is loaded from an untouched database.

- During the server operation, any time a container is opened, moved, or otherwise changed, it is flagged as touched. This includes player backpacks, banks, boxes in houses, etc.

- Each time the world saves, all 'touched' items are saved to a separate temporary save file. Untouched items are not saved, as they have not changed.

- Each day before the server is taken down (Or once every maintenance interval), the untouched database and the touched database are compared, merged, and saved off as a new untouched database for the next days operation. This process can be slower, as it will only happen once and only happen once players are booted.

- There would also need to be a restore functionality to relaunch the server using both the untouched database and the touched database, just in case there is a server crash. This would be, I imagine, similar to the above merge and save code and would basically be loading all untouched data, then loading touched data (being sure to overwrite any untouched that it encounters).

So for you RunUO experts out there. Why isn't something like this already implemented? Are there any big gotcha's or walls that you'd hid? I am not an expert on RunUO but I am an experienced software engineer. Based on what I know and see, the whole 'saving' system seems a little clunky and could use some improvement.
 

Jeff

Lord
What you are talking about sounds like a transactional database. Ideally it could work, problem is the amount of effort that would go into it, and the potential problems it could attract if just 1 property forgets to set the "is dirty" flag to true on change. Especially with custom scripts. You could no longer drop scripts and expect them to work, you'd have to constantly go through each script and modify it to work with your current serialization system. While this may work with servers with administrators who know what they are doing code-wise. The idea behind RunUO is to support a much broader audience in which maintaining a system like this would require to much effort.

The idea sounds nice until actually attempt to implementing it, at which time the idea becomes daunting, overwelming, and not much reward for effort. I honestly don't understand the whole why serialization is such a controversy, I mean really, look at what we are talking about. You are talking about writing a complex serialization engine in order to gain at most 10-45 seconds per hour of playtime ;). When you put the whole thing into perspective, its really quite a meaningless subject. Sure, seamless is great, and it would be nice if RunUO didn't freeze the server for a world save. But is hours upon hours of effort worth a few seconds of time? In this particular instance, I do not believe so.

As for a 'clunky' saving process, sure RunUO's saving process is pretty basic, but if you look at how any other MMO saves their worlds you can see they use databases. Now, is this doable with RunUO? Absolutely. Should it be done? Probably not. Why? If you look at all the other companies and look for differences you will notice they have a few things. 1. Money, 2. Employees, 3. DBAs that are constantly monitoring database load 24/7. Now, for RunUO Administrators, items 1 and 2 are easy, they run it, so they are the employees and in this instance their time is the money. The big problem is item #3. Maintaining a database like this would require a lot more time on the Adminsitrators part, and if done properly all transactions would be queued (instead of 1 big world save every hour). This queue could become bottlenecked for several reasons, in which the Administrator would need to immediately find an answer to and solve otherwise, the queue could cause out of memory errors, in which time the server would crash and all saved data in the queue would be lost. This is not something I see feasible by a large majority of the community. Is there room in between for some other option, sure, but I always come to the conclusion I state before, the reward for the amount of effort is so small, that I doubt its worth it.

By the way, I'm assuming this came up on the second age server due to their saves taking a roughly long amount of time. Have you/them/anyone looked into the new .Net 4.0 save strategy created by Asayre? It uses TPL, allowing the save process to spool up the entire save in memory, and using TPL write it to disk while the server keeps running. In testing this had a 20-30% increase in performance on an average server. Perhaps someone over there should look into this.
 

tass23

Page
Jeff, is the database usage the reason why OSI shards don't have a freeze during a save? Or was it because there are more than one server running one facet at a time?
 

tass23

Page
Okay. I don't have an issue with saves, as I'm sure most don't, under 4 seconds seems a fair trade for not using a database and having the ability to have your own custom shard. The only way I see a databased save process working is if every addon were written following the same structure, and I don't see that happening because I think it would limit the designers abilities to "think outside the box" as it were. If you look at some of the addons that are posted here, a lot of them are entirely new concepts, yes, based on EA and RunUO code, but I don't see any database able to adapt to a new structure every time a new concept addon is installed. You're right Jeff, it would require the shard owners and/or admins to constantly change the database, even going so far as rebuilding it. What a nightmare.
 

Armageddon

Wanderer
Jeff, thanks for the response. Very informative.

I was under the impression that RunUO was more class based and had figured that there would be a base script/class which handled containers being dirty/clean. If a person would have to manually update every script, then it would be a nightmare for sure.

On a slightly different note, how much effort would it be to have things saved to a mysql database instead of serially? I am assuming in a database set up you would pretty much save data as you go instead of set 'world saves'. Thus if I logged in, did stuff, and logged out I would be saved immediately instead of waiting until the next overall save? I could see dupe problems with this...and perhaps that is where the old black hole dupe came from. If my character gives your 1 million gold then you log out it saves you. Then I force my character to get corrupted (black hole bug) and it reverts me to a previous save and I have the million gold back.
 

Jeff

Lord
There was a project going by Acronis a while back. He was trying to save (serialize and deserialize) to a database, and from what I read, loading took forever, but saving was pretty quick. However, his database was pretty bland, he had an item table, with 1 column designated to something like a comma seperated list of properties not included in Item (so anything that inherited from Item got jammed into 1 column). Seems like lazy work imo. If you were to do it properly, it would require nearly the same re-write stated above (you'd have to queue a change to the database for each property, that'd be a bit of a mess).

As for the black hole bug, i doubt that'd be an issue if you did things properly. (On OSI, that was a bug... a bug caused by improperly saving, and it was fixed).
 

Peoharen

Sorceror
If you really wanted to speed up saving the first step would be to save less stuff.

An example would be Item. Every item saves Dupe information, Bounce Info, Direction, LootType, ItemID, Hue, Layer, Name, Parent Information, Items, Weight, Map, Location, Blessed for, Insured for, and various other stuff. Further ALL weapons, armor, shields, jewels, clothing, and even spell books save an entire lists of proprieties. Mobile wise, Oh. My. God.

There should have been a "BaseItem", all it would have saved is Location, Map, ItemID, Hue, and Name. Then this item is what should be decorating your world. This small edit would halve saved data on small shards where the decorating for five maps exceeds the player's stockpiled loot. Stuff like Layer, Weight, Parent, Can't-Lose-Me should default to stop screwing with me you retarded GM. Likewise, the only reason AosAttributes, AosWeaponAttributes, AosArmorAttributes, AosSkillBonuses, SAAbsorptionAttributes are saved on every damn warble ever was code author simplicity and the option to GM crap up. What should be done is have a Loot type item class where that information is saved (since loot is randomized) and a none saved loot class.

The nonloot class would skip saving damn near everything. Seriously, why is Weapon Speed even saved to begin with? It's a GM-altered only value, screw them. Use the scripted value instead and never code some in game alterable prop your forced to save & load. Just out of BaseWeapon you can skip Speed, MinDamage, MaxDamage, Range, Stat Requirements, Skill, Animation, Type, & Sound. Guess what you can skip in the rest? Anyway, the nonloot class would go one step further and simply use the default properties assigned to it. You're not saving the information at all, both end up creating a new AosAttributes isntance but one reads from scripted information while the other is used to reading from a file so you even save a few cycles of resources in world loading.

Moving on to Mobiles. As I mentioned on Orb, kudos to every other MMO out there. Exact states of creatures are not saved. In WoW, instance stuff aside the only value saved is if they were killed or not. When a WoW dungeon instance crashes you are kicked back to the start and everything not fully dead is still alive. DDO (and probably LotRs) goes a bit further and saves the Player's Location so at least you're not kicked back to the entrance. You can do the same damn thing with RunUO. Don't save any creature's hits/stam/mana/etc., including players. Starting with max HP/Stam/Mana from a server restart isn't a detail anyone cares about. Further skip saving location information of creatures, just let Spawners run TotalRespawn at world load.

Implementing even 1/4 of the ideas every other MMO uses would result in a huge reduction of save time and world load. And what's stopping that? The ability to restore in the event of an unexpected crash exactly to the last world save? I call bullshit. Most freeshards save once an hour, or 30mins, or 15mins. Heck, even at the default 5mins you can still kill a dozen monsters and even fully complete several quests. This concept of save the exact state was thrown out the window to begin with, why pretend otherwise?
 

Jeff

Lord
Having a baseitem doesnt save space.... the same amount of data is stored.... Our base item is called Item.cs and it contains the stuff you proposed to store... even if it didnt, and those things were in each class like Weapon, Armor... its still the same amount of info... not more...

Having things not save like min/max damage is stupid.... that removes the ability to customize... You wouldn't be able to create a script that generated items, instead you would have to create a hard script copy of every single item you ever wanted to introduce to the game...

The save size is fine, flat file saving is always the bottleneck... don't like it, write something else, but everything else requires a ton of maintenance and work...
 

Peoharen

Sorceror
Having a baseitem doesnt save space.... the same amount of data is stored.... Our base item is called Item.cs and it contains the stuff you proposed to store... even if it didnt, and those things were in each class like Weapon, Armor... its still the same amount of info... not more...
I was referring to decorating type items (ie static) wouldn't use such details as weight, loot type, blessed for, insured by, bounce information & container data (as they would be fully intended to remain on the map), and what ever else which would not only cut save information down, but also the overhead in simply having them in game as there is simply less data tied to them.

Having things not save like min/max damage is stupid.... that removes the ability to customize... You wouldn't be able to create a script that generated items, instead you would have to create a hard script copy of every single item you ever wanted to introduce to the game...
I said What should be done is have a Loot type item class where that information is saved (since loot is randomized) and a none saved loot class. then spent an entire paragraph explaining that. Yes, the ability for a GM to props in game and alter the min/max base damage of a weapon would be blocked. Now ask your self why in the hell that is a bad thing? Jesus, you get several search hits for people bitching about GMs tweaking items and exploiting their power but you want to say just let them change the base damage on a whim and say the idea of not having that option is stupid? Oh, and again overhead would be reduced as you don't save yet another integer in the memory when for all purposes it is the one you already compiled in and are using.

Hard coded items, as example used artifacts, wouldn't need to save their attribute information - just use the default. Random loot of course would have to, but that is the only real hard part.

The save size is fine, flat file saving is always the bottleneck... don't like it, write something else, but everything else requires a ton of maintenance and work...
It was a lot of work to stop using Hair & Beard items and it's not like all those HashTables & ArrayLists converted them selves to generics.

All I'm saying is there is plenty of ways to reduce the information saved (and yes, memory consumption) and gave my thoughts on that. I really don't think your trying to come off as saying optimizing stuff is horrible, just trying to defend the reasons for not you not doing such. Well I'm not pointing figures at you, or anyone. People have been working on better code and will continue to do so, so have some ideas.
 
I was referring to decorating type items (ie static) wouldn't use such details as weight, loot type, blessed for, insured by, bounce information & container data (as they would be fully intended to remain on the map), and what ever else which would not only cut save information down, but also the overhead in simply having them in game as there is simply less data tied to them.

Statics without altered properties should be frozen, and statics that aren't frozen usually have a need for them to be, such as hue, lighting or other properties that need to be saved

Asayre does most of the save strategy stuff, so ideas, if valid, would do the most good sent to him for evaluation
 

Peoharen

Sorceror
I don't recall saying lighting or hue should be removed, though I wonder if handling lighting differently than it is might be viable.

And also, you should look in Data\Decoration, everything in those folders that isn't a door or container added there is the stuff OSI didn't even care to freeze to the map files and RunUO has been forced to include. Now think of your average shard's custom deco they added on top of that. Unless they know exactly what they are doing, likely ALL of it is unfrozen.

Is Asayre still around?
 

Jeff

Lord
Is Asayre still around?
A bit, yes...

If you were to create a world, with just the decor... how big is the save file? Few mb... if that? Really... statics arent the problem, it's the mobs and items people horde that are the problem.
 

tass23

Page
Jeff said:
If you were to create a world, with just the decor... how big is the save file? Few mb... if that? Really... statics arent the problem, it's the mobs and items people horde that are the problem.

I almost didn't wanna get involved in this with Jeff and Peo duking it out back and forth, but I will have to say I agree with Jeff on this, not to say I disagree entirely with Peo either. I noticed, before I even added mobs to my shard that it was a rather small backup, I think it was around 9MB. granted, I don't have a ton of statics, but I did create a lot of dungeons, and no, peo's right, I had no idea what I was doing at first, so they aren't frozen, I'm getting there though. Adding in mobs on all facets; Fel, Tram, Malas, Ilsh, and Tokuno, plus my custom dungeons, custom mobs, etc, addons, blah, blah, I'm over 650MB. Big change from the 9MB originally.
 
I don't recall saying lighting or hue should be removed, though I wonder if handling lighting differently than it is might be viable.

And also, you should look in Data\Decoration, everything in those folders that isn't a door or container added there is the stuff OSI didn't even care to freeze to the map files and RunUO has been forced to include. Now think of your average shard's custom deco they added on top of that. Unless they know exactly what they are doing, likely ALL of it is unfrozen.

Is Asayre still around?
I was playing devil's advocate. First you complained about regular items, then statics. So I knocked the static issue down with a "Well in a perfect world this ..." Just pointing out that things can ALWAYS be improved, but that doesn't mean they are broken. Of course a bigger irony is that various issues have been brought up regarding saving, and noone has actually made, well anything.

Also the majority of items in decoration are not static at all. There's a few notable fixes yes, like a bunch of missing static water tiles near coasts where osi fucked them up majorly, but for the most part they are dynamic items with some level of interactivity. Thats not to say that players should fear having unfrozen statics for light building.

And yes, Asayre is the one who introduced the multiple saving strategies. He s more active in code than talking.
 

Zippy

Razor Creator
If you're seriously worried about save time, saving 12(!!) whole bytes per Item is not going to make a huge difference. Most modern hard drives can write at well over 100 MB/s sustained. Think about that, if you manage to cull all this information that you think is useless and save 100 (!!!) bytes per item, you'd shave a cool second off the save time of a large (1,000,000 items) world. Blink and you'll miss the savings.

All those properties you don't like are useful to a lot of people. Seriously, players on some shard bitched that a GM changed the base weapon damage, so the server software used by hundreds of people should remove the ability to do that? Are you insane? The fact is tons of people download RunUO and set up a default world with their friends to screw around and the *first* thing everyone does is create a sword that slays balrons in one swing.

If you're seriously interested in decreasing the save time you should start by doing a serious measurement of the performance of the world save code. Find something that does line-by-line and/or instruction-by-instruction metrics, and see what is actually taking the most time. That's how you solve this problem, not by guessing. My guess is you'll have trouble locating the source of the problem even then. That's because my guess is things are already pretty well optimized.

No one really knows how the 'real' servers handle world saves. But we can make some educated guesses: they probably use a large commercial (expensive) database solution. Probably it has some sort of neat transactional features that allow it to take a "snapshot" of the data at any moment of time and write it out to disk in the background.

"Snapshots" like that can actually be *really* easy to implement on a non-Windows operating system because of simple OS support for copy-on-write pages. You can fork() a copy of the server and have on copy save while the other goes on running. The two become separate processes by at the OS level they share the same physical memory until one of them tries to change it, and then the OS makes a copy for each one. It's too bad Windows doesn't have anything like fork(), because it would be truely trivial to implement a 0 second save time in that case.
 

Peoharen

Sorceror
A bit, yes...
If you were to create a world, with just the decor... how big is the save file? Few mb... if that? Really... statics arent the problem, it's the mobs and items people horde that are the problem.
Well it also helps overhead. Take the damage Min/Max thing its self, it's too less integers in the memory per horded weapon and people go nuts over house deco and addons which can be reduced on the server side.

I almost didn't wanna get involved in this with Jeff and Peo duking it out back and forth, but I will have to say I agree with Jeff on this, not to say I disagree entirely with Peo either. I noticed, before I even added mobs to my shard that it was a rather small backup, I think it was around 9MB. granted, I don't have a ton of statics, but I did create a lot of dungeons, and no, peo's right, I had no idea what I was doing at first, so they aren't frozen, I'm getting there though. Adding in mobs on all facets; Fel, Tram, Malas, Ilsh, and Tokuno, plus my custom dungeons, custom mobs, etc, addons, blah, blah, I'm over 650MB. Big change from the 9MB originally.
It's less of a duking as what is it worth doing or not.

I highly suggests you freeze some stuff and look into a spawner package. Despawning cuts the number of mobiles down by a lot, both in saves and in your world. Hmm, isn't my comments on not saving monsters essentially a step even further?

I was playing devil's advocate. ... Also the majority of items in decoration are not static at all. There's a few notable fixes yes, like a bunch of missing static water tiles near coasts where osi fucked them up majorly, but for the most part they are dynamic items with some level of interactivity. Thats not to say that players should fear having unfrozen statics for light building.
A few notable fixes?
Wiped State: Mobiles: 1, Items 10.
Decorated: Mobiles 4092, Items 107,628.
Using [global interface where static the menu has 1,685 pages, each page lists 15 items, default RunUO world has around 25,275 statics (you know, those uninteractive items).
The word few is horribly inaccurate, did you mean one quarter of the items in game prior to player hording?

And I'm not even running the newest SVN (try last year's) here, how many people have worked on decorating Tur Mur out? How many have throw in some custom decro? How many have players have build houses? How many addons are there? You know, I'm not an idiot making crap up under the excuse of playing devil's advocate with smart assed quips of "well you haven't made it yet *sneer*". I'm genuinely submitting an idea that cuts memory consumption and save file size down and taking input on what could be trimmed or not.

Maybe you don't need to handle light in decroation items and instead just use the default for what it should be.
Maybe you really do need container information since you could handle single item decorations without having to result to an addon.
Maybe it would really be to hard to separate loot from precoded artifacts, but the static concept holds merit.
Things like that are helpful.
 

Peoharen

Sorceror
If you're seriously worried about save time, saving 12(!!) whole bytes per Item is not going to make a huge difference. Most modern hard drives can write at well over 100 MB/s sustained. Think about that, if you manage to cull all this information that you think is useless and save 100 (!!!) bytes per item, you'd shave a cool second off the save time of a large (1,000,000 items) world. Blink and you'll miss the savings.
Interesting point out (and not said in a bad way).

All those properties you don't like are useful to a lot of people. Seriously, players on some shard bitched that a GM changed the base weapon damage, so the server software used by hundreds of people should remove the ability to do that? Are you insane? The fact is tons of people download RunUO and set up a default world with their friends to screw around and the *first* thing everyone does is create a sword that slays balrons in one swing.
If fun of stabbing a Balron in one hit is so great, why not copy paste a weapon script and add a few zeros? Or idk, code one that you can prop the min/max damage on just for your GMs to entertain them selves with?

If you're seriously interested in decreasing the save time you should start by doing a serious measurement of the performance of the world save code. Find something that does line-by-line and/or instruction-by-instruction metrics, and see what is actually taking the most time. That's how you solve this problem, not by guessing. My guess is you'll have trouble locating the source of the problem even then. That's because my guess is things are already pretty well optimized.
Not by guessing, my guess is. Haha funny.

No one really knows how the 'real' servers handle world saves. But we can make some educated guesses: they probably use a large commercial (expensive) database solution. Probably it has some sort of neat transactional features that allow it to take a "snapshot" of the data at any moment of time and write it out to disk in the background.

"Snapshots" like that can actually be *really* easy to implement on a non-Windows operating system because of simple OS support for copy-on-write pages. You can fork() a copy of the server and have on copy save while the other goes on running. The two become separate processes by at the OS level they share the same physical memory until one of them tries to change it, and then the OS makes a copy for each one. It's too bad Windows doesn't have anything like fork(), because it would be truely trivial to implement a 0 second save time in that case.
Any chance someone has rolled out a fork() tool or you know of a project working on one for Windows?
 

tass23

Page
Zippy said:
No one really knows how the 'real' servers handle world saves. But we can make some educated guesses: they probably use a large commercial (expensive) database solution. Probably it has some sort of neat transactional features that allow it to take a "snapshot" of the data at any moment of time and write it out to disk in the background.
Jeff and I were talking about this in another thread awhile back. Since OSI shards (why are we still calling them that btw? lol) don't have "custom" items, a DB solution would make perfect sense. Also they have multiple servers running single facets, with dungeons tied to separate servers as well (Or they used to when they had money to run it all). Crossing server lines was a pain in the ass back when...you could never see what was on the other side of that line. An expanding DB solution could be applied to RunUO shards as well, but it would require the server owners to have very good DB knowledge.

Peoharen said:
It's less of a duking as what is it worth doing or not.

I highly suggests you freeze some stuff and look into a spawner package. Despawning cuts the number of mobiles down by a lot, both in saves and in your world. Hmm, isn't my comments on not saving monsters essentially a step even further?
Yeah, I'm using xmlspawner and I've made some custom spawns from base scripts using it. The xml attachment feature is wonderful. I had MegaSpawner and Premium Spawner at first, but I didn't like the way the spawners were accessed and the features of xml far outweight anything else I've seen. I'm by no means an expert at RunUO, but I've learned a few important shortcuts from forum threads here and over at Orbs too. Without a support system like the forums, no one would ever get a shard off the ground lol
 

duponthigh

Sorceror
Sorry if this is not the right spot but i have been reading up on all this db's and sql stuff.A couple questions to clairfiy if i freeze like 20 champ rooms 100x100 redTiles on my map0.Does this cut the item count down on the server?Second I have not started to work with my shard cause i have been reading about db's.Is it possible at the moment to use sql client, sql db, .Net db to be able to add unlimited items in the game?Cause i know there are very little free slots, that are free an able to be used.So i dont want to delete any items ever :) love uo.Would love to build on a massive db of art and then build my game around all these fun items.Also If so are db's limited in what they can do ?NOT talken about save's cause i cant complaine bout a couple sec.I think most people need to take a dang break anyways.If I am able to use db's in such as art.mul am i able to use db's to drop custom scripts that i make or aquire in a live shard.And be able to call it up in the game and use it without restart?If somone could let me know this.I would be greatful.I just dont want to be limited to those slots for art,mobiles,etc even if i had to buy some more computers to host Computer 1/art sql , Computer2/ scripts sql.Guys you are the best i just hope to be able to give back.I have learned so much.And i dont even have a highschool deplomia failed math.You guys it look easy.Hats Off to all contributiors
THANKS
 
Top