RunUO Community

This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

Quicker Save Times (Discussion/Etc)

MarciXs

Sorceror
Yeah it will cut down the items if they are saved as statics.
I don't quite follow what you mean with db's and art.mul. I don't see the link.

As for topic, it most likely is done before, but if you want to speed up the saving just save to memorystream and have a snail saver running in background that saves it to a file. Would need more RAM though.

Oh and also you could split Items in groups(per 1024 or whatever). You know like you had Items.bin.part1 etc. Thus the idea you had two or more savers running at the same time.
 

Jeff

Lord
As for topic, it most likely is done before, but if you want to speed up the saving just save to memorystream and have a snail saver running in background that saves it to a file. Would need more RAM though.

Oh and also you could split Items in groups(per 1024 or whatever). You know like you had Items.bin.part1 etc. Thus the idea you had two or more savers running at the same time.
http://svn.runuo.com/repos/runuo/devel/Server/Persistence/DynamicSaveStrategy.cs this already does that..
 

duponthigh

Sorceror
Ok thank you guys first and 4most for reply.Jeff Does this script let you basically extend the art.mul file?Really let me put this better.There is only limited amount of slots for custoum art.
I want as much as my heart desire's no fun to have a cap.So i just want to know how to extend the files that are limited to slots.Such as art.mul.So possibley gumps.mul too?As for saving dont care about that.Speed for that not a concern.Just limit :)
 

Pure Insanity

Sorceror
If there's a way to get the client to reload the art files, it may be possible to extend it like how Praxiiz plans on extending the amount of maps.

And that "script" just increases your save times basically.
 

Peoharen

Sorceror
Wasn't there once a hack posted for the client to support extra maps using a standard hexeditor? (might be thinking something else).

Maybe you can do the same for Gump IDs, hex edit a new limits.
Of course, editing the client is illegal. So yeah, thinking less of seeing the client hacked and more like some other program I use...
 

Acronis

Sorceror
I have recently released a patch for my shard doing what Jeff mentioned earlier. It took me 3 years to do it but mostly because it was never good enough for my tastes and I kept rewriting it. The final rewrite I planned better, and was dead simple and took like a month to do. The DB structure is not really proper SQL, but the goal was to separate each object into self contained sets that can be updated individually, and not normalize it. I was actually going to make separate text files at one point but it seemed faster and better to use SQL.

To do it properly using a normalized DB would pretty much require an entire rewrite of RunUO and rethink everything. Because of the vast amount of item types and the fact that they have various fields, it makes it really tricky data to work with from a DB perspective. I had lot of ideas but at the end it just seemed to make sense to keep serialization, but instead of doing it to a file, just do it to a DB. It was quite a big project, but mostly because of the research and development part. Once I finally found the gold, it was not that bad.

I may potentially release my code, but I honestly don't remember every single edit. It's quite large and not very plug and play. One of these days I'd have to get the latest SVN and see how easy I can fit it in so I can make a tutorial on implementing it. There are tons of different ways to approach it, but this was one of them.

If I was to do it over again, I'd possibly consider some kind of delta file system instead. Instead of world saves, there would be delta saves. Just write the changes since the last delta which would take maybe 100ms or so and not be noticeable. This delta file would just accumulate all the changes. Then once an hour or whatever, the delta would be merged to the load file somehow, or perhaps just do a full save. So rather than a save every 15 minutes, it would be every hour. Maybe even once a day. If the shard crashes, it would load from the load file (that could be a day old) then apply all the delta data. But this could be quite long on a big shard. Would be worth experimenting with though.
 

fdsprod

Wanderer
Delta file system isn't the way to go. The best way is sql, but your sql engine should generate tables on server load. Verify the schema, then load the data. This should make it so each property maps to a column and reduce the amount of sql you have to write. This would also always ensure a normalized database. The hard part would be adding some sort of ColumnAttribute to each property or field that requires saving. Delta saving is not backward compatible, and if you were to make it that way, you would just be making the save larger then need be. I'm still amazed people have issues with saves. The way hardware is, I cannot imagine a regular server with 50+ players taking more then a second or two to save, even then, you should only save 1 time an hour. So, when you think about it, you are saving yourself what, 2 minutes out of every 24 hours? Are the amount of hours that would go into a system like this even worth that kind of gain? Not to mention the new DynamicSaveStrategy saves in parallel to the server running, so save times are even shorter...
 

Pure Insanity

Sorceror
I think it would be rather stupid to store and save items in a mysql database. Accounts on the other hand would be a pretty smart idea, servers could even share accounts this way. Or allow you to easily ban/add accounts from your site securely. My saves atm are 0.2 seconds, and item/mobile count isn't exactly low. I don't see the point in going through all that for a whole 0.2 seconds (and I'm only using dual save with 2 cores.)

If you move it mysql, you'll need a way of doing some type of versioning. Plus I really don't see the point, as the memory and cpu overhead from running a mysql server (unless you have one running already) isn't worth what you'd save by getting rid of the way runuo saves mobiles/items. But to each their own I guess...
 

MarciXs

Sorceror
I have recently released a patch for my shard doing what Jeff mentioned earlier. It took me 3 years to do it but mostly because it was never good enough for my tastes and I kept rewriting it. The final rewrite I planned better, and was dead simple and took like a month to do. The DB structure is not really proper SQL, but the goal was to separate each object into self contained sets that can be updated individually, and not normalize it. I was actually going to make separate text files at one point but it seemed faster and better to use SQL.

To do it properly using a normalized DB would pretty much require an entire rewrite of RunUO and rethink everything. Because of the vast amount of item types and the fact that they have various fields, it makes it really tricky data to work with from a DB perspective. I had lot of ideas but at the end it just seemed to make sense to keep serialization, but instead of doing it to a file, just do it to a DB. It was quite a big project, but mostly because of the research and development part. Once I finally found the gold, it was not that bad.

I may potentially release my code, but I honestly don't remember every single edit. It's quite large and not very plug and play. One of these days I'd have to get the latest SVN and see how easy I can fit it in so I can make a tutorial on implementing it. There are tons of different ways to approach it, but this was one of them.

If I was to do it over again, I'd possibly consider some kind of delta file system instead. Instead of world saves, there would be delta saves. Just write the changes since the last delta which would take maybe 100ms or so and not be noticeable. This delta file would just accumulate all the changes. Then once an hour or whatever, the delta would be merged to the load file somehow, or perhaps just do a full save. So rather than a save every 15 minutes, it would be every hour. Maybe even once a day. If the shard crashes, it would load from the load file (that could be a day old) then apply all the delta data. But this could be quite long on a big shard. Would be worth experimenting with though.

Why can't you just write it as BLOB? I still don't quite understand the need for your saves to be located on MySQL. There's no gain in it. And the way RunUo save system is built it actually should be quite easy to write them in MySQl db...
 

Pure Insanity

Sorceror
With the added overhead, for 0.2 seconds a few times a day. I see no gain at all. Only gain I could see from mysql, is moving the accounts to mysql instead of the ole xml file.
 

Acronis

Sorceror
Why can't you just write it as BLOB? I still don't quite understand the need for your saves to be located on MySQL. There's no gain in it. And the way RunUo save system is built it actually should be quite easy to write them in MySQl db...

I ended up using longtext, but blob could probably work too. I can't recall why I did not use it, maybe it was performance.

My saves were getting around 10 seconds each, and every 15 minutes, so it adds up. Was well worth it in the end. The beauty now with this system is I do not have to worry about item count at all other than the 32 bit limit, which I'm far from. I could even go a step ahead and have a separate server for SQL maybe even two and use SQL replication for redundancy. Then have a standby UO server. It opens up lot of options depending what extreme I want to go.

For the versioning, it is still the same as before as the data format has not really changed, it's still serialization. Right now if I wanted I could actually go back and load a backup from the old system, though I will be phasing out that code soon and resetting all the versions to 0 to cleanup the serialize code. But yeah, I can still add versions if I want to, so I can add more fields and what not.
 

MarciXs

Sorceror
I ended up using longtext, but blob could probably work too. I can't recall why I did not use it, maybe it was performance.

My saves were getting around 10 seconds each, and every 15 minutes, so it adds up. Was well worth it in the end. The beauty now with this system is I do not have to worry about item count at all other than the 32 bit limit, which I'm far from. I could even go a step ahead and have a separate server for SQL maybe even two and use SQL replication for redundancy. Then have a standby UO server. It opens up lot of options depending what extreme I want to go.

For the versioning, it is still the same as before as the data format has not really changed, it's still serialization. Right now if I wanted I could actually go back and load a backup from the old system, though I will be phasing out that code soon and resetting all the versions to 0 to cleanup the serialize code. But yeah, I can still add versions if I want to, so I can add more fields and what not.

What about the actual saves though? I mean does sending all the data to the MySQl db really is faster than writing it to a hard drive?
 

Acronis

Sorceror
It's not really faster due to the overhead involved, but because only changes are saved, the entire world is not sent each time. The actual SQL writes also happen in another thread. This way there is no interruption to the users. The data serialization happens in real time, but these saves take about a fraction of a second and aren't noticeable. It will save up to 2500 objects at a time. Once these are saved to a temp memory location then the SQL process starts, which takes 1-2 seconds in another thread, and the cycle repeats. My server is also taxed pretty badly due to other stuff on it, so I need to shell out more cash per month and upgrade, or just save up and build one I can colo. Been debating on the best route. Colo I would save in the long run.

So it's definitely more overhead, but it's less invasive and there's lot of other perks like easier backups etc.
 

Pure Insanity

Sorceror
Easy backups...Right click save folder, click add to archive. Upload to ftp server, and email to yourself. Pretty damn easy. Some people just like to complicate things...and call it new and ground breaking. :/
 

Acronis

Sorceror
I meant automated backups. mysqldump FTW. I have two local backup jobs per day and one offsite per day. I can also do an ondemand backup that syncs to TC1 or Dev within 10 minutes if I want to. I did do this with the old system, but it was more touchy. The real point was not really easier backups though, but real time saving.
 

MarciXs

Sorceror
after taking a look at how easy it would be to change save strategy during runtime, I was rather disappointed. Because it isn't possible... due to fact that save strategy is obtained before each save(which would be fine) but it doesn't obtain it from a field or member, instead it is a method that checks the following:
Is our processor a multi core and the amount of cores it has... I might be wrong but can you actually modify processor on a running server? And then depending on that it would return a save strategy ... am I the only one who thinks this is weird?
Would really like to see either the method taking the SaveStrategy from a field or better yet having this field added to the Core that it would use...
Thoughts?
 

Vorspire

Knight
.2 seconds an hour, or every few hours is nothing worth doing an overhaul like that. Just don't see the point is all.

Yea, 0.2 seconds an hour... for YOU, however successful shards with a large player-base would greatly benefit from an unobtrusive save strategy, just because YOU won't doesn't mean you should keep jamming your opinion about it down everyone's throat, if you've ran out of input for this discussion (which you clearly have, since you're repeating yourself) then maybe it's time to just leave it be...

Also for the record:

Easy backups...Right click save folder, click add to archive. Upload to ftp server, and email to yourself. Pretty damn easy. Some people just like to complicate things...and call it new and ground breaking. :/

This is totally ignorant, for one, not everyone has direct file-system access to their O/S at all times and manual back-ups can be easily corrupted, lost or even stolen.
A completely automated, comprehensive self-repairing and self-sustaining backup service is far, far more secure than trusting any human, including yourself, to keep track of and maintain those backups.
An easy backup would involve NO work at all, it would have been pre-determined by the aforementioned automated backup service.
Finding your root directory and manually selecting and archiving backups is way more time consuming and complicated than just letting a scheduler do it for you.

If you're going to upload back-ups to an FTP server, why would you bother e-mailing it to yourself too? Wouldn't it be more logical to just email the FTP file URL to yourself... I mean, sure emailing it to yourself gives you one extra backed-up file, but I think you forget just how many people don't have access to an advanced email account or one with enough quota to store files that can potentially be tens to hundreds of megabytes in size.

Some people complicate things by offering a solution without first thinking about what they are saying... and call it "easy".

************

For the record though, there is no point in storing RunUO world data in SQL, at the end of the day you're still writing the data to the HDD of the machine, stored in 2 or more files (depending on the DB engine used) in SQL's schema directory. - Couple this with the possibility of the SQL machine being a remote host, you have to add on top the file transfer time, but that would just be silly in the context of housing RunUO world saves. - Then there's the overhead of SQL data filtering and indexing, the overhead from the SQL API and the SQL service itself.

World of Warcraft emulator ArcEmu uses MySQL for EVERYTHING and it really SUCKS, it's hard to set-up, it's hard to maintain and for the best part, there is no real way to interface properly with the data other than to use the native methods provided, any attempt to directly manipulate the data in MySQL causes synchronization problems with the server itself and things start bugging up.

The whole hype of faster save times is getting silly IMO, if you really want to impact your save times, buy more RAM, increase your page file and buy an SSD that supports fast write speeds.
There are 3rd party drivers on the net that replace the standard Windows drivers for SSD I/O access, which actually improve performance of the SSD by a noticeable amount.
I used this technique when installing Win7 on a Netbook with an 8GB SSD and 512 MB of RAM, prior to upgrading to 3rd party drivers, Win7 was a snail, I saw at least a 70% improvement in write speeds from an SSD that has a low write speed rating.

I guess the one thing to take away from this seminar would be to upgrade your hardware if you think your save times are too slow, or at least wait until someone comes up with the perfect solution, which will probably not be any time soon.
I'm toying with the idea of a new serialization system that could seamlessly replace the current one, without a .NET upgrade, the theories are written, they are just yet to be tested and dis/proven, none of them reference SQL at all.

There are a LOT of awesome things that should be in RunUO, there are countless things that have been disregarded in the past that SHOULD have been added to RunUO, a lot of those things were fronted by RunUO Team members, but never made it to release, if people look very close, especially at threads started in the past 4 months, they will come to realise why awesome features such as faster saving times have not been implemented. I wouldn't expect to see an improved serialization framework in RunUO any time soon, probably never, unless this community stops dragging it's balls along the floor and steps up to the task at hand.

P.S; Just because I know that I can come across a bit cocky with lengthy posts like this, I'd like to point out that the above was written with absolutely no offensive intention, there are a lot of people around here that can take a wall of text at face value and presume it's offensive and because of that I gladly challenge anyone who has a contradictory opinion as I love a good, well-mannered debate without all the poop-flinging :)
 

Acronis

Sorceror
The point of the SQL is not really the speed, but the fact that you can separate data into logical units. Each item, mobile, guild and account is a row, and you can update/delete/add. TBH the SQL actually slows things down a bit, but because of the structure and being behind the scenes it does not matter.

My saves were approaching around 10-30secs and they were every 15 minutes to avoid the possibility of a huge roll back if the shard or OS crashed. Upgrading is not always an option especially when renting a dedicated server. Adding 1GB of ram can cost as much as 20 per month, a SSD would probably be like 100. When leasing, you are really limited in the power of the server, so you need to compromise other ways. At one point colo makes more sense but there's still the upfront cost of building the server.
 
Top