Ultima Live - In Game Map Editing Framework

[Praxiiz]

Just thought I'd give a status update. I spent a lot of time trying to get the client to reliably memory map new maps into its address space. While I was moderately successful in my attempts, it wasn't where I wanted it to be in terms of reliability and stability. After many loads and switches of maps, the client memory space would eventually fragment and the client/razor would crash because of it. At that point, I realized that even if I did get it stable, once EA released another expansion or map, I would be fighting the address space again.

So I decided that a new paradigm was needed. If I couldn't reliably unload and reload big maps and their statics into the memory space over and over again, the only option was to map memory space once (which is normally how memory mapped files are handled) and then swap data out by reading it from disk and writing it to memory. The problem with this is that the client would spend a lot of time reading from a map and writing it to disk. Map0.mul and map1.mul are almost 90 MB each, and if a player swapped maps frequently, it would cause delays especially on slow systems. They would not only have to load the map files, but the statics too.

The solution that I came up with is to load a 1032 x 1032 section of the map at one time. This keeps the amount of data being read from the hard drive down to under 5 mb per map#.mul file, which I think is okay. When a player gets close to the edge of the 1032 x 1032 section, the client loads a new 1032 x 1032 region into memory centered around the player. The result is that the player will have to move close to 500 tiles on a map before it loads a region. Tiles already loaded will be cached of course, which means that the client will have to read even less data from disk. If a player changes maps frequently, the loading delay will be quite small.

Now this only affects additional maps of course. The regular EA maps will remain in memory and will work like they always have.

The other piece of good news is that since I'm not trying to load an entire map at once into the memory space, (in fact now I'm only loading close to 20 megs total), the maximum map sizes should increase to 32k x 32k. It will, however, put a restriction on the minimum map size, at least for now. The minimum map size will be 1032 x 1032.

The biggest trade off will be in terms of the development of the system. Now instead of just loading an auxiliary map and letting the client operate normally, I will have to use razor to catch each packet and translate any map coordinates into coordinates relative to the clients view of the map.

So far, switching to additional maps seems to be stable. I still need to develop some of the packet algorithms (razor takes care of most of this for me). Once I finish those, I should be able to do another release.

 

[Thagoras]

I am SO happy you're still working on this! You haven't posted anything about this in a while and I was getting worried you were losing interest in the project. I recall when UO Cartographer came out how solid it was...then the author dropped the project just when it was so close to being exactly what we all wanted.

I'm so excited about the auxiliary maps...even more than about the in game editing (which is still freaking awesome!).

 

[Warstone]

Praxiiz,

Best practice I'm searching so far:
1) Split "cached" map space to 9 squares (1 square 256*256 tiles).
2) When player get to center of "corner" square - reload opposite square with new data (So current square will be center).

Such tequniqe will give you smooth updating. Even more - you can load that square in different thread, so player won't see any lag.

About 32K*32K - maps... Our investigations showed that if player cross Y ~ 10K(this digit is floating even on same computer) - radar won't show statics. I didn't find - where it is, but it's true.

BTW: You need to process another thing... I'm talking about "dungeons area", When you reach the right border of the main map(not system, but logic... Britannia sea) client will load opposite side of the map. So you need to check this out.

 

[Pure Insanity]

I thought map wrapping was all done by the server...still nice to see people trying to make headway on this. But starting to think it may be easier to just build a new client. =P

 

[Praxiiz]

I thought map wrapping was all done by the server

Wrapping player movement around the map is done server side. When you get near the edge of the map, however, it is the client's responsibility to display tiles from the opposite side of the map. In this way you never actually see the edge of the map. It appears continuous. My system isn't really doing anything with wrapping map (server side or client side).

My goal is to allow additional maps on the client side besides the regular OSI maps. I can already do this reliably and fast if the Auxiliary maps are small (close to 1000x1000).

 

[Praxiiz]

To allow larger maps, all I'm doing is telling the client that it has a small map, and then loading a portion of the larger map into the client's smaller map. When you get close to the edge of that smaller map, I simply load more of the larger map.

It has the effect of allowing the client to use larger maps, but gives it the stability of the smaller maps.

 

[Praxiiz]

Best practice I'm searching so far:
1) Split "cached" map space to 9 squares (1 square 256*256 tiles).
2) When player get to center of "corner" square - reload opposite square with new data (So current square will be center).

Such tequniqe will give you smooth updating. Even more - you can load that square in different thread, so player won't see any lag.

About 32K*32K - maps... Our investigations showed that if player cross Y ~ 10K(this digit is floating even on same computer) - radar won't show statics. I didn't find - where it is, but it's true.

BTW: You need to process another thing... I'm talking about "dungeons area", When you reach the right border of the main map(not system, but logic... Britannia sea) client will load opposite side of the map. So you need to check this out.

A separate thread would be very efficient, it would introduce some concurrency issues, but they wouldn't be too bad. I don't think the radar will be a problem because the client has no knowledge of the map beyond 1032 tiles. It will never see a location beyond 1032. When the server sends a location that is greater than 1032, it gets translated into a coordinate that is lower than 1032, but corresponds to the location in the larger map.

 

[Praxiiz]

Of course the most simple option would be to just say that all maps have to be smaller than 1032, and just leave it as is.

 

[Sythen]

this all sounds awesome in theory but what I think would be cool is to allow us to visit a demo shard where this is active. My fear is that maps wont be wrap around, but rather regional... like the old pen and paper style dungeons and dragons maps. Otherwise I am looking forward to the release because I love map making and would enjoy squashing my negativity toward not being able to add as many maps as I want and keep them as active worlds rather than cycling through them... right now I have tram and fel and the rest are for dungeons and events lol.

 

[duponthigh]

I cant wait till you get the next release out . This is the most awsome thing that i have seen in a long time.Please dnt give up , your doing such a great job.Thanks for your hard work and time you ahve put in this.Also thank you for the update we all apperciate it very kind sir *Hugs*

 

[Praxiiz]

So far the system seems to work well. I have some corner cases I need to fix and then I'll do an experimental release to see if anyone else has issues with it.

 

[Nockar]

Nice.

 

[clark71822]

Glad to hear this is still being worked on. Keep up the good work Praxiiz

 

[Warstone]

To allow larger maps, all I'm doing is telling the client that it has a small map, and then loading a portion of the larger map into the client's smaller map. When you get close to the edge of that smaller map, I simply load more of the larger map.

It has the effect of allowing the client to use larger maps, but gives it the stability of the smaller maps.

This way will hit on script writers. I'm using EasyUO for automatic mining/chopping... This system caches all gathered data from Terrain analyzing (including pathfinding, dynamic threats, etc.), if you will yse this way - some scripts must be rewritten, or even can't be used with your way of larger maps. To avoid it - you need to change system values in client executable. It's easy to find, because there is only one pointer to map name in client (You need to search "Trammel" string in exe, get offset from beginning of exe, add 40000(NT_IMAGE_BASE from NTHeader) revert it and find that number in executable. There only one place you'll find in exe, next to it there will be dimensions of map and some other flags. More of it you can read here, but it's on russian). So... You need to change only 8 bytes in exe to make map bigger. If you want to make it in dynamic - you need to get map dimensions from server before you allow execution fo main executable code, and rewrite it directly in memory. There even more strange approach may be needed... Shadow memory thru 0-ring like it was done in WoW bot to avoid ther Warden code.

A separate thread would be very efficient, it would introduce some concurrency issues, but they wouldn't be too bad.

Just open new connection to server and read all data from there )))

this all sounds awesome in theory but what I think would be cool is to allow us to visit a demo shard where this is active. My fear is that maps wont be wrap around, but rather regional... like the old pen and paper style dungeons and dragons maps. Otherwise I am looking forward to the release because I love map making and would enjoy squashing my negativity toward not being able to add as many maps as I want and keep them as active worlds rather than cycling through them... right now I have tram and fel and the rest are for dungeons and events lol.

You can try to do it by yourself Sorry for ad, but... I'm writing (right now it suspended because of real life) client extension system that will allow to make same things in a plugin manner (not only this one, but new gaphics, new mechnics, etc...). You can get it here (Yes... It's Delphi).

 

[Praxiiz]

This way will hit on script writers. I'm using EasyUO for automatic mining/chopping... This system caches all gathered data from Terrain analyzing (including pathfinding, dynamic threats, etc.), if you will yse this way - some scripts must be rewritten, or even can't be used with your way of larger maps. To avoid it - you need to change system values in client executable. It's easy to find, because there is only one pointer to map name in client (You need to search "Trammel" string in exe, get offset from beginning of exe, add 40000(NT_IMAGE_BASE from NTHeader) revert it and find that number in executable. There only one place you'll find in exe, next to it there will be dimensions of map and some other flags. More of it you can read here, but it's on russian). So... You need to change only 8 bytes in exe to make map bigger. If you want to make it in dynamic - you need to get map dimensions from server before you allow execution fo main executable code, and rewrite it directly in memory. There even more strange approach may be needed... Shadow memory thru 0-ring like it was done in WoW bot to avoid ther Warden code.

No need to do this before the executable is run. Instead of searching for the word trammel, I just search for trammel dimensions after the executable is loaded. I find the dimensions of trammel and the wrap dimensions of trammel in the next 8 bytes. Then I can adjust the map size for all the maps while the client is running.

Even before the client is loaded, I make a collection of the base pointers for each map. (Since I'm hooking the file view function calls, I know what addresses to search for.) These two things allow me to alter the size of any of the maps while the client is running. I can make them bigger smaller, or even swap them out.

Originally I was making the auxiliary maps larger and mapping their files into the address space of the client. This caused problems later when the client was running because I would have to unmap one auxiliary map in order to map another. Eventually the address space of the client would become fragmented and it would start throwing out of memory exceptions.

To avoid this entire problem, I only map a small view of the file, and I just copy the data into the view when I need to. Yes, this will break some scripts that depend on exact positioning within the map. But it will only happen on the additional maps, not the original OSI maps.

The result is that it can support very large maps, while preserving the address space of the game client. It also can support over 200 additional maps.

This screenshot shows map32 loaded into the client. If you look at the map radar, you can see that the system is loading the blocks around the client as I have been moving around. The full map is on the server, but the client just has a blank map that it fills as I move around.

I could also load a blank map on the server end and just start decorating it.

I can switch between trammel, felucca, ilshenar, malas, tokuno, ter mur, and map 32 without any problems. If I added more maps to the definition, I could switch between them as well.

 

[Thagoras]

Yes, this will break some scripts that depend on exact positioning within the map.

How, exactly do you mean?

The full map is on the server, but the client just has a blank map that it fills as I move around.

This probably won't be an issue. I kinda like that unexplored feeling. Ever play Civilization? However, once the data is updated on the client, does it show in the radar when you go back to those locations? I mean, once you've been there, it acts normally right?

 

[Praxiiz]

What I mean is that it may break some client side scripts in Easy UO. Since the client only sees a map that is 1024x1024, easyuo scripts won't have true coordinates while they're on the auxiliary maps. This can be overcome, but it requires players to be creative when they write their easy uo scripts.

It doesn't affect server side scripts.

 

[Praxiiz]

Also - the screenshot I was showing was just one way of using the system. If you actually do distribute your maps, then players will just load the map and it will appear normal.

 

[Thagoras]

It doesn't affect server side scripts.

That's all I was confused/concerned about.

Nobody on my shard uses easyuo...but I actually like the sound of it being broken...er...require players to be creative to get it to work.

 

[Sythen]

how is what you're doing any different than these:

http://www.runuo.com/community/threads/general-map-question.94606/#post-796943
http://www.runuo.com/community/threads/regions-custom-maps-and-moongates.87435/

from my own experience when you add more facets to uo, you can only clone facets already on the server because the clones will "share" map files; that doesn't mean you can't add differences in-game... but you can't use a program like CentrED, make changes, and expect the changes don't show up on each facet that uses the map you changed - because it will. What I did to combat this issue was make an entire map of generic dungeons as a replacement world for malas, by duplicating this map several times I was able to have an endless amount of dungeons i could add into the game. Would they be identical? yes, but it isn't that noticable unless you're looking for it since each of those facets are deco'd differently. Anyway thats just my 2 cents, I'm really trying to understand what you're doing because your theory sounds interesting, but it goes against everything I've come to know about custom maps and the client and I've been at this for years.

Here is what I've learned for myself:
(1) the client has 6 maps hard coded in it.
(2) In order to add new facets you must edit both the map.cs (console mod) and the mapdefinitions.cs (script mod)
(3) all new facets added have to have a value or id of a map already hardcoded in the client and be registered as Map32.mul, Map33.mul, Map34.mul etc...
(4) these new facets will emulate the facets they take their id's from
(5) to change the facets, one can only deco from within the game because editing using a 3rd party editor will make changes to all facets sharing the same map id.

Anyway... confused as usual :/