RunUO Community

This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

Outage notification 18 november 2012 - one long day...

Ryan

RunUO Founder
Staff member
As most of you are aware by now the entire RunUO network was offline for the better part of 12 hours yesterday. This post is an attempt to explain the issues we had, what it means for you as users and how we're planning to mitigate it from happening in the future.

At approximately 13:00 EST the entire network destabilized to the point it was virtually unusable, followed by a total and complete collapse around 15:00 EST.

The cause of this outage is now known, the service provider who runs our core infrastructure in Chicago had a complete meltdown of their Cisco 6509 Core switching platform. This issue required Cisco to come on site and provide a fix for a very serious software issue. This took nearly 12 hours to complete and at this point the network is again stable for the time being.

While this outage was completely out of our hands losing the entire network is completely unacceptable and cannot happen again in the future.

To keep this from happening again we are going to be separating the game server services and the web infrastructure while completely getting rid of the current provider. The original thought was to source another provider in Chicago to handle the game servers but our search was fruitless for a cost effective provider to meet our high demands/requirements. We've elected to move to the game servers to Dallas Texas.

While I recognize some of you may not like the choice of moving the game servers to Dallas we're choosing a world class facility run by Internap:

http://www.internap.com/colocation-provider-facility-overview/dallas-data-center/

This facility houses some of the biggest companies in the world and will provide us with a much more stable platform for UOGamers going forward. We have yet to determine where we will be moving the web sites to but will update you as soon as we have that information. For the time being they will stay in Chicago providing us redundancy in the event of a failure in the future.

Another interesting thing is that the facility (being Internap) utilizes the Internap Flow Control platform. This is widely regarded as the end-all-be-all of internet bandwidth optimization and can dynamically re-route or re-allocate routes to provide all of you with the best routing and latency possible worldwide. I'm providing the following test IP's for you to utilize but keep in mind the full effect will only be realized when we're online and you're an active IP in their network and the routing to you is fully optimized.

Test IPv4: 199.231.226.12

I recognize how painful these last 24 hours have been and I appreciate the amount of patience the vast majority of you have shown. This was not an easy time and the holidays always make for a very tight time for these larger un-budgeted items like an emergency move.

Thank you again for your support of RunUO and UOGamers and we look forward to many more years with you!

Update 1: Demise move is complete. World load time dropped by about 80% and save times are about 40% better. Please enjoy.

Update 2: Hybrid move is complete. World load time dropped by about 25% and save times are down about 30%! This is an amazing improvement.

Thank you for your support!
 

Gages

Squire
Ryan, it's great to hear the problems are being addressed. Also giving the playerbase feedback Is GREATLY appreciated. A little bit of a word from you guys goes a long with us.
 

Ryan

RunUO Founder
Staff member
Ryan, it's great to hear the problems are being addressed. Also giving the playerbase feedback Is GREATLY appreciated. A little bit of a word from you guys goes a long with us.

We recognize the importance of being able to share information in these events. This is why it's important to follow us on the following mediums:

www.twitter.com/ryanmcadams
www.twitter.com/runuo
www.facebook.com/groups/UOGamers
www.facebook.com/groups/RunUO
www.facebook.com/runuo

...as well as why we're splitting the infrastructure over multiple locations.

Thanks,
Ryan
 

Noraevia

Wanderer
Thank you so much for giving us the scoop Ryan. Its great to be in the loop about issues and whatnot! We, the players, truly appreciate it :D
 

svnstrms

Sorceror
Fantastic, I get better pings in Dallas than Chicago anyway! :D It will be nice to have low pings and stable connection for a change. The lag spikes make my PK sad face.
 

Ryan

RunUO Founder
Staff member


Just FYI, this is from the new server ;)
 

ScubaSean

Sorceror
Thank you Ryan, this move will cut my ping in half! Sucks for the last provider, woo for the new provider! Cutthroat business!
 

KalivDIESEL

Sorceror
sounds good.. im not mr geography soo will it be faster for me in newyork? lol... p.s ryan ull have more fun in newyork!!
 

ScubaSean

Sorceror
When moving, will there be a rollback? I'd guess that moving that amount of data would have a gap in service... But I don't know anything about moving data :p
 

Ryan

RunUO Founder
Staff member
When moving, will there be a rollback? I'd guess that moving that amount of data would have a gap in service... But I don't know anything about moving data :p

There would be a small time lapse, probably about 15 minutes. We'll shut down the current server and bring up the new one. We will keep you updated.
 
Wierd, that test ip you gave which I tested twice and my ping was actually worse. On the other sites you were contemplating before this new move it was around 50ish, this test ip gives an average of 80. Unless I misunderstood and that's actually chicago's ip, idk.
 
ill ping like 45 but i guess that is good.
Yeah, I appreciate the status update and move to a better facility. In the previous poll with those Texas cities I was pinging to it with around 50ish. For some reason this test ip is 30 ping higher and it's from the same state.
 
Top