Connectivity Issues April 1, 2017 (Saturday Morning)

No I actually loved reading it, I miss all they tecky convos.. thank you!

 

In the gig before my last gig, I was at a medium sized company (150 vm environment). I actually fired up the VMware Airwatch (called the hybrid cloud back then) (AWS too, just lesser so) for it, migrated 25% of my infrastructure up there. This was just after release and it turned out, I ended up being one of their "beta testers". I say that because it was not ready for release and I spent a year on the phone with the VMware engineers trying to get their stuff to work, saying that, they have some a long way and now actually use AWS on their back end. I think the cloud has its uses but from now on, Ill stick to a local on premise cloud. I still remember even office 365 going down for 8 hours a while back. It happens. Now I dont have any issues really with what you and Port are doing, largely because this is the first outage I have seen, which means you are doing your job. I mainly responded to your VMware comment largely since its something I have never heard utter by mid to high level system/network/datacenter administrators. The low ones usually say "ya, we use hyperV because its not "Linux" (that one drops me to the floor) or "we use it because its what we are used to".

 

So the next time someone whines about horse statue costs on the add-on store vs the license and asks if everyone is ok with it, we should ask how many hours we want to wait when the service is down instead.

 

Around 80% of mid sized business's are viritualized. Of them, 82% of them are running VMware and 38% of them are running hyperV. 3% are using "other" which is sounds like you guys are a part of. Point being, this does define "industry standard" and its really really odd to hear about companies not going with the things that actually work in the enterprise market (in tiny to epically huge environments)

I'd bet you guys can run everything on 2 latest gen server filled full of ram, you can buy VMware packs to run full HA for less then $5K (vSphere Essentials).

Tell Starr to start diverting all my pledge money to some vmware licenses hahahahah!!!

 

<delete>

 

Thanks for the quick response and relatively quick recovery. Every MMO and Internet service suffers downtime.

 

Several UO shards that I played in the past used dedicated servers over VMs due to some weird network issue that throws lag/rubber effects when the number of connected clients is getting high.

Does SotA use VMs? If so, could it contribute to lag we've been getting when we go into the place with lots of dynamic item loads?

Just curious about the hardware architecture here.

 

A few hours since persistence, for a game "in development"? Aren't we supposed to be still testing/fixing not assuming perfection? A solution was found quickly, in the off hours.

Looked at the right way, this was a very good thing to happen. It would be silly to offer anything other than encouragement (and for those who can, technical ideas) to everyone involved here. Keep up the good work.

 

I think this problem happened at the perfect time. I was not even at my computer during the entire outage so I didn't even know about it until the next day!

Still, for it to happen on a weekend at this stage of development... and get fixed before the following Monday (actually before the next day) is very promising to me. I have played other games that were already "released" and a server crash on a Friday night meant don't bother trying until Monday.

So kudos to the team for making this incident virtually unknow to me.

 

The servers being virtualized wont really effect those things unless configured pretty poorly. At this point in the world, most servers are virtualized (80%). This goes for massive servers hosting everything you see on the internet from small to huge sites.

I can explain it best like this:

Computer hardware has far exceeded operating systems and software. 10 years ago you would buy one hardware server per server (say Windows server 2012 with is a derivative of Win7 or server 2016 which is a derivative of Win10). But then 64 bit computing came around and lifted the 4 gig ram limits.

The last server I built had 32 cores and 2 TB of ram and could run 100's of virtual servers on it and barely crack 10% cpu utilization (it could run 1000 windows 7 VM's easily on it if we wanted it to).

What people dont realize is the same is true for home hardware with the exception of video cards which are still highly utilized in games.

This i7 Skylake I run right now could run dozens of win7 machines (ram dependent) all at one time and barely crack 10% cput utilization. If those machines were just doing office kind of work, internet browsing and email.

The servers in this case and Win7 VM's dont even know they arent really on physical hardware (well, they are smarter about it now, in fact designed for it), they have for example vmware drivers for all of their virtual hardware. Machines that are virutalized also run far better then their counterparts because of this, in a VMware environment, there is hundreds of millions of servers out there running the same extremely optimized drivers which means they are extremely efficient and its one of the reasons why you get huge up times. You go from have a 20 gig windows install on a drive that uses drivers attached to the hardware (flawed) to VMware esxi on the server which is a 250 MEG install with the virtual severs using these really efficient drivers. In a virtual environment, the only time you need to reboot a server is for patched, 1 year is like normal and they very rarely crash. If they crash, the main point of using virtualization is that at the point, all the virtual servers pick up and immediately move to other physical servers running in the cluster. 10 years go, this was like magic.

So that takes me to why I asked them if High availability kicked in and all the SOTA virtual server migrate to a physical server thats still running.

 

+1 great idea!! To accomplish this you could add layering deeds to the add-on store that allow players to change the slot which their masks go in to.

For example, I could change the slot of my plain carnival mask to the cloak slot and then wear an epic cotton hood over it! That would look amazing I think.

Only 1 DR masks would work with the layering deeds, of course, since cloaks always only give 1 DR as well, I think. Not quite sure.

Just make sure that whatever is in the cloak slot would go underneath whatever is in the head slot and it should be just fine.

A great addition to artistic freedom I should think.

 

You mean a third floppy drive, they already have two in the lobby.

 

This is when its great to have DRS on the cluster and have vROPS installed so that one rouge VM cant take down all the VM's on one server. Shared GPU's are ok, but pretty limited, we use them now in a 250 win7 VM Horizon build. The cruddy thing is that Nvidia now wants to charge per use licensing fees to use these cards. vRops gives you every and any piece of information you could ever imagine on the VM's in the cluster over time. Having vROPS installed allows you to proactively watch for major issues in hardware. For example, I used vrops to prove that one of our massive SQL servers that the SQL team built wrong was pegging the VM so hard at times that it was sitting at 120 ms latency to storage and as well was causing the other 70 VMs on its data store to have 80+. Very powerful stuff.

 

The network seems to be down again (Thursday 10PM CEST)