Release 1 Postmortem and Breakdown

While not without its flaws, Release 1 was a huge success as a test run for the game. We had a long list of things we needed to do real world testing on and almost all passed with flying colors. For the casual gamer who doesn’t think much about what goes on behind the scenes, they hopefully patched, logged in, played for a while and said “Yep, looks like a great start!” What most non-game developers don’t know is the massive amount of work that went on behind the scenes to go from zero to that point in 8 months all while growing our team from miniscule to just tiny (for a AAA RPG).

Behind the scenes, the list of things that had to come together in a short amount of time was huge. The fact that most people didn’t notice was a testament to just how well things went overall. The incomplete list of things that got done includes:

• Installers had to be created
• Bootstrap system to patch the installer
• System to build and apply patches so we only had to send the differences (deltas) in the installed files
• A way to distribute them that wouldn’t kill our budget
• A backup distribution for those who are behind secure networks that don’t allow peer connections
• A publish process created to remove human error
• Process for multiple code and data sets created so we can be working on one version of the game while testing another and have another version already live
• A way to have login data securely pulled from a MySQL DB from one cloud system and used on the game server set in another cloud which uses a NoSQL DB for performance
• A log system that can handle 10,000 messages a second for 3 days straight and allow us to instantly search them to find problems on the fly
• A metrics tracking server system to help us make sense of more high level data from the players
• Game servers that handled logins and handled data serialization for player and house data and a way to publish them
• A load balancing solution so we can seamlessly add more servers if needed for future scaling
• A database system that can cluster and scale as needed
• Systems to enable various levels of login access so we can let some in and lock others out
• Rolling publish system for the servers so we can update the servers without booting everyone off
• Actual machines setup in the cloud with all of the above stuff installed on them
• Security lockdown on all the boxes in the cloud so no extra ports are open

And that is only part of what went into the work for just the server side that most people probably never even thought about!

So the good news is that with only a few exceptions it all went great. There was a bug in the patcher that caused some people to randomly get low patch rates but the median install time was just over 12 minutes. Login issues were very limited and mostly had to do with people using their forum login info instead of their website logins or people who had split pledges. We had one server hiccup that we quickly corrected that delayed us opening the virtual doors by 13 minutes. (Trust me, 13 minutes doesn’t sound like long but it felt like an eternity here!) After that, it was generally smooth sailing. We had around 700 concurrent users at the end of the first hour.

Our goal for the patcher system was to make it near zero cost for us to deploy. This was important to us to allow us to not stress about how often we patched. For those who haven’t had to distribute huge amounts of data before, most options end up costing on the order of 12 cents per GB of data dropping to around 4 cents per GB for high volume CDN distribution.

For this initial release and patching we ended up distributing around 8TB (8,000,000,000,000 bytes) of data. We expect to double the project size for each of the next few releases. Then, factor in that we will just be leaving the servers up at some point and distributing patches and increased users and participation and our projection is that we’ll end up needing to distribute around 5PB (5,000,000,00,000,000 bytes!) of data before launch which even at the cheapest rates runs the bill up to around $200k!

For those who didn’t know, the patcher system is based on mostly off the shelf torrent technology. Our automated publish process also updates a shared RSS feed so players lucky enough to have high bandwidth connections can just subscribe to the feed and automagically help us distribute new patches. For those who are on networks that block torrent traffic we have a fallback system that patches over a standard http connection off of our servers.

The following data is from our log server and metrics server I mentioned above. This first chart shows the login rate over the first 15 minutes. We had 75 people login during the first 10 seconds after we opened the servers and then about 25 per 10 seconds for the next 5 minutes. Even with that type of surge and a single server, the server was never at more than about 5% load. The only bottleneck in the login was pulling account data from the MySQL database and we’ll be scaling that up for the next release.


Here is a chart showing concurrency for the first hour the game was up.

Again, even with 700 concurrent users on a single box, the server never showed significant load and that was with only a single server box sitting behind the load balancer.

On release 1 there were 9670 characters created by 8080 total unique players. The average number of times people logged into the game was just under 3 times with the average play session length running at 23 minutes and 31 seconds per session. Of the 8080 unique players, 1912 had a single session that lasted more than an hour.

Hopefully this trend won’t continue but 6322 Male characters were created vs 3348 women!

As for country of origin we had a great mix:
3969 – USA
673 – Germany
512 – Canada
457 – UK
369 – NULL
343 – Australia
165 – France
158 – Finland
128 – Italy
100 – Sweden

More than 70 countries were represented in all including Egypt, Moldova, Peru, Belarus, Zambia, Vietnam, Lebanon, China, Thailand, Qatar and dozens more. NULL is obviously due to a small glitch in our metrics software that logs NULL if it can’t tell where they are from.

For the historical records, the first 5 character names created for Shroud of the Avatar by a non-dev on the official opening of Release 1 (not the Dev+ sneak preview) were:

• Naweh
• Ivashana Sicarius
• Yorke Blacksnow
• Hex
• Thorsgard Aurelius

On the client side, things also went great. I think there were far more issues visible in the client but only a few nasty problems. There were two things that caused some people to crash, running out of memory and to a much lesser extent, video card driver issues. We’re looking into both but I strongly encourage everyone to update their video card drivers before the next release because most looked like they were on passable video cards but probably hadn’t updated their drivers since they got the computers!

The memory issue is on us and usually comes with the optimization of the project, which tends to happen towards the end of the project. One of the most respected members of the graphics programming world, John Carmack, once said, “Premature optimization is the root of all evil.” Due to how we’re developing, we’re going to have to ignore his words a wisdom a bit and do some more optimization on each release but know that we will be doing some serious work on performance late in the project so don’t panic if the game doesn’t run great just yet.

Overall, Release 1 went much better than expected. We learned lots, tested all our infrastructure, and hopefully let people get just enough to whet your appetite for the upcoming releases! Release 2 should show incremental improvements in everything we showed for Release 1, crafting, another city, and hopefully we’ll even be able to integrate a few changes based on player feedback!

Chris Spears
Tech Director++