Chris's Release 3 thoughts

Before I dive into my R3 write up I wanted to share two relevant quotes by Leonard Bernstein and another by Patton that are for the SotA team more than for the fans. I’m sharing it here because it will probably be helpful for the fans to understand some of our challenges:

“To achieve great things, two things are needed; a plan, and not quite enough time.” - Leonard Berstein

“A good plan violently executed now is better than a perfect plan executed next week.” - George S. Patton

While I’m not quite ready to get these quotes tattooed across my chest, I do think they seem more and more relevant every release to keep things in perspective. As we continue with this crazy experiment of doing live development on a tight schedule and budget but a big vision, we have to keep reminding ourselves of the sentiments contained in these two quotes.

The first quote can be interpreted several different ways and they are all relevant. To make this crazy experiment work, the whole team needs to always feel the urgency. This isn’t a 100+ person team on a 5 year project, we don’t have time to sit back and relax because someone else is carrying the load if one person slacks off.

The other way that I hope the team takes it is that the greatness of accomplishments must be judged somewhat by relative expectations. Even though we all know each release could be better, the fact that we’re doing so much with so little is a great achievement in itself and we have to use that positive energy to fuel us so we can push to continue to complete our plan with not quite enough time for each release.

The Patton quote is more for the grizzled veterans on the team who cringe every time we get ready to do a release and there are 200+ bugs still open. With this crazy development experiment we’re conducting, we can’t be perfectionists, we just don't have the time. We have to instead focus on doing well enough that we can get the information we need out of each release and hope that our fan base is informed enough to know what they are seeing and look by the little bugs. If we delay plans, it needs to because we aren’t going to get what we need out of the milestone and not because of minor cosmetic issues.


So, now that I got my usual soap box speech out of the way, on to R3 stuff!

I think if I had to pick one word to summarize R3 it would probably be surprising!

First, I think it was surprising how much work we still had to do to get multiplayer working _well_ despite having had it mostly working for months. I think it was surprising how quickly we were able to improve our bandwidth usage which we managed to reduce by about a factor of 20 in a matter of less than a week. I think it was surprising that the release wasn’t buggier since we were essentially coding directly on the live branch until the very moment we let people in on Thursday and many last minute fixes went live with little or no testing.

Once we let people in, I think it was surprising to see how many issues came up that we had never seen before internally. Lastly, it was surprising how quickly we managed to fix some very major issues.

Most people didn’t know this but we opened a Dev+ test several days before the main test just to make sure all of our ducks were in a row. What we found was that our ducks were scattered all over the yard, stuck down in storm drains, and wandering around in the street drunk.

On the first test we managed to make it 10 minutes and get just over 100 people into the game before everything caught fire and burned to the ground. The Dev+ forums were full of people posting “Great test, too bad I couldn’t get in to see it!” This was less than a week before we were expecting 7500+ people to play over a 72 hour window. There were at least 4 major things that were identified as problems and all were fixed and tested again in a matter of a few days thanks to some Herculean efforts by people on the team.

While release 3 had some major glitches, overall we think it went pretty well. This release had more people than release 2 but a few hundred less than release 1. The biggest difference though was that the play time per unique person was through the roof for this release. Of the 7500+ people who played, more than 3600 of them had at least one session of an hour or more even though we actually removed houses and some other content. With the exception of a few hours during the nights, there were at least 100 players online throughout the official 72 hour test window.

The good news is that we learned that the servers can easily handle 64 players on a map without any problems. Unfortunately, many clients couldn’t! We have spent most of this week cleaning up performance issues and found lots of interesting bits that have resulted in massive performance improvements on the clients for multiplayer. We’ll probably do a Dev+ test soon with that 64 number turned up some more and see if our client performance fixes hold up.

We also found that some people had connection problems that we hadn’t seen before. We think we have now tracked these down and they had to do with issues with our DNS causing time outs on the client connection.

The worst issue we had was rare but very painful. Rarely, a player would change maps but actually end up on a different map on the server. So on their client they would think they were in Kingsport but the server and master client had them in Owlshead. The result was an insane number of errors being logged and everything being out of sync. Fortunately this didn’t happen too often but when it did it was cats and dogs living together and everyone on the map suffered while their clients logged messages like crazy about the trouble maker.

I’ll be spilling more details on technical stuff and humorous bugs over in the Dev+ forum <obligatory Dev+ plug!> so keep an eye on those forums later today. As you might expect with the first multiplayer test, there were some very humorous issues that came up!

Again, thanks to all those that joined us in testing. We had over 7500 unique players join in this weekend to help us stress the servers and we’re happy to say that you all failed miserably and the servers barely showed any load at all! Now to fix that client to get some of those hitches and low frame rate issues cleaned up before R4!

Thanks for reading!

 

That is 7500 unique accounts logged in. I didn't even look into how many total characters were created.