Release 1 Postmortem and Breakdown

Discussion in 'Announcements' started by Chris, Dec 20, 2013.

Thread Status:
Not open for further replies.
  1. Chris

    Chris Tech Lord Moderator SOTA Developer Ambassador

    Messages:
    2,406
    Likes Received:
    27,412
    Trophy Points:
    190
    Gender:
    Male
    While not without its flaws, Release 1 was a huge success as a test run for the game. We had a long list of things we needed to do real world testing on and almost all passed with flying colors. For the casual gamer who doesn’t think much about what goes on behind the scenes, they hopefully patched, logged in, played for a while and said “Yep, looks like a great start!” What most non-game developers don’t know is the massive amount of work that went on behind the scenes to go from zero to that point in 8 months all while growing our team from miniscule to just tiny (for a AAA RPG).

    Behind the scenes, the list of things that had to come together in a short amount of time was huge. The fact that most people didn’t notice was a testament to just how well things went overall. The incomplete list of things that got done includes:
    • Installers had to be created
    • Bootstrap system to patch the installer
    • System to build and apply patches so we only had to send the differences (deltas) in the installed files
    • A way to distribute them that wouldn’t kill our budget
    • A backup distribution for those who are behind secure networks that don’t allow peer connections
    • A publish process created to remove human error
    • Process for multiple code and data sets created so we can be working on one version of the game while testing another and have another version already live
    • A way to have login data securely pulled from a MySQL DB from one cloud system and used on the game server set in another cloud which uses a NoSQL DB for performance
    • A log system that can handle 10,000 messages a second for 3 days straight and allow us to instantly search them to find problems on the fly
    • A metrics tracking server system to help us make sense of more high level data from the players
    • Game servers that handled logins and handled data serialization for player and house data and a way to publish them
    • A load balancing solution so we can seamlessly add more servers if needed for future scaling
    • A database system that can cluster and scale as needed
    • Systems to enable various levels of login access so we can let some in and lock others out
    • Rolling publish system for the servers so we can update the servers without booting everyone off
    • Actual machines setup in the cloud with all of the above stuff installed on them
    • Security lockdown on all the boxes in the cloud so no extra ports are open

    And that is only part of what went into the work for just the server side that most people probably never even thought about!

    So the good news is that with only a few exceptions it all went great. There was a bug in the patcher that caused some people to randomly get low patch rates but the median install time was just over 12 minutes. Login issues were very limited and mostly had to do with people using their forum login info instead of their website logins or people who had split pledges. We had one server hiccup that we quickly corrected that delayed us opening the virtual doors by 13 minutes. (Trust me, 13 minutes doesn’t sound like long but it felt like an eternity here!) After that, it was generally smooth sailing. We had around 700 concurrent users at the end of the first hour.

    Our goal for the patcher system was to make it near zero cost for us to deploy. This was important to us to allow us to not stress about how often we patched. For those who haven’t had to distribute huge amounts of data before, most options end up costing on the order of 12 cents per GB of data dropping to around 4 cents per GB for high volume CDN distribution.

    For this initial release and patching we ended up distributing around 8TB (8,000,000,000,000 bytes) of data. We expect to double the project size for each of the next few releases. Then, factor in that we will just be leaving the servers up at some point and distributing patches and increased users and participation and our projection is that we’ll end up needing to distribute around 5PB (5,000,000,00,000,000 bytes!) of data before launch which even at the cheapest rates runs the bill up to around $200k!

    For those who didn’t know, the patcher system is based on mostly off the shelf torrent technology. Our automated publish process also updates a shared RSS feed so players lucky enough to have high bandwidth connections can just subscribe to the feed and automagically help us distribute new patches. For those who are on networks that block torrent traffic we have a fallback system that patches over a standard http connection off of our servers.

    The following data is from our log server and metrics server I mentioned above. This first chart shows the login rate over the first 15 minutes. We had 75 people login during the first 10 seconds after we opened the servers and then about 25 per 10 seconds for the next 5 minutes. Even with that type of surge and a single server, the server was never at more than about 5% load. The only bottleneck in the login was pulling account data from the MySQL database and we’ll be scaling that up for the next release.
    [​IMG]

    Here is a chart showing concurrency for the first hour the game was up.
    [​IMG]
    Again, even with 700 concurrent users on a single box, the server never showed significant load and that was with only a single server box sitting behind the load balancer.

    On release 1 there were 9670 characters created by 8080 total unique players. The average number of times people logged into the game was just under 3 times with the average play session length running at 23 minutes and 31 seconds per session. Of the 8080 unique players, 1912 had a single session that lasted more than an hour.

    Hopefully this trend won’t continue but 6322 Male characters were created vs 3348 women!

    As for country of origin we had a great mix:
    3969 – USA
    673 – Germany
    512 – Canada
    457 – UK
    369 – NULL
    343 – Australia
    165 – France
    158 – Finland
    128 – Italy
    100 – Sweden

    More than 70 countries were represented in all including Egypt, Moldova, Peru, Belarus, Zambia, Vietnam, Lebanon, China, Thailand, Qatar and dozens more. NULL is obviously due to a small glitch in our metrics software that logs NULL if it can’t tell where they are from.

    For the historical records, the first 5 character names created for Shroud of the Avatar by a non-dev on the official opening of Release 1 (not the Dev+ sneak preview) were:
    • Naweh
    • Ivashana Sicarius
    • Yorke Blacksnow
    • Hex
    • Thorsgard Aurelius

    On the client side, things also went great. I think there were far more issues visible in the client but only a few nasty problems. There were two things that caused some people to crash, running out of memory and to a much lesser extent, video card driver issues. We’re looking into both but I strongly encourage everyone to update their video card drivers before the next release because most looked like they were on passable video cards but probably hadn’t updated their drivers since they got the computers!

    The memory issue is on us and usually comes with the optimization of the project, which tends to happen towards the end of the project. One of the most respected members of the graphics programming world, John Carmack, once said, “Premature optimization is the root of all evil.” Due to how we’re developing, we’re going to have to ignore his words a wisdom a bit and do some more optimization on each release but know that we will be doing some serious work on performance late in the project so don’t panic if the game doesn’t run great just yet.

    Overall, Release 1 went much better than expected. We learned lots, tested all our infrastructure, and hopefully let people get just enough to whet your appetite for the upcoming releases! Release 2 should show incremental improvements in everything we showed for Release 1, crafting, another city, and hopefully we’ll even be able to integrate a few changes based on player feedback!

    Chris Spears
    Tech Director++
     
    Soleidad, Deathblow, cs2501x and 72 others like this.
  2. Myrcello

    Myrcello Avatar

    Messages:
    2,668
    Likes Received:
    9,213
    Trophy Points:
    153
    Gender:
    Male
    Thank you all for your hard work and giving us a little insight what you are doing behind the scenes.

    I am also happy to see that your Release1 did grab attention also outside of our little community and received overall positive look out for the future.
     
  3. High Baron O`Sullivan

    High Baron O`Sullivan Avatar

    Messages:
    3,503
    Likes Received:
    8,109
    Trophy Points:
    165
    Gender:
    Male
    Location:
    is everything.
    Super excited for R2. You guys are doing an amazing job! Keep it up-
     
    Skalex, Alexander, rild and 3 others like this.
  4. mike11

    mike11 Avatar

    Messages:
    1,592
    Likes Received:
    1,564
    Trophy Points:
    113
    Gender:
    Male
    I think this may be a massive understatement because for me, I think I spend at least 8-12 hours each day PLAYING this historical 1st pre-release.
    Has any other game in history had as much actual playtime at this stage of release??

    Thank you all for giving us the chance to participate and give feedback and to have fun while doing so, which it certainly was.

    You all deserve a huge pat on the back, have yourselves a awesome holiday.
     
    Skalex, Nystyl, rild and 5 others like this.
  5. Sir Frank

    Sir Frank Master of the Mint

    Messages:
    4,066
    Likes Received:
    10,904
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Kansas City
    Thank you, Chris.
    Even though a lot of this is over my head, I still enjoy reading the details.

    Phenomenal work.

    Congratulations.
     
    rild, OSullivan and Myrcello like this.
  6. Link_of_Hyrule

    Link_of_Hyrule Bug Brigade - Bug Hunter

    Messages:
    1,394
    Likes Received:
    2,142
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Ogden, Utah
    Very excited to read it was such a success! Also this post was very informative I didn't know patch distribution was so expensive. Looking forward to the next release!
     
  7. monkeysmack

    monkeysmack Avatar

    Messages:
    628
    Likes Received:
    1,453
    Trophy Points:
    105
    Gender:
    Male
    Location:
    your mom's house
    Awesome Job Team! The first release blew me away and showed how great this game is going to be. I'd love donate my bandwidth to help distribute the game. Is there any chance an indicator could be placed on the launcher UI to show if it's using torrent versus just http?
    Thanks again!
     
  8. Joviex

    Joviex Avatar

    Messages:
    1,518
    Likes Received:
    3,137
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Burbank, CA
    Excellent information, congratulations on a successful R1!


    Just a curiosity question, those points above: MongoDB, Cassandra, ....?

    They are just similar issues we have here, at scale, and Mongo was our go to (with NLog pumping out the actual logs from internal code).

    Wondering what other backend solutions you guys settled on to take this to scale.
     
    sgt_pEppEr and Bramagola like this.
  9. GimmeUOPlz

    GimmeUOPlz Avatar

    Messages:
    241
    Likes Received:
    276
    Trophy Points:
    18
    Awesome great to hear it went so well. I too didn't realize how expensive patch distribution is, and I'll definitely help as much as I can by seeding for future releases.



    Cheers to a great first test guys! Can't wait for crafting next!
     
  10. Doppelganger [MGT]

    Doppelganger [MGT] Avatar

    Messages:
    496
    Likes Received:
    1,201
    Trophy Points:
    55
    Gender:
    Male
    Location:
    Tampa Bay, FL
    Thank you for the detailed post. I like that you guys continue to keep the player base in the loop and well informed :)
     
  11. Umbrae

    Umbrae Avatar

    Messages:
    2,598
    Likes Received:
    4,267
    Trophy Points:
    153
    Gender:
    Male
    Nerd-tastic Data! Love it, Chris. Thanks for sharing.

    But I will say the first non-characters are loaded because you guys started early after I scheduled my vacation! Shame on you!. ;)
     
  12. TheMadHermit

    TheMadHermit Avatar

    Messages:
    666
    Likes Received:
    3,410
    Trophy Points:
    93
    Chris,

    Great informative summary! Thanks for giving us a glimpse into the world at this early stage. I haven't been this excited about an RPG since Ultima. Definitely takes me back to the excitement I felt as a teenager playing the early Ultima games.

    Keep up the great work and thanks again for letting us take a virtual tour of what's yet to come!
     
  13. Alayth

    Alayth Avatar

    Messages:
    223
    Likes Received:
    269
    Trophy Points:
    18
    Thanks so much, Chris. It's really good to get a breakdown of this sort of thing for those of us who don't have a good idea of what's going on behind the scenes! Glad to hear everything went so smoothly.
     
  14. Vallo Frostbane

    Vallo Frostbane Avatar

    Messages:
    1,762
    Likes Received:
    3,585
    Trophy Points:
    113
    Still waiting for Lum's best quotes from the Player-NPC chat dialogs ;)
     
  15. Reigner

    Reigner Avatar

    Messages:
    114
    Likes Received:
    154
    Trophy Points:
    18
    Chris,

    Thanks for sharing so many of the behind-the-scenes details. As a Dev+ backer trying to learn more about what it takes to create/publish a AAA game, these are gems.

    The entire SotA team should be very proud of what they have accomplished.

    Happy Holidays!

    ~Reigner
     
    Cather and TheMadHermit like this.
  16. Aartemis

    Aartemis Avatar

    Messages:
    1,390
    Likes Received:
    3,836
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Cleveland, Ohio
    Fantastic and Congratulations!

    Looking forward to R2. :)

    Happy Holidays to the Entire Dev Team and Community!
     
    rild likes this.
  17. smack

    smack Avatar

    Messages:
    7,078
    Likes Received:
    15,278
    Trophy Points:
    153
    Metrics Mega Post! I love it!

    Thanks Chris for the detailed postmortem and a huge thanks to the entire Dev team for a successful Release 1! Would love to hear Lum's postmortem on the NPC conversation system too. :)
     
    TheMadHermit, rild and monkeysmack like this.
  18. LadyM

    LadyM Avatar

    Messages:
    46
    Likes Received:
    38
    Trophy Points:
    8
    Gender:
    Female
    Location:
    Texas
    Thanks so much for this! Proud to be one of the 3348 female characters created ;-) Can't wait for Release 2 - woot woot
     
    TheMadHermit likes this.
  19. Mitch [MGT]

    Mitch [MGT] Avatar

    Messages:
    494
    Likes Received:
    1,057
    Trophy Points:
    43
    Gender:
    Male
    Well said Chris! Congrats on a very successful R1 Portalarium team!

    Can't wait until R2, time to do me some craftin'!
     
    TheMadHermit likes this.
  20. Hotsacks

    Hotsacks Avatar

    Messages:
    2
    Likes Received:
    1
    Trophy Points:
    3
    Tears of joy whilst walking through "New Britannia"

    Christmas came early this year! :D
     
    TheMadHermit likes this.
Thread Status:
Not open for further replies.