1. This forum section contains Blog style threads from the Developers, and/or threads with details about what they have been working on, previews of upcoming content, and sometimes backstory associated with the content. Players are permitted to reply to these threads if the developer has indicated it is permissible by leaving the thread unlocked. Please note that TOS applies, and any abuse or trolling will be moderated.
    Dismiss Notice

Chris' Daily Work Blog

Discussion in 'Developer Work and Blog posts' started by Chris, Apr 5, 2017.

Thread Status:
Not open for further replies.
  1. Chris

    Chris Tech Lord Moderator Ambassador SOTA Developer

    Likes Received:
    Trophy Points:
    Long time no blog! I figured our recent crazy changes were as good of an excuse as any to restart my blog work.

    So, as we announced on Friday, SURPRISE! We ditched the office! So first up, let me address what I’m sure is being said on Reddit right now. Yes, money was a factor but NO, not having enough money to pay for an office was NOT a factor. We save a total of roughly $8-10k a month by not having an office and that means we now have $8-10k a month more to spend on advertising.

    So the whole story goes something like this….

    Roughly 6 months ago our previous lease was expiring and the landlord wanted more than $11k/month AND a 3 year commitment. When we first started in that space it was a bit over $5k a month. We said no thanks and looked for a new space.

    Finding good leases in Austin is CRAZY hard right now because of how strong the economy is BUT I managed to find a low commitment, 6 month sublease in a great location and we were told the rent would be reasonable if we wanted to continue after the sublease. Well, we love the location of the new space, but after a brief trial in the space, we found it had several deal breakers.

    First, the office temperatures were unacceptable. We moved in around October and found that the AC was able to keep up with a Texas winter! Zero chance we would be sweat free in the summer. The previous space had A/C issues as well and it was a HUGE deal that resulted in people just working from home on many hot days.

    Second, there were no good internet options. Likely because that area was mostly retail with little demand for internet, we were in a fiber desert! The best option we could get was to pay $500 a month for 200 mbps down and 50 mbps up and that 50mbps up was on a good day with the wind at your back! This mean that our big builds would take as much as 2 hours to upload. Doesn’t sound huge but if there is something requiring a critical patch, 2 hours is a killer. It was enough that we were changing how we did builds just to deal with 50mbps up.

    So, time came around to renew our lease and we were already having second thoughts. Then the landlord informed us the rent would more than double AND that they wanted a 4 year lease and that pretty much sealed the deal.

    Moving sucks BUT moving the physical stuff is the easy part. For this move I did most of that myself. The painful part is moving the digital stuff and rebuilding lots of security and infrastructure over and over again. So, around December I started cooking up a whacky plan. After a quick inventory, I found we had 17 in house server boxes that we had to move. Why not virtualize the smaller ones to combine them, move them all onto blades, throw them all into a data center, and then just have people VPN in to that? Then, we never have to move our servers again and we remove the most painful part of shifting offices.

    So starting in January, that is what I started doing. I continued doing that in February and finished up the last few boxes while I was on my vacation at Disney World. When I got back, Undone and I carried a bunch of blades to the data center and hooked them all up and spent a few days configuring and screaming “WTF, why doesn’t this work!” about 100 different things.

    In the end, we got it all setup and working great. The end result is now employees can connect from anywhere. Ok, technically they could already do that but now they can do it at 1 gbps. Also, now all of our critical work servers are in a super secure data center with fire protection, power generators, A/B power, and with rock solid internet.

    Also, just to be clear, these are not the game servers. These are things like our email server, Jira, Wiki, Perforce, Backup server (mirrors offsite as well), dev servers, 4 build servers and integration servers, etc.

    The only big gotcha we ran into is, once again, UNITY!!!!!!!! We moved from some overclocked quad core boxes that ran about 4.5ghz to blade servers that each have dual 3ghz hex core xeons. So obviously 12 3ghz cores will run as fast or nearly as fast as 4 4.5ghz cores, right? RIGHT? Choo choo, all aboard the Unity single threaded fail train! Builds took almost twice as long on the new boxes. Part of it due to the 3ghz vs 4.5ghz but also probably some because the 3ghz chips were several generations older and on slowe ecc ddr3 vs ddr4. We actually knew they would run slower but we thought it would be more like 20 or 30% slower.

    My March task is to build out some ghetto blades with low core counts and high chip speeds for cheap! Wish me luck… I’ll have to rig up some ghetto blade configurations since no server chips are overclockable and we really don’t care about ECC or other server features for build machines.

    So, now that we have this monumental task out of the way, we can easily be as nomadic as we want to be! All people need to work efficiently now is an internet connection. This doesn’t mean we won’t ever have an office again. It just means that moving isn’t as painful.

    With that said, I’ve already started talking to a few people about possible office space options. We have a few people who think they will have trouble working from home due to kids, bad internet, or no good work space so I’ll start shopping around for options. We pretty much will have to have an office for some of the new hires we will need to make later this year since there is a lot of training that will be done to onboard new employees.

    For roughly 2 ½ months, I’ve been spending the vast majority of my time on moving stuff. First, moving our game servers after our previous data center was acquired. Then, moving our office servers and all the stuff in the office! I am excited to get back to more hands on game balance, coding, and design! Look forward to all that stuff I promised you months ago finally showing up in game. :)

    As for working at home, it is exciting! I save 2 hours a day on traffic. I get to work with my dog smiling at me all day long. I don’t have to fight Lord Subtleton over what temperature to keep the office! :p My fridge always has good food in it. I can start work at 4AM in my PJs!

    Best of all though, I can now do live streams whenever I want and not worry about disturbing people around me. I plan on doing this fairly often. Most of it will just be me playing the game and taking notes but I’ll probably answer a few questions.

    I am also going to restart my blog. Don’t expect every day but maybe twice a week!

    See you in game!
Thread Status:
Not open for further replies.