1. Here you will find official announcements and updates. These announcements are also linked in the Official SotA Discord server.
    We encourage comments from the community! To keep the announcements official, we ask that comment threads be created in the General forums for player input.

                                                 Thanks!

Server relocation successful!

Discussion in 'Announcements' started by Chris, Feb 1, 2019.

Thread Status:
Not open for further replies.
  1. Chris

    Chris Tech Lord Moderator Ambassador SOTA Developer

    Messages:
    2,470
    Likes Received:
    27,551
    Trophy Points:
    190
    Gender:
    Male
    Greetings Avatars! Unless you had been paying attention to updates you might have missed that we accomplished a major feat during the R62 release. In a 2 1/2 hour downtime, we moved our entire server cluster from one data center to another! Didn't notice it had been moved? Well, that is exactly as planned!

    While we continue to verify that the new data center has a rock solid connection and we're not seeing lost packets like we were at the previous location, we are seeing some momentary hiccups. These actually have nothing to do with our connection and more to do with us running the game on a much smaller box than before. Currently we are running the main game server on a single 3ghz quad core CPU with 16GB of Ram.

    As we projected, this is plenty to handle 1000+ players at once. What we didn't realize until yesterday was that about every 20 minutes we get a big CPU spike for garbage collection. On the old box (3x the cores and 3x the memory) this wasn't noticeable. However, on the new box this is causing a 1-2 second server hitch that is resulting in players seeing the "Lost connection" icon briefly and in a few cases has caused disconnects.

    We'll be relocating the old, bigger boxes today and try to get them setup over the next few days. Since this has now been brought to our attention, we will also look into allocations that would be causing these issues on the server. Also worth noting that our servers are setup so we can run any number of servers and distribute the load if needed for scaling so this will also not be a problem that would ever prevent us from scaling.

    The added boost to all this is that the current data center is lower priced, more easily accessible, more flexible, has lower ping times, and more secure than our previous solution!

    If you read the daily stand up notes over the past month, you're aware of what a pain in the arse this was but I think in the end it was for the best that we had to move as it will be better for us and for the players in the long run.

    </begin horn tooting>
    Also, now that we've made move successfully and without the world catching on fire, I want to toot our collective horn a bit to remind people stuff they might take for granted! In 5 years we have had:
    • Roughly four brief outages of 4 hours or less
    • We have exactly 1 roll back of roughly 15 minutes and that occurred right after an update and was our own choice. (We forgot to do one data migration and didn't want to wait a month to do it)
    • We've had no data loss
    • We've had ZERO dupe bugs
    • We've had zero data breeches
    • We have a grand total of only 2 hours of planned downtime per month

    Now I'm going to go and knock on every single piece of wood I can find in the office so I don't jinx us! Just wanted to remind old and new people that while we have plenty of flaws in the game, we do try to be professionals at all times when it comes to the critical stuff.
    </end horn tooting>

    Also, we might have a brief down time sometime in the next week to migrate the game server to the server bigger box. There is a good chance we'll be able to do this by just rolling the server but there is some non-zero chance we'll need to take it down briefly. We'll announce when this is happening if needed!

    Now go play and have fun!
     
    Bom, Astirian, Kalidorn and 53 others like this.
  2. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,409
    Likes Received:
    3,738
    Trophy Points:
    125
    Location:
    earth... mostly
  3. Brass Knuckles

    Brass Knuckles Avatar

    Messages:
    3,958
    Likes Received:
    7,707
    Trophy Points:
    153
    Nice work @Chris we getting taming ballance in R63?
     
  4. Duke Gréagóir

    Duke Gréagóir Legend of the Hearth

    Messages:
    5,676
    Likes Received:
    11,810
    Trophy Points:
    165
    Location:
    Dara Brae
  5. Chris

    Chris Tech Lord Moderator Ambassador SOTA Developer

    Messages:
    2,470
    Likes Received:
    27,551
    Trophy Points:
    190
    Gender:
    Male
    @Lord Subtleton deserves most of the credit for things going smoothly. I did a lot of the manual labor and machine setup and then pointed at stuff. He's the detail oriented one who makes everything work behind the scenes. Huge thanks to him for making me look so damn good though. :)
     
  6. Vladamir Begemot

    Vladamir Begemot Avatar

    Messages:
    6,194
    Likes Received:
    12,076
    Trophy Points:
    153
    Gender:
    Male
    Impressive!
     
    Fionwyn Wyldemane likes this.
  7. thesometimeslurker

    thesometimeslurker Avatar

    Messages:
    521
    Likes Received:
    507
    Trophy Points:
    75
    Gender:
    Male
    Location:
    A hovel
    Yay! I'm so happy that it went smoothly!
     
    Fionwyn Wyldemane likes this.
  8. majoria70

    majoria70 Avatar

    Messages:
    10,347
    Likes Received:
    24,869
    Trophy Points:
    153
    Gender:
    Female
    Location:
    United States
    Thanks @Chris and to the team for all you do.
     
    Fionwyn Wyldemane likes this.
  9. CrandalltheFoole

    CrandalltheFoole Avatar

    Messages:
    594
    Likes Received:
    624
    Trophy Points:
    75
    Gender:
    Male
    Location:
    DFW, Texas
    That is crazy - My legacy system that I game on is "almost" the same:

    Xeon CPU - 6 core - X5660 @ 2.80GHz
    48GB RAM Registered ECC DDR3
    NVIDIA GTX 970 4GB
     
    Fionwyn Wyldemane likes this.
  10. Ancev

    Ancev Avatar

    Messages:
    1,150
    Likes Received:
    1,956
    Trophy Points:
    113
    IT can sometimes be a thankless job, thanks Chris and team for all of the hard work that goes on in the background!
     
    Fionwyn Wyldemane likes this.
  11. Godra

    Godra Avatar

    Messages:
    589
    Likes Received:
    946
    Trophy Points:
    75
    Gender:
    Male
    Location:
    Philadelphia
    @Chris I didn’t doubt you a second!!! I even put 100 COTOs on the line. So again FLAWLESS.... Atos 1 - servers 0
     
    Fionwyn Wyldemane likes this.
  12. Barugon

    Barugon Avatar

    Messages:
    15,679
    Likes Received:
    24,293
    Trophy Points:
    153
    Gender:
    Male
    For large, time critical systems, GC=bad.
     
  13. Duke Gréagóir

    Duke Gréagóir Legend of the Hearth

    Messages:
    5,676
    Likes Received:
    11,810
    Trophy Points:
    165
    Location:
    Dara Brae
  14. Magnus Zarwaddim

    Magnus Zarwaddim Avatar

    Messages:
    975
    Likes Received:
    1,884
    Trophy Points:
    93
    $100 says someone, somewhere out there in the ether, spins this into something negative (if they haven't already).

    Kudos.
     
    Boris Mondragon likes this.
  15. Anpu

    Anpu Avatar

    Messages:
    7,944
    Likes Received:
    9,015
    Trophy Points:
    153
    Location:
    Hemut
    Congrats on the smooth transition!!! :D

    Things have been running just fine for me today.
     
  16. Earl Atogrim von Draken

    Earl Atogrim von Draken Avatar

    Messages:
    6,331
    Likes Received:
    12,109
    Trophy Points:
    165
    Gender:
    Male
  17. Duke Olahorand

    Duke Olahorand Localization Team

    Messages:
    1,287
    Likes Received:
    2,465
    Trophy Points:
    125
    Gender:
    Male
    Location:
    North Germany
    Thanks for all the background work :)
     
    Fionwyn Wyldemane likes this.
  18. Cirsee

    Cirsee Avatar

    Messages:
    484
    Likes Received:
    1,013
    Trophy Points:
    43
    Gender:
    Female
    Great job Port for a smooth transition!
     
  19. Barugon

    Barugon Avatar

    Messages:
    15,679
    Likes Received:
    24,293
    Trophy Points:
    153
    Gender:
    Male
    Make that five.
     
  20. Moussaka

    Moussaka Avatar

    Messages:
    145
    Likes Received:
    243
    Trophy Points:
    18
    Location:
    Switzerland
    Force the garbage collection to run every minute or so

    less work at once = less performance impact
     
Thread Status:
Not open for further replies.