1. Here you will find official announcements and updates. These announcements are also linked in the Official SotA Discord server.
    We encourage comments from the community! To keep the announcements official, we ask that comment threads be created in the General forums for player input.

                                                 Thanks!

Connectivity Issues April 1, 2017 (Saturday Morning)

Discussion in 'Announcements' started by DarkStarr, Apr 1, 2017.

Thread Status:
Not open for further replies.
Page 1 of 3
  1. DarkStarr

    DarkStarr Executive Producer SOTA Developer

    Messages:
    6,692
    Likes Received:
    45,561
    Trophy Points:
    202
    Gender:
    Male
    Apparently the gods do not like April Fools pranks and decided to punish us.

    At approximately midnight CDT one of our servers at our data center experienced a catastrophic hardware failure. Players inside the game could remain connected as long as they did not change scenes but anyone changing scenes would lose connection and anyone trying to log in to the game could not.

    As of 7:50 AM CDT we were able to temporarily resolve the issue and the servers can be reached again. We were (and are) dealing with a hardware error that we haven’t fully identified, yet, but were able to resume service nevertheless, for now. We’ll continue looking into it, with hopefully no additional downtime. We might need to replace a machine, though, which would come with a short maintenance downtime, in case of. In that case, we’ll announce it ahead of time.

    We apologize for the inconvenience this caused anyone.
     
    DarkStarr, Apr 1, 2017
    #1
    Time Lord, Mhtic, manufacturedsoul and 21 others like this.
  2. Sheamus McGuinness

    Sheamus McGuinness Avatar

    Messages:
    595
    Likes Received:
    1,644
    Trophy Points:
    93
    BWAHAHAHAH!!! Serves you pranksters right!!

    (I enjoyed it tho)
     
    Sheamus McGuinness, Apr 1, 2017
    #2
    Time Lord and Roycestein Kaelstrom like this.
  3. James Thunder

    James Thunder Avatar

    Messages:
    19
    Likes Received:
    26
    Trophy Points:
    3
    As anyone in IT knows, build assuming failures. Regardless, kudos to Portalarium for the impressive uptime! And with an ALPHA product no less.
     
    James Thunder, Apr 1, 2017
    #3
    Time Lord, Montaigne, Jens_T and 2 others like this.
  4. Onyx

    Onyx Avatar

    Messages:
    460
    Likes Received:
    1,199
    Trophy Points:
    55
    Shucks. I almost liked the idea of TRAMUCCA. Almost!
     
    Onyx, Apr 1, 2017
    #4
    Time Lord and Roycestein Kaelstrom like this.
  5. rpgmaniac82

    rpgmaniac82 Avatar

    Messages:
    262
    Likes Received:
    367
    Trophy Points:
    40
    Gender:
    Male
    Location:
    Greece-Galahad Dragonsbane & Erevis cale
    i almost thought this true..you buggers:p
     
    rpgmaniac82, Apr 1, 2017
    #5
    Time Lord, Fionwyn Wyldemane and Roycestein Kaelstrom like this.
  6. Arcanoxer

    Arcanoxer Avatar

    Messages:
    408
    Likes Received:
    761
    Trophy Points:
    55
    Gender:
    Male
    Location:
    rl
    Sota already has a Trammel, it's called Single Player (friends only).
    Only Felucca is missing...
     
    Arcanoxer, Apr 1, 2017
    #6
    The Hendoman, Time Lord, Brass Knuckles and 2 others like this.
  7. mdsota152

    mdsota152 Avatar

    Messages:
    763
    Likes Received:
    2,047
    Trophy Points:
    105
    Gender:
    Male
    Location:
    Virginia
    Karma sucks.... that's what you get for taking the forums down in your joke... :D
     
    mdsota152, Apr 1, 2017
    #7
    Time Lord, Roycestein Kaelstrom, Alley Oop and 1 other person like this.
  8. Terrence Phillip

    Terrence Phillip Avatar

    Messages:
    342
    Likes Received:
    742
    Trophy Points:
    43
    Gender:
    Male
    Does that mean TRAMUCCA is not happening?
    ;)
     
    Terrence Phillip, Apr 1, 2017
    #8
    Time Lord and Roycestein Kaelstrom like this.
  9. Widsith [MGT]

    Widsith [MGT] Avatar

    Messages:
    714
    Likes Received:
    646
    Trophy Points:
    93
    <emily_latella>Never Mind!</emily_latella>
     
    Widsith [MGT], Apr 1, 2017
    #9
    Time Lord, Fionwyn Wyldemane, Roycestein Kaelstrom and 1 other person like this.
  10. 2112Starman

    2112Starman Avatar

    Messages:
    3,613
    Likes Received:
    7,989
    Trophy Points:
    165
    Someone didnt configure the HA in your vSphere cloud correctly??? No server migration to working machines?
     
    2112Starman, Apr 1, 2017
    #10
    Time Lord, Alexander and Roycestein Kaelstrom like this.
  11. Roycestein Kaelstrom

    Roycestein Kaelstrom Avatar

    Messages:
    4,627
    Likes Received:
    10,229
    Trophy Points:
    153
    There is only one god in this universe... Actually goddess know as Lady RNG.
     
    Roycestein Kaelstrom, Apr 1, 2017
    #11
    Time Lord likes this.
  12. Duke Gréagóir

    Duke Gréagóir Legend of the Hearth

    Messages:
    5,685
    Likes Received:
    11,821
    Trophy Points:
    165
    Location:
    Dara Brae
    Not to be nickpicky - BUT there are 2 M's in TRAMMUCCA to preserve the legacy that is the greatness of TRAMMel ;)

    *hides*
     
    Duke Gréagóir, Apr 1, 2017
    #12
    Time Lord and Roycestein Kaelstrom like this.
  13. Traveller13

    Traveller13 Bug Hunter

    Messages:
    2,171
    Likes Received:
    2,717
    Trophy Points:
    153
    Hey, a topic for the R40 postmordem.
     
    Traveller13, Apr 1, 2017
    #13
    Time Lord, Brass Knuckles and Roycestein Kaelstrom like this.
  14. tphilipp

    tphilipp SotA Dev Moderator SOTA Developer

    Messages:
    535
    Likes Received:
    1,747
    Trophy Points:
    63
    You mean vSphere, the vm management tool that is bigger and more bloated than the OS plus the entire gameserver itself? Sorry, no... ;)

    But on the other hand, yes, we got a good illustration of HA stuff being certainly useful for the future. Stuff like this is planned (and already was, before this incident) for the launch, however so far our small team sacrificed some things b/c of so many things to do. So, I guess even better to gather some experience in that area now, than later after launch.
     
    Last edited: Apr 1, 2017
    tphilipp, Apr 1, 2017
    #14
    Time Lord, Elwyn, Hornpipe and 13 others like this.
  15. 2112Starman

    2112Starman Avatar

    Messages:
    3,613
    Likes Received:
    7,989
    Trophy Points:
    165
    My last gig was a data center engineer at a 50 million dollar federal nerc sip data center backed by Cisco UCS and Netapp that had 3000 VM's running vCenter with NSX. Id challenge you to think of a less "bloated" and reliable solution then VMware. We didn't have lil 100 gig VM's... we had the big ol monster SQL and Oracle servers running 64 virtual procs, 10 TB datastores, 256 gigs to 2 TB of ram each. Yes, for a small mom and pop organization running a few servers with say a dozen virtual machines and cloud integration, ya it can be bloated... but you just had an outtage and as a boss myself now, I would be asking the question why you were not running the industry standard enterprise virtualization platform. Just saying...
     
    Last edited: Apr 1, 2017
    2112Starman, Apr 1, 2017
    #15
    Time Lord and Lesni like this.
  16. 2112Starman

    2112Starman Avatar

    Messages:
    3,613
    Likes Received:
    7,989
    Trophy Points:
    165
    Gartners: 90% of business's who have a 2 week full outrage of their datacenter are out of business in one years time. Something for the boss's to think about.
     
    2112Starman, Apr 1, 2017
    #16
    Time Lord likes this.
  17. CatherineRose

    CatherineRose Avatar

    Messages:
    1,069
    Likes Received:
    4,030
    Trophy Points:
    113
    Gender:
    Female
    Location:
    Resting in Devil's Lair in Mistrendur
    When we raised our children we taught them not to play with fire because they could be hurt as well.
     
    CatherineRose, Apr 1, 2017
    #17
    Time Lord likes this.
  18. tphilipp

    tphilipp SotA Dev Moderator SOTA Developer

    Messages:
    535
    Likes Received:
    1,747
    Trophy Points:
    63
    I think you pretty much describe what we are not. Not sure what to say... we used to be all about the cloud, and slowly moved away from it, b/c for most of the time there is no need to run stuff on yet again some other people's computers, with their own set of issues. Just to cite a few (not making those up, real issues we had):
    - new instance comes up to help with load, but no other server can talk to it for a few minutes... turns out that that cloud's data center just had some issues getting their routes ready in time... feedback? No. So, stabbing in the dark... is it our setup, is it a bad instance, etc.. oh, suddenly, there it is.
    - box is down... actually... it's gone.... completely, and it won't come back either. Oops.. well, the hardware of the host died, nothing is there, no way for us to analyze
    - box suddenly is super slow... oh yeah, some other instance went crazy on it

    I fully understand that there are plenty of advantages for massive orchestration needs, as you describe your scenario, but you can also believe me that we gathered our share of experience in the cloud, with virtualized instances, with cloud managed dedicated machines, with services like RightScale, etc. and that we didn't blindly decide to not use what you call industry standard. We actually went through this and we know that for what we want and can provide, we are better off not relying on even more layers we don't have full control over.
    Also, we actually *still* do have vmware in use for a non-game-related network at Portalarium, and well... we are discussing to replace it, it's just not that important that we would spend backer money on it right now, b/c it's not blocking anything. Unfortunately vmware costs money, also, though...

    That said, yes, we just had an outtage in our alpha as you noticed, and it also came at a pretty bad timing right after release. Believe me, doesn't make me happy. It also happened at night at a bad timing to go out there, and we didn't have automated failover setup, yet, b/c we are a few people trying to make a game for 4 years on a budget that's a fraction of what you describe your last gig was, by itself. But you know, we learn from it. We aren't happy that this happened, we can do better, and the bosses surely realize that before launch we need to allocate some more resources on that. And for Portalarium, I think the biggest risk that would make us go out of business would be a bad final game. So we probably allocated more of our limited resources to that, lately, and crossed our fingers on our hardware. It's a struggle, and well... it's not easy, and often pretty taxing.

    Thanks for your input, though, it's good to be nudged by other people in the business, from time to time :)
     
    tphilipp, Apr 1, 2017
    #18
    Duke Olahorand, Time Lord, Zader and 19 others like this.
  19. Duke Gréagóir

    Duke Gréagóir Legend of the Hearth

    Messages:
    5,685
    Likes Received:
    11,821
    Trophy Points:
    165
    Location:
    Dara Brae
    We believe in you and all of Portalarium! :)
     
    Duke Gréagóir, Apr 1, 2017
    #19
    CatherineRose, Time Lord and Fionwyn Wyldemane like this.
  20. Brass Knuckles

    Brass Knuckles Avatar

    Messages:
    3,958
    Likes Received:
    7,707
    Trophy Points:
    153

    Got it Ill buy something in the addon store so you can get a second floppy drive..


    Kidding all that was way over my head.
     
    Brass Knuckles, Apr 1, 2017
    #20
    Time Lord, Hornpipe, Fionwyn Wyldemane and 3 others like this.
Page 1 of 3
Thread Status:
Not open for further replies.