Dismiss Notice
This Section is READ ONLY - All Posts Are Archived

Server Lag due to final wipe player increase

Discussion in 'Release 32 Feedback Forum' started by SabeSr, Aug 6, 2016.

Thread Status:
Not open for further replies.
  1. SabeSr

    SabeSr Avatar

    Messages:
    459
    Likes Received:
    916
    Trophy Points:
    55
    Gender:
    Male
    Location:
    Paradise, Florida
    As I stated before, you would receive a large influx of players and you needed to be sure you had sufficient server hardware to handle it or you would have lag issues... you are having them now.

    7P's

    Prior Proper Planning Prevents Piss Poor Performance

    Fix your hardware stability to handle twice what you think you need.

    If I am paying money (as are all) to play a game I want to play it not sit waiting for the server to respond.

    You focus players into small areas having crafting/banking/vendor/trainers in one central location add more to the cities... I understand small towns only having one but cities should all have a few.

    I would have made several market/crafting/bank areas in Adoris north south east and west (just one example)

    It's nice to gather people in one area to promote social aspect of game but in order to do so you need a stronger server hardware setup.

    The lag is not occurring in low pop areas its occurring when in a city in market areas.

    Buy more hardware, stronger... you have the money. Rough guess at your budget...30 programmers $50k a year... thats $1.5million a year... been around what? 3 years... thats $4.5million for payroll so far... according to main page you've received twice that... $4.5million for hardware... you should have some of the best and baddest hardware around. Of course my numbers are probably off but thats just a quick guess at what you got... spend it.

    Need me send more money? We will take up a pool and all send you some more if you promise to buy better hardware.
     
    AvatarGG and Time Lord like this.
  2. Cassandra Corax

    Cassandra Corax Avatar

    Messages:
    679
    Likes Received:
    2,194
    Trophy Points:
    105
    Gender:
    Female
    Location:
    Rhondda Cynon Taff South Wales United Kingdom
    For me the only place I found impossible to enter and walk about is Owls Head. Brittany central to my surprise is running fine. That is me playing from the uk
     
    Leelu, Dhanan and Time Lord like this.
  3. Time Lord

    Time Lord Avatar

    Messages:
    8,336
    Likes Received:
    28,405
    Trophy Points:
    165
    Gender:
    Male
    Location:
    ~SOTA Monk~ ~Monastery~ ~Thailand~
    I'm feeling a bit awkward in saying this because so many people are saying that they have lag, But....

    "I live in the center of Thailand were there's only busses in or out"... and thus far,

    "I've never had lag":confused:
    Odd but true ...o_O
    ~Time Lord~:rolleyes:
     
    Last edited: Aug 6, 2016
  4. Dhanan

    Dhanan Avatar

    Messages:
    116
    Likes Received:
    203
    Trophy Points:
    30
    Gender:
    Male
    Location:
    Toronto, Ontario, Canada
    It's so much more than just hardware. Optimisation takes time. Need I remind you (and it seems like I do), the game is still in development. Also, your perception of their budget doesn't take into account so very many factors (building rent, for one). Also, the big cities do have more than one designed area of congregation. Everyone and their mother knows about high population lag issues. It's an issue that gets optimised over time. I've backed a few early access games now, and something I've learned is server optimisation takes time, hard work, and creativity. Sometimes it is even necessary to take a step back before you move forward.

    The huge influx of players stressing the servers is all part of the development process. And for the record, I actually don't have lag issues in either of those cities.

    Maybe you should upgrade YOUR hardware.
     
  5. r-DotPhunky

    r-DotPhunky Avatar

    Messages:
    80
    Likes Received:
    233
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Behind you! *BACKSTAB*
    Hardware? In today's world this is all done via virtual boxes... instances of servers that "spin up" when the demand (IE players) dictates it... and ramps down when the demand decreases.

    It's all handled automagically. There's no concept of "buy more hardware!" these days.

    That said... Owl's Head is the only town I have a tiny bit of an issue... but that's because everyone has made their lawns look like the living room of a 65 year old with an obsession for knick-knacks.

    It would be helpful if you shared what kind of PC you are on... processor, memory, graphics card etc. Odds are you could use an upgrade somewhere.
     
    EclipseMaiden and Time Lord like this.
  6. Time Lord

    Time Lord Avatar

    Messages:
    8,336
    Likes Received:
    28,405
    Trophy Points:
    165
    Gender:
    Male
    Location:
    ~SOTA Monk~ ~Monastery~ ~Thailand~
    @r-DotPhunky could be very right!
    [​IMG]
    I play from the remote center of Thailand,
    "And I've Never Had Lag"
    Odd fact but true... o_O
    ~Time Lord~:confused:
     
  7. Black Tortoise

    Black Tortoise Avatar

    Messages:
    1,961
    Likes Received:
    3,655
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Storm's Reach
    Yeap, sorry youre so laggy, and sorry for you that aggressive language, fear mongering, and baseless accusations are the only way you choose to get your point across.

    No lag whatsoever here! There is some client side lag in the big cities (which has always been the case), and that is really noticeable on Linux and not nearly as noticeable on Windows. Packet-wise, theres no more lag than there is doing anything else I do on the internet. Also, my ISP sucks. No problems whatsoever communicating back and forth with Port's game servers.
     
    Time Lord likes this.
  8. Dhanan

    Dhanan Avatar

    Messages:
    116
    Likes Received:
    203
    Trophy Points:
    30
    Gender:
    Male
    Location:
    Toronto, Ontario, Canada
    Virtualisation is irrelevant. It's still all hardware in the end. I wasn't aware that Portalarium had outsourced to a server farm. Is it Amazon's? In either case, it costs money to spin up extra servers. If you own them, then the cost is of the equipment and energy. You also have to take into account the lifetime value of the hardware. Sometimes it's better to rent the hardware, rather than buying it outright. If a farm owns the servers, it is still going to cost money to spin up additional servers. That is why optimisation is very important.
     
    Time Lord likes this.
  9. DarkStarr

    DarkStarr Executive Producer SOTA Developer

    Messages:
    6,692
    Likes Received:
    45,561
    Trophy Points:
    202
    Gender:
    Male
    None of what you are experiencing is Server Lag. Our servers are consistently operating at below 10% capacity. No amount of hardware will solve the issue as you suggest.

    It is all client side and networking messaging issues which we are patching literally daily. As noted in the release notes optimizations are ongoing.
     
    Acred, AvatarGG, Cianna and 15 others like this.
  10. Chris

    Chris Tech Lord Moderator Ambassador SOTA Developer

    Messages:
    2,470
    Likes Received:
    27,551
    Trophy Points:
    190
    Gender:
    Male
    So just to add to that, we just put in a number of fixes to load times. Below is the chart from the past 12 hours showing the mean (average) load times for all non-player owned towns in the game. These are the worst 50:[​IMG]

    So currently I'm investigating the Wharfs as it seems there is something seriously wrong there. The others are down but still not acceptable in our opinion. We still have more optimization to do but that will continue to come over time. The biggest improvements will come with us changing up our build process to better organize data and more intelligent caching of frequently used assets. The other big improvements will come with more work on POTs and data serialization. We've made some great progress but we aren't pretending to say it is fast enough yet.

    Since you seem to be suspicious of our servers, currently we have 12 nodes installed, each with 48GB RAM and dual 3.7ghz hex-core Xeons, raided SSDs on hardware raid, and SFP, 7gb/s fiber networking. If we need them, we have another 32 nodes we can install.
    We use 3 for the DB. Really one is the DB, one is a replication DB so we can do queries without bogging down the main box, and the third is the auto failover DB. They run at something like 1% load.

    Then there is 1 box running the ELK stack for our internal metrics and monitoring. That is where the pretty chart above came from. We can generate live charts on game systems in a second to track what is actually going on.

    Then the game server runs on a single box. The max concurrency I've seen is around 13% and usually it runs at 5%. The server is threaded so it can take advantage of all 24 of the 3.7ghz hyper-threads on the box AND it also clusters, meaning it can run on multiple boxes. IF we ever see significant load, we will split things out and switch to clustering.

    While our web server is in the cloud, our game servers are in a co-location center with double redundant power and 6 internet backbone connection. We went with that one, for cost. We stumbled into some amazing hardware for free and the center we're hosting at has local cloud boxes should we over flow the 44 nodes with 1056 3.7ghz hyper threads and 2 tb of RAM but that seems unlikely we would need that.

    In short, what Starr said. It isn't the servers... it is the client that we will continue to improve on.
     
    Fringe, Acred, Wilfred and 24 others like this.
  11. Time Lord

    Time Lord Avatar

    Messages:
    8,336
    Likes Received:
    28,405
    Trophy Points:
    165
    Gender:
    Male
    Location:
    ~SOTA Monk~ ~Monastery~ ~Thailand~
    Great Work Portalarium Guys! Thanks :D!'
    [​IMG]
    ~Time Lord~:)
     
  12. Oba Evesor

    Oba Evesor Avatar

    Messages:
    974
    Likes Received:
    2,188
    Trophy Points:
    105
    Gender:
    Male
    Location:
    S.F.

    Don't engineer me bro.


    :p
     
    Fringe, Acred, Leelu and 6 others like this.
  13. Alex_Wildeagle

    Alex_Wildeagle Avatar

    Messages:
    145
    Likes Received:
    83
    Trophy Points:
    18
    From what I've seen the worst lag is lots. For instance Soltown, I apear at the entry and run no problem. But the last 2 turns as players lots start loading is where a major LAGfest begins. The folks with the lot next to the village walls is a major factor as they have loads of junk just laying out and the computer goes nutz trying to load it all.

    As to it not being server, hmmm? I've done multiple trace routes from my PC and I never get above 35 ms delay on a 150meg connection. So if it ain't your servers it might be your provider or your load balancing hardware/software.
     
    Time Lord likes this.
  14. Roycestein Kaelstrom

    Roycestein Kaelstrom Avatar

    Messages:
    4,627
    Likes Received:
    10,229
    Trophy Points:
    153
    @Chris could you also check Celetis while you're at it. Several players have experienced severe lag to the point that the client froze up to minutes. I suspect it may be related to NPC path finding since they started to run at lightning after players regain control.
     
    Alley Oop and Time Lord like this.
  15. Roycestein Kaelstrom

    Roycestein Kaelstrom Avatar

    Messages:
    4,627
    Likes Received:
    10,229
    Trophy Points:
    153
    Could be the rendering on the client side. The server does not send polygon in binary in the packet. Instead, it sends the info on what items need to be rendered and the client is responsible to render it.

    Regardless of the root cause, there are still many things that need to be optimized.

    It would be more productive if players report the areas that give them lag as many have already done. Trying to point finger wheter it's server or client isn't helpful when it comes pinpointing the problem and fixing it.
     
    Last edited: Aug 6, 2016
    Alley Oop, Time Lord and Baalice like this.
  16. Alex_Wildeagle

    Alex_Wildeagle Avatar

    Messages:
    145
    Likes Received:
    83
    Trophy Points:
    18
    Makes sense, but why does it take SOTA so much longer then EQ2, MWO and others that I just don't have near as bad issues with?
     
    Time Lord likes this.
  17. Link_of_Hyrule

    Link_of_Hyrule Avatar

    Messages:
    1,394
    Likes Received:
    2,142
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Ogden, Utah
    in the end it's Unity and how it pans a lot of stuff over to the CPU some of these issues should be resolved once they switch over to Unity 5.4 which brings cpu usage down and frames up quite a bit. Also a lot of areas stop bed heavy optimization like Owls Head and Brittany. I get like almost 60 fps in bright bike pass but in some of the major cities it's crap like 15 fps. Also i have huge frame drops caused by something local in the town's I'll be at 30 fps and drop to 5 I'm really hoping they figure out what's causing that age fixing it.
     
    Time Lord likes this.
  18. Anvar

    Anvar Avatar

    Messages:
    425
    Likes Received:
    613
    Trophy Points:
    43
    Because all of those are more finished products and have been optimised, also to my knowledge they dont have customisable housing within a town. Put simply every object on a lot has to be syncronised and the info sent to your client and loaded.
    The more housing and housing decos the more youll have to load.
     
    Time Lord and Baalice like this.
  19. Link_of_Hyrule

    Link_of_Hyrule Avatar

    Messages:
    1,394
    Likes Received:
    2,142
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Ogden, Utah
    Those games cost like $100 Million Dollars SotA has $9.5M in funding but needs way more.
     
    Time Lord and Baalice like this.
  20. DarkStarr

    DarkStarr Executive Producer SOTA Developer

    Messages:
    6,692
    Likes Received:
    45,561
    Trophy Points:
    202
    Gender:
    Male
    Because those are complete games that have been optimized. As we have clearly stated multiple times we are still optimizing
     
    Duke Albert, Acred, Saosis and 6 others like this.
Thread Status:
Not open for further replies.