Dismiss Notice
This Section is READ ONLY - All Posts Are Archived

[Known] BSOD Crash after Shoud.exe consummed 18GB of Memory. Leaking Memory.

Discussion in 'Critical Issues (Blockers, Performance, Crashes)' started by John Striker, Dec 13, 2018.

Tags:
Thread Status:
Not open for further replies.
  1. John Striker

    John Striker Avatar

    Messages:
    27
    Likes Received:
    11
    Trophy Points:
    3
    Gender:
    Male
    12/13/2018 14:51
    Title: BSOD Crash after Shoud.exe consummed 18GB of Memory. Leaking Memory.

    Reproduction Rate:100% Should.exe is consuming much more memory than normal over time.

    Blocker? Yes! Crashed BSOD with Windows Resource Error. :eek:

    Details: While teleporting my system, with 24GB of RAM, ran out of resources and Crashed to the Windows 10 blue screen!:( This is the first time I have ever seen this error. I did visit a lot of places to load them up for zone scrolls. The crash happened in Lord British's home in Brit while teleporting to Krampus. After the computer rebooted and restarting the game I went to kill Krampus and the memory consumed by Shoud.exe grew to 14GB which is higher than I ever seen. Normally RAM usage is around 6 to 8GB RAM. The more I play the more RAM shoud.exe uses. I did not see this problem of high memory use when on the test server for the last few days so it is odd to see this problem on release unless there was a last minute change.

    This is the event at the time of the crash:
    Windows Event Log error: Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Shroud of the Avatar.exe (7584) consumed 18247000064 bytes.

    Memory Usage tests I ran: The Commit memory keeps rising.
    This is the starting RAM Usage after playing for an hour after the crash: 13.5GB
    Image PID Hard Faults/sec Commit (KB) Working Set (KB) Shareable (KB) Private (KB)
    Shroud of the Avatar.exe 648 0 13,576,596 4,818,352 90,400 4,727,952

    I teleported to a swamp killed a few mobs and then back to Central Brit and memory climbed to 14.5GB
    Image PID Hard Faults/sec Commit (KB) Working Set (KB) Shareable (KB) Private (KB)
    Shroud of the Avatar.exe 648 0 14,421,708 4,696,136 91,080 4,605,056

    Then visited a few more cities: 14.6GB used
    Image PID Hard Faults/sec Commit (KB) Working Set (KB) Shareable (KB) Private (KB)
    Shroud of the Avatar.exe 648 0 14,626,284 5,334,552 94,372 5,240,180


    Steps to Reproduce: Just normal game play will use up memory over time. Combat with mobs in the world does seem use more memory faster.

    User Specs:
    OS: Windows 10 (10.0.0) 64bit
    CPU: AMD FX(tm)-8350 Eight-Core Processor (8) System RAM: 24524
    GPU: NVIDIA GeForce GTX 1050 Ti GPU RAM: 4018
    SotA.Win.64.882.Date.12.12.18
    Area: POT_island_metropolis_01_template/Justice Hold
    Area Display Name: Justice Hold
    Loc: (-3.2, 89.0, -49.8)
    Debug: UE9UX2lzbGFuZF9tZXRyb3BvbGlzXzAxX3RlbXBsYXRlfEp1c3RpY2UgSG9sZHwoLTMuMTc5LCA4OS4wNDksIC00OS44NDEpfCgwLCAtMC42OTksIDAsIC0wLjcxNSl8LTI3OC42OTgyfDIzLjM3MDE5fDMuNzU2NzU1
     
  2. John Striker

    John Striker Avatar

    Messages:
    27
    Likes Received:
    11
    Trophy Points:
    3
    Gender:
    Male
    After some more playtime here is the current memory shroud is using.
    Now up to 16.5 GB. I will restart the client to avoid another crash.

    Image PID Hard Faults/sec Commit (KB) Working Set (KB) Shareable (KB) Private (KB)
    Shroud of the Avatar.exe 648 0 16,505,576 3,942,508 129,932 3,812,576
     
  3. Chris

    Chris Tech Lord Moderator SOTA Developer Ambassador

    Messages:
    2,403
    Likes Received:
    27,403
    Trophy Points:
    190
    Gender:
    Male
    We are currently aware of a bug that is causing some nasty memory explosions. This is sadly a bug in the Unity engine that we have reported with a repro project. We can't fix it but we do have a work around if we can identify the asset causing the issue. I'll open a bug to have someone look at the Justice Hold map to see if we can identify the problematic art piece.
     
  4. Chris

    Chris Tech Lord Moderator SOTA Developer Ambassador

    Messages:
    2,403
    Likes Received:
    27,403
    Trophy Points:
    190
    Gender:
    Male
    Unity just released a patch this morning that we believe is the fix for many of our memory growth problems. We'll be trying to get a build out with it ASAP.
     
  5. John Striker

    John Striker Avatar

    Messages:
    27
    Likes Received:
    11
    Trophy Points:
    3
    Gender:
    Male
    The crash did not actually happen in Jusice Hold but in Central Brit. I ran back to the spot and created another debug key to Central Brit at the lower floor of the castle where Ronin, the Krumpus quest giver, is located. Central Brit used over 1GB of memory when I teleported into the city.

    I did some more testing on how the Unity Engine was allocating memory to Windows 10 Committed Memory pool. While recalling or teleporting the memory use can spike as much as 1GB while loading the instance but Unity never releases all of the memory it committed with each instance change. So, when Windows runs out of physical and virtual memory resource while recalling, zoning, or teleporting then Windows will crash the client or cause a BSOD which was recorded in the Windows 10 Event Viewer message that I posted. This is probably the cause of all manner of "crashes" when ever people zone in and out of instances since the instance change requires enough free or virtual memory to complete. Yesterday I visited 20+ instances to build my Zone Teleport list which was not a normal day of play which is why I have not seen this crash before but it is very repeatable.

    The updated Unity engine is never releasing all the memory it has Committed to the Windows 10 Committed Memory pool which then just grows until the client/system will eventually fail. On systems with very little physical memory this problem could also be the cause of some of the "hitching" seen by players since, as you know, Windows/Linux will create "virtualized/Swap file" RAM which it writes to a hard drive which is orders of magnitude slower to access than Free physical RAM.

    This problem can be managed by monitoring memory usage and restarting the client when the system gets close to using all the memory but that is not an optimal solution.

    I hope this input is helpful.

    This is the actual debug location of the original crash BSOD.

    12/14/2018 08:16
    Title:Location of original crash.
    Reproduction Rate:
    Blocker?
    Details:
    Steps to Reproduce:
    User Specs:
    OS: Windows 10 (10.0.0) 64bit
    CPU: AMD FX(tm)-8350 Eight-Core Processor (8) System RAM: 24524
    GPU: NVIDIA GeForce GTX 1050 Ti GPU RAM: 4018
    SotA.Win.64.882.Date.12.12.18
    Area: Novia_R3_City_Brittany
    Area Display Name: Central Brittany
    Loc: (210.9, 61.6, 81.0)
    Debug: Tm92aWFfUjNfQ2l0eV9Ccml0dGFueXx8KDIxMC44OTksIDYxLjYyMiwgODEuMDI2KXwoMCwgLTAuNDM5LCAwLCAtMC44OTkpfC0zODcuNjgyMXwyMi4zMzkwN3wzLjMxNTYyNw==
     
  6. John Striker

    John Striker Avatar

    Messages:
    27
    Likes Received:
    11
    Trophy Points:
    3
    Gender:
    Male
    Great to hear that Unity has a patch after I wrote my last long post! :)

    There was much rejoicing throughout the kingdom!:D
     
Thread Status:
Not open for further replies.