[Linux] Frequent crashing attempting to allocate 1.8TB of memory.

Title: [Linux] Frequent Crashing attempting to allocation 1.8TB of memory.
Reproduction Rate: Random, roughly 80% if played long enough.
Blocker: Yes
Details: The game is frequently crashing (about 80% of games I play) with a bad attempt to allocate 1.8TB of memory.
Steps to reproduce: Play the game on Linux if you run into the bug, it will at some point the Unity Heap Memory Manager randomly decide to do a "huge allocation" well over the system memory, even extending beyond the Memory Mapped sparse buffer allocation for the entire uncompressed game data. In my case it attempted 1.8TB, on my system which only has 16GB (12GB~13GB available, ~8GB Free 2GB~3GB used, 1GB reserved for safety margin to stop badly behaved games like SOTA from halting my computer in OOMs.) This results in a short cycle of OOMs and null reference exceptions. (My first guess is that the insufficient memory reply hook is missing, the mem-pressure hook is missing or the Soft-OOM hook is missing. Most likely blame is on Unity as a generic should be attached by compiler and/or baselib if override isn't installed.)

Additional information:

System information lines from the initialization sequence log:

Tail of the log:

 

As a note, I will verify my game, remove my ULIMITs on memory and move the limits to sysconf which seems more reliable for the purpose of adjusting active memory headroom minimums, reboot, and try again.

Edit: verification found three bad files. Repaired the installation and will retry in the morning.
Will shut down my computer for now to minimize possible outside factors in the morning.

 

^ this? I thought we were on 2019.X of Unity???

 

@AoiBlue Off topic, are u using mesa_glthread on your system? This improves performance a lot on mesa and amdgpu
And are u using -force-vulkan to play SotA or just the default OpenGL?
As I did not yet have any OOM on my system while playing SotA.

 

I dont think we even tried 2019 of unity with sota yet..

 

i do not believe this is linux-specific.

https://www.shroudoftheavatar.com/f...ashing-while-going-around-mount-midir.154837/

 

Really, then I understood wronly on one of Chris his streams. He was saying about upgrading to latest Unity 2019.x because the 2018.x went LTS (Long Term Support) and that version only received fixes no new features anymore.
And I do recall them stating somewhere that Unity2019.x had some very good performance improvements.

 

yes the worry is every time you leap major version it seems to bork something, the last leap was 2018->2018.3 that borked enough I think.

 

Thanks for the confirmation.

 

It appears as a normal crash to desktop. I only found out it was OOM due to checking the log. (See log.)

I don't think I have mesa_glthread enabled. I know it can increase performance at risk to stability. I may try it later though.
I have tried -force-vulkan. Bottom framerate is notably higher and average framerate slightly higher for me but lots load slightly slower. Stability is roughly the same.
I also tried -force-glcore in the past, which does provide increased stability. I will try this again later.

I'm still not sure why they are shipping a EGL and GLES dummy library in the binary directory. I should try removing those. Last time I removed those it improved speed and stability. I suspect those libraries are in case people don't have EGL and GLES installed, because the game might crash looking for them in that case. A better solution to that would be to deactivate EGL and GLES support in the build settings.

As a note, the OOM bug is still present in the latest build. I'm still having trouble determining the exact rate of replication, but it is somewhere between 20-80%. Also, I cannot determine a single cause, at least not in game behavior, because of the issue.

Tail from latest version's log when crashing:

Code:

PushLevel: Added level RandomEncounter_Plains01_Road/Undead_HiddenVale, spawnPoint:PlayerSpawnLocation, hex:6009,6021
Network storage reset
NetworkingPeer: LeaveRoom
local player destroyed.
SceneServer: OnStatusChanged:Disconnect
PlotManager: SceneServer disconnect. Load status = Initialized
NetworkingPeer: LeaveComplete. Disconnecting.
Network storage reset
NetworkingPeer: Connecting to "38.140.214.157:5055"
Network storage reset
AsyncLoadSequence begin. SceneToLoad: (5cd5177dd5cc23baef92d5c6) RandomEncounter_Plains01_Road/Undead_HiddenVale:PlayerSpawnLocation
02:18:22: Loading Scene...
10
02:18:23: Level load loop started.
The referenced script (Unknown) on this Behaviour is missing!
(Filename:  Line: 195)

Size overflow in allocator.
(Filename:  Line: 98)

Could not allocate memory: System out of memory!
Trying to allocate: 18446744072016623423B with 16 alignment. MemoryLabel: String
Allocation happened at: Line:314 in
Memory overview

[ ALLOC_DEFAULT ] used: 3666396511B | peak: 0B | reserved: 4286067031B
[ ALLOC_TEMP_JOB ] used: 639443B | peak: 0B | reserved: 36700160B
[ ALLOC_TEMP_BACKGROUND_JOB ] used: 0B | peak: 0B | reserved: 1048576B
[ ALLOC_GFX ] used: 457567031B | peak: 0B | reserved: 636738087B
[ ALLOC_CACHEOBJECTS ] used: 537065824B | peak: 0B | reserved: 653464896B
[ ALLOC_TYPETREE ] used: 13717776B | peak: 0B | reserved: 16777216B
[ ALLOC_TEMP_THREAD ] used: 253324B | peak: 0B | reserved: 4030464B
Could not allocate memory: System out of memory!
Trying to allocate: 18446744072016623423B with 16 alignment. MemoryLabel: String
Allocation happened at: Line:314 in
Memory overview

[ ALLOC_DEFAULT ] used: 3666396511B | peak: 0B | reserved: 4286067031B
[ ALLOC_TEMP_JOB ] used: 639443B | peak: 0B | reserved: 36700160B
[ ALLOC_TEMP_BACKGROUND_JOB ] used: 0B | peak: 0B | reserved: 1048576B
[ ALLOC_GFX ] used: 457567031B | peak: 0B | reserved: 636738087B
[ ALLOC_CACHEOBJECTS ] used: 537065824B | peak: 0B | reserved: 653464896B
[ ALLOC_TYPETREE ] used: 13717776B | peak: 0B | reserved: 16777216B
[ ALLOC_TEMP_THREAD ] used: 253324B | peak: 0B | reserved: 4030464B

(Filename:  Line: 1161)

I noticed the line "Size overflow in allocator." This may have something to do with it.

 

I am guessing the kernel OOM protection is sacrificing the PID because of over allocating. Check your dmesg when the crash happens. Not that it matters tho, I am sure its a Unity bug and we will have to wait for a fix from the gods at Unity3D

 

Read how much it's trying to overallocate. It's an overflow value. Specifically the type that happens when you try to punch a negative integer into a positive integer.

There is even an error suggesting this might be the case.

The heap allocator had a total rewrite in 2019.1 I think switching to that major release will fix it.

I've played two times with -force-glcore and it hasn't happened. I am wondering if this fixes it over the auto section. It also seems to display some surface map detail shaders better (particularly procedurally generated psudorandom UV Maps and/or Tesseleration shaders such as ground shaders. (Don't know which is used, but the ground in novia actually looks to have a fine bumpiness to it instead of the smooth old-school fake-grass texture.)

 

Will give it a try, but as far as I know. The launcher "auto" selection should be the -force-glcore option.
Maybe the auto selection uses an older "safer" OpenGL context version. Try: -force-glcore45 or 44

Depending on your support for OpenGL versions of course.

 

i don't think there's a point in forcing 45 in going for an old stable version: unity doesn't support anything past that anyway.

 

They do support (OpenGL 3.2 to OpenGL 4.5): -force-glcoreXY: XY can be 32, 33, 40, 41, 42, 43, 44 or 45; each number representing a specific version of OpenGL. If the platform doesn’t support a specific version of OpenGL, Unity will fallback to a supported version.

But I did some experimenting with 43, 44, 45 and vulkan. They all render exactly the same.
Only that the performance on non vulkan is waaay slower no my system.

 

I think the reason GLCore seems to improve stability, performance and quality is the GL Compatibility context function of Mesa is based on standard specifications, and restricts certain functions that may be supported by other implementations. It also is known by the Mesa team to be lower performance the core context.

Back on the topic of the memory bug, I haven't experienced issues with the latest version. I think it might be fixed. It's hard to tell with these bugs that have low replication rate and unknown trigger.

 

I got the bug again, but the log tail was more useful this time:

Code:

PlotManager: Finished Initializing Player town plot
The file 'archive:/CAB-c36ed40140fdc8777594b0f8f116b398/CAB-c36ed40140fdc8777594b0f8f116b398' is corrupted! Remove it and launch unity again!
[Position out of bounds!]
(Filename:  Line: 214)

The file 'archive:/CAB-c36ed40140fdc8777594b0f8f116b398/CAB-c36ed40140fdc8777594b0f8f116b398' is corrupted! Remove it and launch unity again!
[Position out of bounds!]
(Filename:  Line: 220)

Size overflow in allocator.
(Filename:  Line: 98)

Could not allocate memory: System out of memory!
Trying to allocate: 18446744072899659278B with 16 alignment. MemoryLabel: String
Allocation happened at: Line:314 in
Memory overview

[ ALLOC_DEFAULT ] used: 3647659153B | peak: 0B | reserved: 4360693785B
[ ALLOC_TEMP_JOB ] used: 6162042B | peak: 0B | reserved: 67108864B
[ ALLOC_TEMP_BACKGROUND_JOB ] used: 0B | peak: 0B | reserved: 1048576B
[ ALLOC_GFX ] used: 576736105B | peak: 0B | reserved: 709071001B
[ ALLOC_CACHEOBJECTS ] used: 556534760B | peak: 0B | reserved: 670146576B
[ ALLOC_TYPETREE ] used: 13347984B | peak: 0B | reserved: 16777216B
[ ALLOC_TEMP_THREAD ] used: 54048B | peak: 0B | reserved: 4030464B
Could not allocate memory: System out of memory!
Trying to allocate: 18446744072899659278B with 16 alignment. MemoryLabel: String
Allocation happened at: Line:314 in
Memory overview

[ ALLOC_DEFAULT ] used: 3647659153B | peak: 0B | reserved: 4360693785B
[ ALLOC_TEMP_JOB ] used: 6162042B | peak: 0B | reserved: 67108864B
[ ALLOC_TEMP_BACKGROUND_JOB ] used: 0B | peak: 0B | reserved: 1048576B
[ ALLOC_GFX ] used: 576736105B | peak: 0B | reserved: 709071001B
[ ALLOC_CACHEOBJECTS ] used: 556534760B | peak: 0B | reserved: 670146576B
[ ALLOC_TYPETREE ] used: 13347984B | peak: 0B | reserved: 16777216B
[ ALLOC_TEMP_THREAD ] used: 54048B | peak: 0B | reserved: 4030464B

(Filename:  Line: 1161)

Receiving unhandled NULL exception
#0  0x00000000c95254 in operator new[](unsigned long, std::nothrow_t const&)
#1  0x00000000d7c960 in operator new[](unsigned long, std::nothrow_t const&)
#2  0x00000000d870e8 in operator new[](unsigned long, std::nothrow_t const&)
#3  0x00000000d87aaa in operator new[](unsigned long, std::nothrow_t const&)
#4  0x00000000d87bd9 in operator new[](unsigned long, std::nothrow_t const&)
#5  0x000000006cb0eb in std::pow(float, float)
#6  0x000000006cb5bc in std::pow(float, float)
#7  0x000000006bb209 in std::pow(float, float)
#8  0x000000007712b0 in std::pow(float, float)
#9  0x0000004039be29 in (wrapper managed-to-native) UnityEngine.AssetBundle:LoadAsset_Internal (UnityEngine.AssetBundle,string,System.Type)
#10 0x0000004037e78c in Portalarium.AssetBundles.AssetBundleManager/<MainInternal>d__39:MoveNext ()
#11 0x000000403882b3 in System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:InvokeMoveNext (object)
#12 0x000000402af424 in System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
#13 0x000000402bd6a0 in Portalarium.CooperativeThreadAsync:Update ()
#14 0x000000402bd1e8 in Portalarium.CooperativeThreadAsync/CooperativeThreadManager:Update ()

It reports a corrupted file as an issue. It seems that file is being rarely accessed. I'll go purge and re-download my files in case the checksums are missing it by statistical improbability of matching checksums.

I'll also purge my game directory with it's cache files to insure none of those are the bug.

 

In my experience lots load substantially slower in Vulkan but they create WAY less stutter.
(separate reply for drastically different subject)

 

So did you finally got it fixed by cleaning up?

 

Nope, still happening.

Latest one today right on first login.
This is clearly not normal behavior.

Code:

GameManagerSceneObject.Awake. 05/19/2019 23:21:06
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
NavMeshLoadTime SceneId=POT_island_metropolis_01_template , Time=0.4896851
Pathfinding: Load Loaded
<color=yellow>Ceto (1.1.1) Warning:</color> Can not query overlays clip mask if read/write is not enabled
Camera: CameraRig_Universal (CameraController)
requesting resize 1600 x 900
Using native desktop resolution 1600 x 900
requesting fullscreen 1600 x 900 at 60 Hz
Desktop is 1600 x 900 @ 60 Hz

Unloading 119681 unused Assets to reduce memory usage. Loaded Objects now: 284121.
Total: 1659.982000 ms (FindLiveObjects: 25.016000 ms CreateObjectMapping: 15.170000 ms MarkObjects: 642.402000 ms  DeleteObjects: 977.393000 ms)

Unloading 3 Unused Serialized files (Serialized files now loaded: 53)
GC.Collect() took 0.1 seconds
23:21:13: Level loaded.
23:21:13: Scenario init completed. Scenario:Diamond Fields
Checksum for POT_island_metropolis_01_template/Diamond Fields: 263779904

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 284112.
Total: 651.999000 ms (FindLiveObjects: 18.600000 ms CreateObjectMapping: 7.299000 ms MarkObjects: 625.773000 ms  DeleteObjects: 0.326000 ms)

Network storage reset
NetworkingPeer: Connecting to "38.140.214.157:5056"
Fallback handler could not load library /home/robert/.steam/steam/steamapps/common/SotA/Shroud of the Avatar_Data/Mono/libKernel32
Fallback handler could not load library /home/robert/.steam/steam/steamapps/common/SotA/Shroud of the Avatar_Data/Mono/libKernel32.so
Fallback handler could not load library /home/robert/.steam/steam/steamapps/common/SotA/Shroud of the Avatar_Data/Mono/Kernel32
Fallback handler could not load library /home/robert/.steam/steam/steamapps/common/SotA/Shroud of the Avatar_Data/Mono/libKernel32
Fallback handler could not load library /home/robert/.steam/steam/steamapps/common/SotA/Shroud of the Avatar_Data/Mono/libKernel32.so
Fallback handler could not load library /home/robert/.steam/steam/steamapps/common/SotA/Shroud of the Avatar_Data/Mono/libKernel32
23:21:14: Started NPC Spawn...
GroupServer: OnStatusChanged:Connect
SceneServer: OnStatusChanged:Connect
WARNING: Shader Unsupported: 'Standard (Specular setup)' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard (Specular setup)' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'Standard' - Pass 'META' has no vertex shader
Size overflow in allocator.
(Filename:  Line: 98)

Could not allocate memory: System out of memory!
Trying to allocate: 18446744072203380693B with 16 alignment. MemoryLabel: String
Allocation happened at: Line:314 in
Memory overview

[ ALLOC_DEFAULT ] used: 3519557983B | peak: 0B | reserved: 4229303159B
[ ALLOC_TEMP_JOB ] used: 2977117B | peak: 0B | reserved: 67108864B
[ ALLOC_TEMP_BACKGROUND_JOB ] used: 0B | peak: 0B | reserved: 1048576B
[ ALLOC_GFX ] used: 428187305B | peak: 0B | reserved: 583207393B
[ ALLOC_CACHEOBJECTS ] used: 494431824B | peak: 0B | reserved: 601283968B
[ ALLOC_TYPETREE ] used: 13344368B | peak: 0B | reserved: 16777216B
[ ALLOC_TEMP_THREAD ] used: 53568B | peak: 0B | reserved: 4030464B
Could not allocate memory: System out of memory!
Trying to allocate: 18446744072203380693B with 16 alignment. MemoryLabel: String
Allocation happened at: Line:314 in
Memory overview

[ ALLOC_DEFAULT ] used: 3519557983B | peak: 0B | reserved: 4229303159B
[ ALLOC_TEMP_JOB ] used: 2977117B | peak: 0B | reserved: 67108864B
[ ALLOC_TEMP_BACKGROUND_JOB ] used: 0B | peak: 0B | reserved: 1048576B
[ ALLOC_GFX ] used: 428187305B | peak: 0B | reserved: 583207393B
[ ALLOC_CACHEOBJECTS ] used: 494431824B | peak: 0B | reserved: 601283968B
[ ALLOC_TYPETREE ] used: 13344368B | peak: 0B | reserved: 16777216B
[ ALLOC_TEMP_THREAD ] used: 53568B | peak: 0B | reserved: 4030464B

(Filename:  Line: 1161)

Receiving unhandled NULL exception
#0  0x00000000bd12ac in operator new[](unsigned long, std::nothrow_t const&)
#1  0x00000000d7c960 in operator new[](unsigned long, std::nothrow_t const&)
#2  0x00000000d870e8 in operator new[](unsigned long, std::nothrow_t const&)
#3  0x00000000d87aaa in operator new[](unsigned long, std::nothrow_t const&)
#4  0x00000000d87bd9 in operator new[](unsigned long, std::nothrow_t const&)
#5  0x000000006cb0eb in std::pow(float, float)
#6  0x000000006cb5bc in std::pow(float, float)
#7  0x000000006bb209 in std::pow(float, float)
#8  0x000000007712b0 in std::pow(float, float)
#9  0x00000042045e29 in (wrapper managed-to-native) UnityEngine.AssetBundle:LoadAsset_Internal (UnityEngine.AssetBundle,string,System.Type)
#10 0x000000420453fc in Portalarium.AssetBundles.AssetBundleManager:LoadResource<T_REF> (ShroudDatabase.AssetId)

Edit:
I think I spotted the bug (or a bug at least): In the log there is the unanticipated event of unloading zero assets. This might be causing the unexpected actions by the allocation code.

I suspect placing a catcher shim on this routine to just return without passing to the underlying routine in the event of nothing to do, should work around the bug. (Seriously, Unity should get their s*** together and fix that.)