[FEATURE REQUEST] Line numbers on Out Of Memory errors

[Dave Gilbert]

Hello! So I've been getting a lot of "Out of memory" errors when testing my game lately. Like this:



They are easy enough to fix once I narrow down where in the script the error occurs, but the error message doesn't give the script and line number where the crash occured. So finding that spot can take much longer than it would for a traditional crash. Is it possible for these error messages to spit out where in the script the crash happened?

Thanks!

-Dave

[Crimson Wizard]

Is it possible for these error messages to spit out where in the script the crash happened?

I think it may be possible to print the last known script location, but that won't necessarily help, as the "out of memory error" may occur virtually anytime, such as during game update or render, and not the script execution. Also, some actions in AGS are scheduled and postponed until after current script finishes.

For instance, PP=379 is in the middle of GUI redraw.

[Dave Gilbert]

I see your point, but in most cases (or at least, my cases) the crashes happen consistently at the exact same spots, which leads me to believe that knowing the script position could help pinpoint why it happens.

[eri0o]

8318416=2079604*4=1922*1082*4

It looks like it's a GUI that is the size of the screen that has a nonzero transparency is the specific case reported, but I wonder how is the memory usage in general.

[Crimson Wizard]

The problem with the "out of mem" error, is that when it happens nothing is safe anymore. The state of the program objects may be invalid, and any tiny extra allocation may fail in these circumstances.

Looking at the code now, it seems that I'd have to rewrite all the code related to getting the script location again, because at the moment it also allocates memory for the strings which store function call lists, etc.

But then, asking scripts about their current locations may also not be reliable and lead to a program crash, because "out of mem" could have happened while script was being created, or was allocating something during its run.

EDIT: I guess, theoretically this may be done by having a global preallocated buffer, and store this information there, updating as the game and scripts run... Then such buffer may be used when reporting error.

[Dave Gilbert]

8318416=2079604*4=1922*1082*4

It looks like it's a GUI that is the size of the screen that has a nonzero transparency is the specific case reported, but I wonder how is the memory usage in general.

That's exactly when it happens (when a fullsized GUI's background graphic is changed). I'm curious what those numbers mean though. I see "1922*1082". Does that refer to the GUI's size? It's *slightly* bigger than the game's resolution of 1920x1080. Could that be causing it? I checked the GUI and it's the correct size of 1920x1080.

The problem with the "out of mem" error, is that when it happens nothing is safe anymore. The state of the program objects may be invalid, and any tiny extra allocation may fail in these circumstances.

But then, asking scripts about their current locations may also not be reliable and lead to a program crash, because "out of mem" could have happened while script was being created, or was allocating something during its run.

Hm. This sounds like the problem goes deeper than I realized. Maybe I'm misunderstanding how this works. I feel like if it there was a memory leak, the crash would happen at random spots. But my my case it consistently happens at the same point. I'm making the assumption that I somehow caused a memory leak AT that specific spot, so knowing the location could help find the cause. But you know what they say about "assume!"

[eri0o]

Uhm, I talked a bit with CW yesterday, I think the easiest way right now would be to run the game under the VS debugger. With it we can see how AGS is allocating ram across things to see what's going on - why we get this Out of Memory, maybe something else is using the RAM.

About adding more information when things like this happen, I don't know if CW had any additional thoughts.

[Crimson Wizard]

I feel like if it there was a memory leak, the crash would happen at random spots. But my my case it consistently happens at the same point.

Both situations are possible. If in your case it happens exactly at the same spot, that, in my opinion, might mean one of a few things:
1) In that spot you do some actions that allocate lots of memory at once. For example, create dynamic sprites or dynamic arrays in a loop.
2) Your game constantly requires a lot of memory, but in that particular case it allocates bit more and crosses the limit.
3) There's some bug in the engine, and you do something that triggers it, which in turn leads to engine allocating abnormal amount of memory.

How exact is the "point"? Is it certain interaction, a dialog, a cutscene?

EDIT: On a side note, I might remind again, that Windows version of the engine is still distributed as 32-bit, limiting the RAM usage to around 2 GB (iirc). If your game requires more memory under normal circumstance (if that is not a mistake), then it might need a 64-bit version of the engine.

[Dave Gilbert]

Uhm, I talked a bit with CW yesterday, I think the easiest way right now would be to run the game under the VS debugger. With it we can see how AGS is allocating ram across things to see what's going on - why we get this Out of Memory, maybe something else is using the RAM.

About adding more information when things like this happen, I don't know if CW had any additional thoughts.

Pardon my ignorance but what is VS Debugger? I feel like this issue is beyond the scope of this thread so feel free to DM me.

[eri0o]

So, Dave was really kind to make the game available for me and I (and CW) have been playing with it.

This message is more to start writing a bit on the Engine side to try to figure how to document the things I have been trying so far.

I did find a way to reproduce a different OoM error (different EIP) locally by having the spritecache been set to 2GB limit. Because AGS is a 32-bit application in Windows, the memory is limited is 2GB total for the application, so it will eventually try to allocate a new sprite and crash. One recent idea that I had, is to see if there's some way to know the memory limits we have (Maybe this?) and dynamically lower the memory available in spritecache, but this may be to exagerate and also too Windows specific. And the possibility of simply making AGS 64-bit in Windows, is indeed a thing that would fix.

Now, I tested a bunch of tools in this meantime trying to find memory problems in AGS leakwise, and so far there is no problem that I could find that happens at runtime. I did found out a bunch of things AGS doesn't clean up after it has quit, but this is not necessarily a problem - at least when running in a plataform that has an OS. The bellow is from Memory Validator (I got an evaluation license).

Spoiler

[close]

Now, I used MS VMMap to try to make sense of the memory. In this case, I set the spritecache to 512MB, which looks like enough to run the game smoothly - we have 138 MB of locked sprites, so this gives us some room to work with, but also keeps consumption low.



Now, you can see that the task manager shows the process is using 793.1 MB, but VMMap shows a different memory story.



Here you can see that the Total Working Set in the Heap memory in VMMap is closer to the memory shown in the Task Manager, but there is some additional memory going on.



By keeping the spritecache at 512MB, I managed to play the game with at most around of 1.2GB in total memory consumption. I couldn't play it for a really long time with it, because it, ironic, VMMap itself uses RAM instead of writing the logs on disk, so at some point VMMap will complain of lack of memory, but after closing it I can switch between rooms in the game for seemingly forever time.

Any additional ideas reggarding memory or tools ideas are welcome. So far I tested Intel Inspector, AMD uProf and Very Sleepy, beyond Memory Validator, VMMap and also the default debugger in VS.