Problem with saveLocalData Freezing and Crashing

I have a program (one with quite a few lines of code at this point… about 14,000 lines in 81 classes) which is freezing when calling saveLocalData(). It works fine the first time saveLocalData() is called, but freezes for several minutes until crashing on a subsequent call.

The “use case” is basically that the user can save and load game maps. The save involves three calls to saveLocalData(), one to save the map itself, one to save the updated list of maps, and one to save a thumbnail image for the map. Creating a new game map and saving it works fine when you first run the program, but any attempt to save another map causes the aberrant, fatal behavior. The program freezes for about two minutes, and ultimately crashes Codea entirely (not just the running Lua program, but all of Codea).

I’ve tried to isolate this to a problem in my own code. I call the immediate belief that someone else’s code is responsible “black box syndrome”, i.e. the idea that the problem must be in whatever part of the code the programmer has no insight into or control over, but I’ve worked very hard to be sure that it is not my doing, and it clearly is not (unless I’ve missed something, like a call to the system funciton pleaseDoNotCrashWhenSaving(true)). If I simply comment out all calls to saveLocalData(), it runs fine (but, of course, without actually saving anything). If I put print() statements in to trace where the code is hanging, I get inconsistent results, depending on where I put the statements, so I can’t even localize it specifically in the call to saveLocalData(), but I imagine this is an artifact of buffering with print to the console.

My only thoughts are either that saveLocalData() is not properly closing opened resources, so it is running out of memory, or else some side effect of the fact that my saveLocalData() calls are being made within a coroutine (so that I can show a “saving” animation while it works). I don’t think the coroutines are the problem, though, because I commented that out to let it run “in the main thread”, and that had no impact.

At the moment, the only workaround I can see to this problem (barring a fix in Codea) is clumsy; use raw file io operations to read and write the files myself. I really don’t want to have to do this, but I may do it, just to guarantee that it works, and so the problem really is in saveLocalData(). I’m busy writing a reusable wrapping-scrolling-editable text-box widget, however, and that is far more fun than coding up low level IO read/write calls.

[BTW, I’ve started a blog to outline my adventures in Codea, and more importantly to share a lot of the reusable code I’ve written, such as my class serializer and my UI components, but I haven’t yet created a GitHub project for it, or even found time to write meaningful posts. It’s so much more fun to write code than to write about writing code. I’ll post a link to the blog when I get something tangible out on it.]

@blacatena You say you do 3 calls to saveLocalData. How much data are you saving with each call. Can you comment out 2 of the saves and things work OK or does one of the saves crash while the other 2 work OK. Have you tried doing a collectgarbage() call after each save. Usually Codea crashes when there’s a memory problem and you do have a lot of code.

I did try garbage collection, as well as commenting out the saves. The one that fails is the rewrite (the one that writes over the list of maps). I should have mentioned that in the OP. That’s why I think it may be a failure to close a resource, because it fails on rewriting something written previously in that same execution (but not in a prior execution).

Actually, I did try garbage collection before I’d further isolated the specific saveLocalData() problem. Perhaps tonight I’ll go home and wrap that in two garbage collection calls, one before, and one after, just to see if it helps.

@blacatena Maybe you could try saveLocalData(“maps”,nil) to clear the map data, do a garbagecollect, then do a saveLocalData for the map data.

That’s a good idea. I’ll try that as well.

Okay, so… neither thing entirely worked (adding garbage collection or “erasing” by writing nil before the real write), in that Codea would still freeze-up for 2-4 minutes, but it didn’t crash every time. The problem does seem to get worse with more frequent saves. I added some timing-capture code, and it’s odd, because it doesn’t seem to freeze within the save, the lag comes between saves, but the save itself definitely causes the problem (if I comment out the three saves then everything is fine). It’s almost as if garbage collection or something goes crazy after the save completes. I wish I had a proper debugger!

But as a result, I’m not really sure which saveLocalData() call is the issue, or if it’s a side effect of any of them, or a behavior due to low memory. There’s nothing else peculiar about the code, either, and I don’t feel like I’m using that much memory (but again, without a debugger and better tools, I can’t really know).

Sorry if some of this is confusing, but I am getting very inconsistent results from doing different things. There’s no logic to this. Sometimes one thing seems to be the culprit, sometimes another. All I know is that it is definitely a saveLocalData() problem, because if I comment out all of those calls, it’s fine and instantaneous.

@blacatena Try this. Write another program that all it does is readLocalData and saveLocalData of your 3 data. Stick it in a for loop starting with 1 for loop count and keep increasing the for loop count and see what that does.

I already did that before posting, although I couldn’t add in my data; that’s complicated. I tried writing a program that simply kept randomly rewriting large chunks of data to any of 3 locations, and I couldn’t get it to fail. It’s clearly something specific I’m doing somewhere in my larger code, which triggers a bug in how saveLocalData() is implemented in Codea, but without a debugger I’m stuck.

Maybe my next course needs to be to work on it in XCode, where I can better manage it, but at least for now, that’s hard to do. The iPad is great because I can tinker from time to time with it. I have a full time job as a software architect, so I don’t have tons of time and energy left over to do even more coding, let alone this sort of intricate debugging.

if you save much data then try wrapping it into a coroutine and save in chunks.

@blacatena I reread your first discussion above and I have a question. In the forth paragraph you mention a coroutine. Are your saveLocalData calls within a coroutine. What are the sizes of the data you’re trying to save.

@blacatena I tried doing save and read local data with a 25 MB string. I was able to do a 20 count for loop with saveLocalData and each save took about .09 seconds. I then tried a single readLocalData and Codea immediately crashed all the way out. I thought that there might be a problem with readLocalData and a large file, but when I did a clearLocalData and tried everything again, there wasn’t any problems. I was able to a readLocalData as many times as I wanted. I then tried it again and when I tried a readLocalData, Codea hung for about 40 seconds and then crashed totally. So there’s a problem going on, but I can’t say what.

I’m pretty sure the coroutine isn’t the problem. As I think I said, I commented out the coroutine aspects, so the code simply ran inline, and it still happened.

The size of the data varies… the largest, the map itself, is a few megabytes, maybe ten. I have to put some code in to measure it, but these days, a few megabytes for a file is nothing. The thumbnails are basically the pixels of a 300x300 image, converted to JSON-rgb pixels. Maybe a megabyte? The one that seems to be the problem, the list of maps, is fairly small, just a list of the keys and titles used for the individual maps.

I do do reads of the list of maps between the saves as well, so that may be (almost certainy is) part of the problem. I can probably optimize that to never re-read the map list after the initial read. It just it now because it’s an artifact of the code structure.

I’ll first simply comment out the reads in the program, to see if the problem goes away with the reads gone.

I’d love a version of clearLocalData that accepts a key value, so I could clear the map list before rewriting. Obviously, there’s no use to clearLocalData() as it stands now, for solving this problem. Or (depending on what’s going on) perhaps Codea needs a “reorganize local data” (compactLocalData) method to perform some housekeeping. But that will entirely depend on what the core problem is.

I’ll look for other work-arounds, like changing the key used to write the list-of-maps. Maybe if I write it to a different key value (and write nil to the old one), the problem will go away.

Thanks for looking into this. The readLocalData thing is definitely a huge clue.

@blacatena Maybe you could use something like this for debugging. This will save a function name in a file. Then when your code cancels, you can look at the file and see what functions executed and what was the last function executed. Add the function save(name) to your code and in each function you want the name to be saved, add the line save(“function name here”) at the start of that function. When your code cancels, from another program, do a readText(“Dropbox:test”) and print it to see what was executing. file= gives the file name, saveText(“Dropbox:test”,“”) clears the file when your code starts. use print(readText(“Dropbox:test”)) in another program to read the file. When this example runs, it will write xyz 10 times in the file test.txt . You could also change save() to write other information to the file.

function setup()
    file = os.getenv("HOME").."/Documents/Dropbox.assets/test.txt"
    saveText("Dropbox:test","")
    for z=1,10 do
        xyz()
    end
    print(readText("Dropbox:test"))
end

function xyz()
    save("xyz")
end

function save(name)
    f=io.open(file,"a+")
    f:write(name.."\
")
    f:close()
end

Doesn’t saveLocalData(key,nil) clear that particular piece?

What happens if you replace local data by project or global data?

Okay, so, interesting. I haven’t had time yet to add logging to a file, to see if it gives more accurate information, but, while I wasn’t expecting any differences, I tried using project and global data, and…

Global data ran into trouble while I still had local data saved. When I erased the local data, using Global data ran great for 4 maps (better than local data), but still ran into freezing problems after four maps.

Project data, interestingly enough, doesn’t seem to run into the problem. It so far has worked without trouble with up to 10 maps.

For now, I can use project data. In fact, I’m not sure, once this turns into a “real” app, if there’s any meaningful distinction between project data and local data (or global data), so maybe it won’t be an issue.

I’ll still pursue the logging, when I can, to see if I can figure out better where the real problem lies.