Codea memory and hang problems

@sim @John There are 2 problems here.

The first is memory. I did a Settings/General/ipad Storage. I looked at the Codea Documents and Data and it had a size of 716MB. I did a Codea Projects to zip and that had a size of 95MB. I did a project backup using my own program and then deleted Codea. I reloaded Codea and loaded all of my projects back. I did another iPad storage and the Documents and Data had a size of 42MB. I also did a Codea projects to zip and that was 16MB. What was taking up so much memory I don’t know because I don’t do a lot of extra assets or anything.

The second problem is Codea hanging when I run my backup/restore program. It will restore 15 to 30 projects before Codea hangs. I had to keep restarting it a lot of times to finally get my 860+ projects loaded. Below is a stripped down version that causes the hang problem. It will delete and create the same project and eventually hang. My backup/restore program doesn’t need to delete any projects because I would start with a clean Codea program with no projects. It would eventually hang just like the example code. My program used to run in a few seconds but now takes a lot of time because of all the restarts.

PS. Don’t make the Project name and projName the same.

viewer.mode=STANDARD

function setup()
    projName="0001z"
    str="-- qwertyuiop"
    
    for z=1,200 do  -- delete / create project 200 times
        
        if hasProject(projName) then
            print("before delete",z)
            deleteProject(projName)
            print("after delete",z)
        end
        
        print("before create",z)
        createProject(projName)
        print("after create",z)
        
        print("before saveTab",z)
        saveProjectTab(projName..":".."test",str)
        print("after saveTab",z)
    end
    
    print("done")
end

@dave1707 - just trying out your test project and used a project name of ‘TheCrasher’ and only ran it for a loop of 20. It ran in the first loop and threw an error (see attached image). It created the file but obviously only once.

Perhaps Codea can’t delete the project but for some reason continues on in the loop - or loses access to that memory allocation.

@Bri_G The project name and the name of the project it’s creating shouldn’t be the same. The program doesn’t crash or put out any errors, it just hangs and you have to force close it then it crashes.

OK, missed that got it now - thanks.

@dave1707 - ran your project again, but with only 20 iterations. Got as far as 16 on the loop then hung up and eventually crashed.

Added collectgarbage() after the addition of a tab to the project and it ran out all 30, added a blank tab.

It would be useful if we had some memory variables so that we can see the change in allocation when we run a project. Do you know if there are any in the system?

Secondly - do you think the presence of V3 and V4 in the beta may be influencing the memory allocation ?

@Bri_G For memory there’s print(collectgarbage(“count”). I tried that and memory barely changed. I tried various things but it still hung at different counts. From what I can tell it’s a Codea problem based on the print statements. I had this problem over a year ago and the problem is still happening. I tried putting in a large for loop to slow things down, but it still eventually hung.

Have you tried doing an iPad storage to see how much storage Codea is using. I think there’s a memory leak and something keeps memory.

@dave1707 - much of my experience with earlier computers involved kit which had fixed addressable graphics memory, core system memory and ram which held your apps and data. That made computing relatively straight forward.

My limited understanding of the Apple ‘multitasking’ system is that it’s not true multitasking and the ram is used for all facets - so it holds the OS, your apps, your data then allocates memory for the graphics display. So you can multitask but only one project is active at a time. I’m not sure if the latest iOS has moved to a truer multi-tasking environment.

So - when you run an app memory is allocated to the screen graphics, code, sprites sound etc. This must be ring fenced for each app but I’m not sure how that is arranged. I assume the graphics area could be allocated to be greater than the screen area requires. But, the remaining free memory area must have a start and finishing address in memory. The start address may vary as routines may use it for temporary routines in support of your app - they would require management to maintain or release but, I would have thought he top of free memory should have a fixed address. So it should be possible to calculate the free allocated memory from the top - the flex memories. Rather than calculating the amount of freed up memory after collecting garbage.

Is that a reasonable understanding?

P.s - my concern is that kits with different memory sizes could respond in different ways so sometimes it may confuse the issues we encounter.

@Bri_G When I asked about Codea storage, I meant on the iPad doing Settings/General/iPad Storage/Codea then the Documents & Data amount.

Mine is currently at 179MB, that’s with 860+ projects. After I deleted Codea then reloaded everything it had an initial size of 42MB. I’ve been running the code at the top of the discussion and it looks like memory is building with deleting/creating projects. I do a lot of creating/deleting projects as I test different things. I guess over time something builds up.

@dave1707 - a few questions for you

  1. HOW DO YOU MANAGE OVER 800 PROJECTS ?

on a less emphatic note

  1. Do you use your own archiver ?
  2. Do you use the Codea export facility ?
  3. Do you have multiple assets lists in the Codea asset menu ?
  4. Do you restrict your projects to textual assets ie use links for memory intensive assets like images ?

Final question -

How do you keep sane, I have problems remembering a couple of hundred projects - no where near 860.

Sorry about the list, I’m just intrigued about how you use/manage Codea

@Bri_G My projects are sorted by name so they’re easy to find when I look for something. New test projects I start with numbers which keeps them upfront. Once they’re final I give them a name. If they’re temporary I leave them with numbers until I delete them. I mostly use Codeas assets so I have no problem with that. If I used someone else’s assets from the forum then I used to keep them in the Dropbox folder. I don’t do anything fancy so I don’t use anything more than that. Some projects that I use all the time start with # which keeps them at the start of the project list. So basically I don’t do anything fancy which keeps things simple.

@dave1707 - that sounds like a good system. One of my many problems is that if I often get bogged down with a project I copy what I think are the good features and include them in a new project. This ultimately leads to loadsa partially complete projects.

Another problem is that I rarely fully finish a. Project so I end up with a long list of partially complete projects.

I cleaned my system up when V4 beta arrived and am slowly working through my list of not complete projects. Think I might try out your system.

Thanks for the ideas.

@Bri_G I also have a project search program that lets me search through all my projects. If I look for something specific and don’t remember which project it’s in I do a search for it. In the print area It lists the names of all the projects that contain what I want. In the text area it shows the project name and the lines of code that has what I searched for. If I want to expand a project, I tap on the name in the print area and select the full name and tap copy. It shows the full project. I than can go back to the original views.

Thanks Dave, I think this may be a file access race condition. Thank you so much for the repro code, will debug it tonight

@dave1707 this should be fixed in the next build (at least, I am able to consistently run your test case to the end)

@sim The real test will be when I delete Codea, reload it, and then use my backup program to load all of my projects (890) back.

I tried creating my backup file and noticed that my backup file wasn’t in my backup folder (iCloud). I tried the backup several times but there wasn’t any backup file in the folder. I had to delete the link to the backup folder and relink it before it would work. Before I deleted it, it showed as an external folder. If it shows as an external folder, shouldn’t it work. Also, there wasn’t any kind of error message saying that it didn’t write the file.

Did the link to the external folder break after updating or reinstalling Codea?

It has happened to me too, I am trying to track down the cause

@sim I didn’t reinstall Codea and I just tried after the update and the link to my folder was ok. I was able to saveText to the folder. So I’m not sure when the link broke or what broke it.

@sim I tried doing my project restore and didn’t get any hangs like before, but it would restore only 23-28 projects before I would get one of two error messages.

The first was “invalid buffer name”. That was on the line

saveProjectTab(name..”:”..tabName,tab)

where “name” is a project name and “tabName” is a tab name. It was trying to restore a tab in the project.

The second was “failed to create project”. That was on the line

createProject(name)

where “name” is a project name when trying to create a project.

I would delete the last project that was restored bad and do the restore again. It had no trouble restoring the project it errored on and would do another 23-28 projects till the next error.

@sim In my backup/restore program, I added a 1/20 second delay before each project, but I still get one of the above errors after so many restores. I was doing a large for loop for the delay, but maybe the tight loop isn’t letting Codea update what it needs. So I don’t know what’s really happening.

@dave1707 - I’ve used your backup projects for some time. You’ve changed the Codea and approach several times - so I’m not sure if I’m up to date to check your observations

But – thought occured to me, if your problem occurs due to file handling timing. Could you, after adding each project to your file, add a line to see if a project exists which may ensure the previous filing system call is completed before proceeding with the check. Then move on to the next file for archiving. You could have a small file present used for the check.

Another thought, it’s not an issue of file type case eg .txt or .TXT is it?