Utility: Automatic backup and recovery of your project code

I’ve created a backup and restore utility that does what I need, which

  • backs up my whole project automatically to Dropbox whenever I change my version number (usually when reaching a checkpoint)
  • allows me to restore backed up projects from Dropbox (including class tabs) automatically, without any copying and pasting
  • requires only one line of code to backup or restore (but you need to create a dependency to this utility, of course)

EDIT The latest code is always below, including full notes. Comments and suggestions welcome. Use with care as it’s not exhaustively tested.

http://pastebin.com/VyxjeVmV

Blimey - you’re on fire Mr Ignatz! Another superb addition! Er… Do you ever sleep? Lol

Who can sleep when there’s Codea to play with? :bz

What do you use for storing code, that is easy to copy from?

Hi @ignatz I just cant copy your code from safari… In Pastebin i touch download raw but then i cannot select the text. Any solution to do it from the ipad?
[edit] ok i’ve done it via goodreader. Not that easy, though.

That is strange: i think i could copy your last posts with no problem. Don’t know why i cant select this one. About my storage: i simply upload the txt file to my website. When you click on the link, the text is displayed in safari and seem easy to copy (at leat i think so).

That’s great. Thank you very much.

On split


function split(str, delim, maxNb)
    if maxNB==nil then maxNB=9999999 end

The first line has no effect inside the function. Even worse, it has an unadverted global effect. Did you write it? It’s not in the referenced link. You didn’t read the rest of the function, right?

Also, the function has two warts you don’t need. One is the aforementioned maxNb that you make no use of, the second one is the “bad case”, which is at the most a special case, but upon examining it more closely it is not even that.

Let me present you a more concise split function:


function split(str, delim)
    local result = {}
    local pat = "(.-)" .. delim .. "()"
    local lastPos = 1
    for part, pos in string.gfind(str, pat) do
        table.insert(result, part)
        lastPos = pos
    end
    -- Handle the last field
    table.insert(result, string.sub(str, lastPos))
    return result
end

Why don’t you need maxNb? Either you get the full result or it will be of no use for you.

Why don’t you need the special case? Firstly, let me tell you the the special case is still handled, just not as the very special case that was mention in the original code. Secondly, I assume you will always have at least “end” in a tab.

On your file format encoding

At this point I’ll just accept the your format looks like this:

  • original contents
  • 10 hash signs that act as some sort of your “format specifier”
  • as much padding with character “0” as you need to get a length that is a multiple of 3

Your encoding, taken from Backup:Store:


    txt=origTxt..string.rep("#",10)
    local n=3-string.len(txt)%3
    if n>0 then txt=txt..string.sub("0000",1,n) end
    n=string.len(txt)
    local s=math.floor((n/3)^.5)+1
    img=image(s,s)

The extra padding with "0"s is awkward. Better you remember the length of the text after adding your format specifier, then add “enough” padding so that the string reaches at least the next 3-byte-boundary.


    -- add format specifier
    txt=origTxt..string.rep("#",10)
    -- |n| is the length of the txt to encode
    local n=string.len(txt)
    -- padding so that accessing "a few more chars" results in a legal char, not nil
    txt=txt.."0000"
    local s=math.floor((n/3)^.5)+1
    img=image(s,s)

The key is to remember the length of the string you want to encode, then the length of the padding only matters so much that it shall not be too short. You can get rid of the padding at all if you modify the string access in the loop:


        -- original, requires padding of |txt|
        for j=1,3 do
            e[j]=string.byte(string.sub(txt,i+j,i+j))
        end

        -- without padding of |txt|
	for j=1,3 do
	    -- pad with "0" if string.sub reads beyond the end of |txt|
            e[j] = string.byte(string.sub(txt,i+j,i+j)) or "0"
        end

On your file format

Essentially, you have no file format. Instead you split the code into tabs by scanning it for class definitions. That’s not very elegant but I aknowledge the effort to try to implement this system. I suggest to use something the the Codea project format for future file formats. Perhaps keep your class splitter so you can rearrange foreign code to your liking.

Expanding your file format anyway

Just an idea. You could use the string space after “##########” to store some mata data, e.g. the names and byte positions of the tabs.

“##########Main,1,125;Other,126,659;Extra,660,1011##########”

By scanning the meta data you could recreate the tabs’ contents.

Finally

A great and useful project despite all my ramblings above. Especially the automatic backup when you change the version string is feature to steal.

Thanks for the comments. No, I didn’t write the split function, and I’ll try yours instead.

The hash signs at the end are there so i know where the end is, although I guess if I’m padding with zeros, then any zeros at the end have to be padding and not part of the text, so I don’t need the hashes.

File format - I was being dumb. I can include tab delimiters when I back the code up.

I’ll fix and repost. Thanks for your efforts to improve it, much appreciated.

unbeatable util! will use it for all my projects!

It worked, so it wasn’t too dumb.

Before you rush to shake your code up, here some additional suggestions.

File format

You have a very special problem with a file in an image. Firstly, it looks like an image. Secondly, the contained data is probably smaller than the image itself because the image is constrained to fill its rectangular region.

So my suggestion is to use something like a proper file header, e.g. use the first 3 bytes (I’ve chosen 3 so they make up one pixel (without alpha)) to store a magic number like “Ig1”, means “Ignatz Image File Format 1” (no kidding, really). The next 3 bytes tell you how large the actual content is. No padding needed, you know how many bytes to operate on.

Payload

You could try to use the Codea project format, this could be an appropriate generator:


-- Combine the contents of all tabs into one string formatted in the
-- Codea Project Format.
function Backup:GenerateCodeaProject()
    local txt = "" 
    local tabs = listProjectTabs()
    for i,t in pairs(tabs) do
        txt = txt.."\
--# "..t.."\
"
        txt = txt..readProjectTab(t)
    end
    return txt
end

I think Simeon would be pleased to see this simple format live on. The only disadvantage of this format is if somebody has the habit to start comments with “–#”.

More file formats

If you’re not pleased with it, invent “Ig2”, the “Ignatz Image File Format 2”. Perhaps this format starts with some meta data to describe the rest. “Main,1,100;Notmain,101,345;;”. Indexing starts immediately after the meta data. The end of the meta data is marked by two semicolons in a row.


These are just suggestions, but you see that it is easy to invent a format for text in an image that is more robust or more practical to implement.

Recapitulating the main objectives:

a) Storage of text (or any arbitrary data) in an image.

b) Combining and splitting tabs.

c) a) and b) shall be precise. You get exactly that thing out of them that you put it. No more, no less.

@Codeslinger - Good suggestions, thank you. To be honest, I was experimenting to see what would work, and I haven’t really sat down and planned it properly, like I should have done.

If you’d like to collaborate with me, that would be great, because I don’t see it as “mine”, but as a community utility, and if other people are going to rely on it, then it had better be rock solid

I like the idea of a version* byte, and storing size, too, to get rid of messing about at the end of the text. I flirted briefly with doing some compression, but there isn’t much need because code is so small it doesn’t take much space, and I want to keep the backup code small, given it gets loaded together with user code. Because code is small, I’m also not worried about the image format wasting part of the final column.
*(I come from a business spreadsheet background, and I’m always on at people to include version numbers, so I should have thought of this myself!).

The tab delimiter is interesting. If pasting the code into Main automatically prompted Codea to split it, then obviously the Codea format would be best, but that doesn’t seem to work (in fact, no matter what I do, I’ve not been able to get Codea to put anything back into tabs from a paste). So I’m assuming we need a tab delimiter that is definitely unique - which the Codea format is not.

In fact, because Codea doesn’t use chars above ASCII 127 - except for 194 and 226 for currency symbols - we could use a single char of (say) 255 as a delimiter. (I initially thought that wouldn’t work because colour values can take any value to 255, but colour digits are encoded separately so that’s not a problem). However, in case Codea does start using that range in future, maybe it’s best to find a chat thar will definitely never be used, such as char 3, which is labelled as “end of text” in the ASCII table (very appropriate) and which will definitely never appear in any user code.

If that’s ok with you, I’ll clean up the existing code, which I’ve started doing (and it’s much better already), and post a link soon.

@Codeslinger if you have time, look at this new version

http://pastebin.com/embed_js.php?i=gc0uGGvb

It has a file format, a file header, and is simpler than before ,which is good. I’ve also included a test tab that checks that key components work when you run Backup itself. Before doing so, though, put in the name of one of your projects for two of the tests (it reads from them - but does not write to the!).

More details on the Notes page in there. I haven’t tested exhaustively, that’s next.

I’ve written the stuff below before you came with your new version. I’ll post it anyway and will look into your code later (maybe not even today, sorry).


In fact the editor’s contents are UTF-8 encoded, it just happens to look like ASCII if you don’t make use of any special characters. We don’t have to deal with a BOM so it’s save to use codes like 254 and 255 as tab separators. You may even use 0 to mark the end of all data, so there would be no need to store the length of the data at the beginning. If you make the “Ignatz Image File Format 1” for text only you might even make good use of ASCII control codes.

Retro corner

Let’s use of ASCII control for the fun of it. Their meaning? We decide upon their meaning, we just look up their names.

http://en.wikipedia.org/wiki/ASCII#ASCII_control_code_chart

Perhaps with the codes SOH (1), STX (2) and ETX (3):

“Mainfunction setup() end function draw() print(“Hello”) end”

Where SOH Starts the tab name until STX, content goes until ETX. Then you either have a new SOH for the next tab or NUL for the end of the file.

You can still add a total length a the beginning of the image for double checking.

Code ideas:

(I changed “Ig1” to “Ign1”, 4 byte magic numbers are more common. I’ve initially chosen 3 because they did easily fit into your encoding triple. However, we don’t have severe alignment problems to solve.)


-- Private: Encode all tabs into Ign1 format, including header.
--
-- Returns Ign1 encoded string.
function Backup:EncodeTabsToIgn1()
    local SOH, STX, ETX = "\\001", "\\002", "\\003"
    local encoded = "Ign1"
    local tabs = listProjectTabs()
    for i,t in pairs(tabs) do
        encoded = encoded..SOH..t..STX..readProjectTab(t)..ETX
    end
    encoded = encoded.."\\000"
    return encoded
end

-- Private: Decode one tab of an Ign1 encoded string.
--
-- encoded - Ign1 encoded string.
-- start   - start index inside |encoded|.
--
-- Returns tab name, tab contents, next index on success.
-- Returns nil on failure.
function Backup:DecodeIgn1Tab(encoded, start)
    local SOH, STX, ETX = "\\001", "\\002", "\\003"
    local SOHidx = string.find(encoded, SOH, start)
    local STXidx = string.find(encoded, STX, start)
    local ETXidx = string.find(encoded, ETX, start)
    -- assert some basic format properties
    if not SOHidx or not STXidx or not ETXidx then
        return nil
    end
    if SOHidx ~= start or ETXidx < STXidx then
        return nil
    end
    -- decode
    local tabname  = string.sub(encoded, SOHidx + 1, STXidx - 1)
    local contents = string.sub(encoded, STXidx + 1, ETXidx - 1)
    local nextidx  = ETXidx + 1
    return tabname, contents, nextidx
end

-- Private: Decode all tabs of an Ign1 encoded string and write them
-- into the project.
--
-- encoded - Ign1 encoded string, including "Ign1" header.
--
-- Returns nil on success.
-- Returns error message on failure.
--   Tabs may have been saved until that point.
function Backup:DecodeAndSaveFromIgn1(encoded)
    if string.sub(encoded, 1, 4) ~= "Ign1" then
        return "Not in Ign1 format"
    end
    local tabidx = 5
    local errmsg = nil
    repeat
        -- check for end of data
        if string.sub(encoded, tabidx, tabidx) == "\\000" then
            tabidx = nil
        else
            local tabname, contents
            tabname, contents, tabidx = self:DecodeIgn1Tab(encoded, tabidx)
            if tabname ~= nil then
                saveProjectTab(tabname, contents)
            else
                errmsg = "Ign1 decoding error"
            end
        end
    until tabidx == nil
    return errmsg
end

The image de/serializing and high level backup logic can be taken from your original code.

Some loose comment on your new version:


-- In function Backup:init
    if t~=nil and remind~=false then
        print("Last backup was",string.format("%.0f",os.difftime(os.time(),t)/60),"minutes ago")
    else
        . . .
    end

You surely meant to write:


    if t~=nil then
        if remind~=false then
            print("Last backup was",string.format("%.0f",os.difftime(os.time(),t)/60),"minutes ago")
        end
    else
        . . .
    end

Thanks for your latest efforts. We must think alike because I also considered using the ASCII ETX char, but I saw a difficulty because it couldn’t be included in a Codea string. However, it could of course just be encoded directly into an image rgb property without ever going into a string, so I’ll be interested to see what you’ve done.

I very happy for you to improve the existing code, but I think we must try to not both work on the same thing at the same time, to avoid wasting effort. So please let me know if you plan to rewrite any part of it, and I will avoid working on that part.

What I will do today is look at what you’ve done, and if it’s better than my code, I’ll put it in, otherwise I’ll come back and discuss.

Thanks again, much appreciated =D>

@Codeslinger. I’ve had a look at your code now, and it’s great. I didn’t realise you could put non printing chars into Codea strings.

I only have one minor suggestion. Instead of coding "ign1” into every function name and having to do checks that the encoding complies, why not implement different encoding/decoding versions as completely separate classes, eg Backup_1, Backup_2, etc (I prefer to use a number rather than a variant of my name, because as I said before, I don’t see this as “mine”), and reducing the current Backup and Restore classes to a simple check of which version applies, then they call the right version class to handle it. Then within each class, you can always assume you are in the right place, and use identical function names across all the class versions, for consistency. It also means that if future versions get messy, their code won’t interfere with code for earlier versions.

I’d be happy if you want to fix up my code, and it is probably better if you do it, because you will probably improve lots of other things. However, I am also happy to do it myself if you prefer. Let me know what suits you. Either way I will share credit for this with you.

Just wanted to say this is one of the best utilities on this forum. I frequently screw up my code with fresh ideas that crash. Often, I can’t remember my errant changes. So this little bit of code is just what I needed. Thanks @Ignatz and @Codeslinger. It would be great if it were built into Codea. For instance when you build a new project, the setup function would automatically add a b=Backup("DropBox:YourProjectName_vers_1) line to the project–of course the YourProjectName would be whatever you typed in when you created the new project.

@Ric-Esrey thank you!

@Codeslinger

New version here: http://pastebin.com/embed_js.php?i=hvJ4rRdt

It incorporates your code, with the following minor modifications

  • I have created version specific tabs for backup and restore. The main Backup and Restore functions test for the version and call the version-specific backup/restore tab. This makes it cleaner if future versions need to work quite differently
  • I have shortened version number to 3 bytes so it can easily be read off from one pixel when restoring and checking version number (as in previous point)

I have tested it on a couple of projects but I thought I’d share with you in case it needs more work. But the code is much neater and cleaner now, thank you!