File IO hack (almost)

I have been wanting file IO for a few projects I have been dreaming up and I had an idea: encode the text into a image and save that. So I wrote some code that does just that perfectly, it encodes text (I have tested it with pretty long blocks of text) into images. The way it works is by using the string.byte and string.char functions to convert characters into their ASCII codes, a number between 1 and 255. I then can encode 4 characters into each pixel of an image by setting the RGBA to the ASCII values.

I had a feeling that there might be a problem when I wrote it out to PNG though, figuring that after encoding/compression/dithering whatever I wouldn’t get the exact same pixels back. The problem I didn’t foresee is that when I load the image on my iPad 3 retina, the image size is doubled, and scaling it down seems to cause corruption. Converting to PNG might be corrupting it anyway, not sure.

I am posting my encode/decode functions below in case someone is smart enough to figure out how to get it properly working. I had imagined having programs and text files all saved in my documents folder as garbled looking image files, and was pretty excited to have it. Any help appreciated.

function DecodeToString(img)
    local r,g,b,a
    local dstr = ""
    for y=1, img.height do
        for x=1,img.width do
            r,g,b,a = img:get(x,y)
            if r ~= 0 then dstr = dstr .. string.char(r) end
            if g ~= 0 then dstr = dstr .. string.char(g) end
            if b ~= 0 then dstr = dstr .. string.char(b) end
            if a ~= 0 then dstr = dstr .. string.char(a)end
            -- this is slow, I will speed this up later
        end
    end
    return dstr
end

function EncodeToImage(str)
    local rem = string.len(str)
    local len = rem
    local lines = 1
    while rem > 2000 do
        lines = lines + 1
        rem = rem - 2000
    end
    local img = image(500,lines)
    img.premultiplied = true
    for x=1, img.width do
        for y=1,img.height do
            img:set(x,y,0,0,0,0)
        end
    end
    local p1, p2, p3, p4
    local col = 1
    local row = 1
    local i = 1
    while i <= string.len(str) do
        p1 = 0
        p2 = 0
        p3 = 0
        p4 = 0
        p1 = string.byte(str,i)
        if len >= i + 1 then p2 = string.byte(str,i+1) end
        if len >= i + 2 then p3 = string.byte(str,i+2) end
        if len >= i + 3 then p4 = string.byte(str,i+3) end
        img:set(row,col,p1,p2,p3,p4)
        row = row + 1
        if row > 500 then
            row = 1
            col = col + 1
        end
        i = i + 4
    end
    return img
end

BTW I know it needs to be optimized, I will work on that if we can get the proof of concept working. Thanks.

I now understand your question from earlier about reading back a saved image exactly.

If you want to encode data in images you can perhaps do it like this. Use the following image functions instead:

image.rawWidth
image.rawHeight
image.rawSet()
image.rawGet()

When you make a 500x500 pixel image, it will really be 1000x1000 on your retina device. So allocate the image with half the dimensions you actually want. Then use the above functions to write and read pixels from the image.

When you load your image, your @2x version should be loaded — this one will have your data. (Your scaled version will be corrupted due to the filtering performed when down-scaling the image. So you can’t use that.)

The following code might be useful — note that you won’t be able to share data between retina and non-retina devices unless you name the images appropriately (or an @2x version isn’t available on the retina device).

local dataSize = 256
local img

img = image( dataSize/ContentScaleFactor, dataSize/ContentScaleFactor )

-- Access image using raw* functions

@Vega, I’ve been tackling data saves for Battle Chips in just the same way. I keep hoping that by the time I’m close to finished, @Simeon will have made it possible for end users to browse images in Dropbox or photo gallery.

Interesting. I’ve been taking large table values that need to be saved and slamming them together into a large CSV type string and then reparsing it back out on load. I’ll have to try this method out instead.

Thank you @Simeon, I believe I was tackling the problem backwards. This seems like it has a good chance of working. I have your Batlechips game code on my iPad, @Mark (very cool project, btw) so I will take a look at how you did it. Your code is probably already optimized, so it might give me some good hints.

lol - I love this. You get compression for free!

I’d rather see “real” file i/o (treating the Codea Documents directory as root, and letting us read other Projects and write to our own), but this may be a useful stopgap.

Hi Vega,

Pardon my ignorance, but Lua doesn’t, seem to address in anything less than a word. Is the native word 32 or 64 bit? Also, how does the retina screen allocate the graphics, as I understood that the true resolution was only used in photographs and not applications - or have I misread that.

Bri_G

:-?

@Vega, I don’t think there’s anything to see in the current code, and what I had implemented is pretty specialized: I want users to able to exchange “bots” as little images. All I had to support was a sort of bar code to handle the 'tokens" of the chip program, which I made intentionally chunky to help protect against lossy formats.

Okay, thanks @Mark, I think we had a slightly different goal in mind. But with what @Simeon showed me I think I can get it working (I will have time to fix it tonight hopefully.). Optimally I would like to implement some file IO capabilities and make it so that the programmer never has to even notice that he is saving to an image, he would just ReadFile and WriteFile and the library will take care of the encoding.

@Bri_G, as I now understand it, when creating a graphic on a retina device, Codea will actually create a graphic twice as large in both aspects (4 x as many pixels). When you save it to file, it creates a full resolution image and also downscales it and makes a half size copy which it will load for non-retina devices. For my purposes, I will need to check the scale factor and on retina devices make the Image half as large, and also I need to use the raw functions, which will give me access to real pixels instead of scaled pixels.

Update (in case anyone cares) : The corrections suggested by @Simeon fixed the scaling issues and allowed me to do a real test. There is, however, data corruption. I believe this has to do with the PNG compression algorithm.

I believe that the algorithm is not lossless (even though wikipedia says it is), and when there is a pixel beside it that has a similar value it makes them the same. (or something like that)

Encoding “abcdefghijklmnopqrstuvwx” then decoding it works perfectly.
Encoding “eleveneleveneleven” returns “eeeeeeeeeeeeeee” not good.

So anyhow, I feel I am getting close but I need to figure out a way to beat the compression algorithm. Maybe I will make it a bit more chunky and encode 3 pixels exactly the same and do a redundancy check. If you have knowledge of the PNG algorithm I would love to hear your suggestions.

Interesting. In my little experiment, I drew each value as a 3x3 box on output, then sampled just the center dot for input. It worked for me, but then what I did was a very small subset of data.

Yeah, that is the best solution. I switched it to 1x3 and sampled the middle dot and had much better results, but on close inspection I still had occasional corruption. I am re-encoding for 3x3 and that should be reliable.

Btw, I need to do fast concatenation of thousands of characters in my decode function. Basic concat using … Is amazingly slow. I read that a possible solution is to create a table and add each character as a separate item in the table, then do a table concatenation. But, as I understand it, this will use a serious amount of memory. Anybody know a better solution?

Png is most certainly lossless. Without a doubt.

Having said that, it’s very possibe something else in the chain is slightly corrupting things - OpenGL or the renderer or who knows what.

Okay, good news. File IO hack is finally working perfectly, and after a bit of optimization it is even pretty fast. To fix errors I made the image chunky, at 3x3 pixels for every 3 encoded chars, and I quit encoding information to the alpha number, which is not consistently saved correctly for some reason.

Tomorrow morning I will post the class I created, it is very easy to use. Thanks for the help getting this done.