Writing zero to a string

So, while we don’t have a way to write a file to the file system (yet), I want to save binary data to saveGlobaldata. (SaveLocalData has an issue with string size…whereas saveGlobaldata doesn’t…)

So, I’ve already stripped out zeros from my binary string, and to reconstruct it and saved it as well ad a table of where the zeroes live. Now I want to put the zeroes back. If I put a zero, or pass a variable that equals zero, and in 1-1, then lua converts it to an ASCII zero, char 48.

How do I inject a true zero into a string? The below doesn’t work with r=0, I get a “0” due to string concatenation.

function replace_char(pos, str, r)
    if (r=="0") then r="\\0" end  
    return str:sub(1, pos-1) .. r .. str:sub(pos+1)
end

You can convert a list of decimal values to a string using string.char(…), e.g.:


l = {50, 51, 0, 53}
s = string.char(unpack(l))
-- will print "23" because print stops at null terminators
print(l)
-- will print correct length of 4
print(string.len(s))

Convert it to list again:


l = string.byte(s, 1, string.len(s))

Is this what you are searching for?

Here is a simple program that tries to save a string to global data. See the output of the program I added as a comment at the bottom of the program. When the string c is printed, it prints only to the binary zero. The length of c is correct at 9. The values of c are shown, showing the binary zero between the 2 strings. And most important is the error message printed saying that you can’t save a binary zero as global data.


function setup()
    w="\\0"    --binary zero
    a="abcd"
    b="efgh"
    c=a..w..b    -- put binary zero between 2 strings
    print("string c",c)    -- only prints to the binary zero
    print("length",string.len(c))    -- prints true length of strings
    print("values",string.byte(c,1,9))    -- prints values of string c
    saveGlobalData("c",c)    -- tries to save c as global data
end

--[[         Here is what prints when this is run.

string c    abcd
length    9
values    97    98    99    100    0    101    102    103    104
error: error: [string "-- 00..."]:11: value cannot have null characters

Pausing playback

--]]

Is the problem here that pLists use c-type strings which are terminated by “\0”?

I tried my program above, changing the value of w to “\255” and Codea closes when the program gets to saveGlobalData. I haven’t tried other values, but apparently binary zeros aren’t the only problem when using saveGlobalData with something other than the normal character set.

@aciolino maybe you should 1/ convert you numeric data to hexadecimal 2/ save it as series of characters (‘0’ to ‘F’). This is just a factor x2 in data size compared to octets, and you won’thave to worry about string format problems?

\255 should be \FF in your example. Hex, not binary…

Looks like the problem with my code was the quotes around the zero in the if. I passed a int zero to the function but was checking for string zero.

Hmm this is probably a bug. You should be able to store 0s in the string passed to the save*Data methods.

You need to be c aware, so string terminators are zeroes…I’ve worked around it…can we get real file I/o instead?

Check http://www.twolivesleft.com/Codea/Talk/discussion/1692/saveglobaldata#Item_10 for a code that saves 0 values

While that might work, it…hacky…I have something like 8K I am saving (sound data) and doubling that to 16k isn’t my preference. That said, I do see the merits of doing it this way.

I decided to stick with saving the data into an image for the time being, which also is wasting bytes, as I am currently only using the r in the rgb values for data storage :frowning:

Here is my version of saving a string or table to an image. This version saves 3 values at a time as rgb. It could save 4, but I found that the alpha value changes the rgb values if it’s less than 255 when I was coding something else. See discussion saveimage() readImage(). The 2 functions, compress and expand, do all the work. The other code is just to create and print strings for examples. The compress function has 2 arguments, image name and string. This function will calculate the width and height image size and the string size as 3 bytes. The 3 byte size is put in the 1st entry of the image and will allow a size of 16,777,215 bytes. The expand function has one argument, image name, and returns the expanded string. This function will get the 3 byte string size from the 1st entry and expand the rest into a table based on the size. The string is created using table.concat. I used a table because that’s faster than constantly re-building a string using (…), string concat. Just to keep thing simple, I’m only printing values from the test string if there are less than 257 bytes. I tried different string sizes and saved 8,000 bytes as a 52x52 image. 80,000 as 164x163. 800,000 as 517x516. 1,000,000 as 578x577. 2,000,000 as 817x816. 3,000,000 as 1001x1000. I stopped at 3,000,000 because the app closed after it created the 1001x1000 image. I don’t know if it crashed because it was creating a 3,000,000 byte table and then the 3,000,000 byte string. Since the string size is saved in the image, you don’t have to keep track of it manually. The program ran in 10 seconds when I used a value of 1,000,000. That was to create a 1,000,000 byte string, convert the bytes and save the image, read the image back, expand the bytes and create a 1,000,000 byte table and then create the 1,000,000 byte string. How big a string can be saved, I don’t know. I guess it’s based on the iPads available memory.


function setup()   
    tab={}

    -- create test string containing the values 0x00 to 0xff.
    -- creating a table, then doing a table.concat is faster than
    -- doing a string .. (adding to a string).
    for z=0,255 do
        tab[z+1]=string.char((z)%256) -- keep value in 0-255 range (z>255)
    end   
    inString=table.concat(tab)    -- create string from table
    print("\
input string size",#inString)    --show string size
    -- end create test string
    
    
    compress("Documents:test1",inString)    -- compress string and save to an image
    
    outString=expand("Documents:test1")     -- read an image and expand to a string
    
    
    -- print test string if under 257 bytes, don't print larger ones
    if #outString <= 256 then
        for z=1,#outString do
            print(string.format("%02x",string.byte(outString,z)))
        end
    end
    print("\
output string size",#outString)    -- show output string size
    print("\
end")
    -- end print test string
end

function draw()
    background(40, 40, 50)
end

function compress(name,strng)
    local img1; local count=0
    local size=#strng
    local x; local y
    local v1; local v2; local v3
    local a1=256; local a2=256*256
    local s1; local s2
    
    -- calculate width by height for image size
    x=math.ceil((size+3)/3)
    s1=math.ceil(math.sqrt(x))
    s2=math.ceil(x/s1) 
        
    -- calculate 3 byte string size (b1+b2*256+b3*256*256)
    -- max size of 16,777,215 bytes
    -- 3 byte size goes in the 1st entry of the image
    b1=size; b2=0; b3=0 
    if b1>=a2 then
        b3=math.floor(size/a2)
        b1=b1-b3*a2
    end
    if b1>=a1 then
        b2=math.floor(size/a1)
        b1=b1-b2*a1
    end
        
    img1=image(s1,s2)    -- create image area
    img1:set(1,1,b1,b2,b3,255)    -- put string size in 1st entry
        
    for x=1,s1 do
        for y=1,s2 do
            if x~=1 or y~=1 then    --skip 1st entry, contains size
                v1=0;v2=0;v3=0
                count = count + 1
                if count<=size then
                    v1=string.byte(strng,count,count)
                    count = count + 1
                    if count<=size then
                        v2=string.byte(strng,count,count)
                        count = count + 1
                        if count<=size then
                            v3=string.byte(strng,count,count)
                        end 
                    end                  
                img1:set(x,y,v1,v2,v3,255)    --save 3 values per set
                end
            end
        end
    end
    
    saveImage(name,img1)    
end

function expand(name)
    local img2=readImage(name)
    local tab={}
    local str=""
    local count=0 
    local x; local y 
    local v1; local v2; local v3
    local size
    
    v1,v2,v3,aa=img2:get(1,1)    -- get string size from 1st entry
    size=v1+v2*256+v3*256*256    -- calculate string size
    
    for x=1,img2.width do
        for y=1,img2.height do
            if x~=1 or y~=1 then    -- skip 1st entry, was size
                v1,v2,v3,a=img2:get(x,y)    -- get 3 values each time
                count = count + 1
                if count<=size then    -- put 3 values in table
                    tab[count]=string.char(v1)
                    count = count + 1
                    if count<=size then
                        tab[count]=string.char(v2)
                        count = count + 1
                        if count<=size then
                            tab[count]=string.char(v3)
                        end
                    end
                 end
             end
        end
    end
    
    str=table.concat(tab)    -- create string from table
    return str
end

I wrote something like this a few days ago, but I wasn’t byte efficient. And yes, the alpha value gets overwritten for some reason with 255. It was very annoying.

I found that it’s not the alpha value that gets overwritten. If the alpha value is less then any of the rgb values, either the saveImage() function or the readImage() function changes the rgb values to the alpha value if they were larger. That meant I always had to set the alpha value to 255 so that the rgb values wouldn’t change.

Basically, the alpha value is useless for data storage, for whatever reason.