http.request Accept-Encoding: gzip in headers...

Questions
1

Hi.

http.request adds Accept-Encoding: gzip into headers so some sites send me gzipped content and data I get from them seems to be empty…

I tried to rewrite this header but could not…

What can I do? Can I disable adding this header or rewrite or unzip content to get htmls?

2

The concrete example

-- Downloading data
function setup()
    textData = nil
    params={}
    headers={}
    headers["Host"]="new.mcko.ru"
    params["method"]="GET"
    -- changing useragent does not solve the problem...
    params["useragent"]= "Mozilla/4.0(compatible; MSIE 7.0b; Windows NT 6.0)"
 
    

    params["headers"]=headers

    --   Request some data
    http.request( 'http://new.mcko.ru',
     didGetData, fail, params)

end

-- Our callback function
function didGetData( data, status, headers )
    print(status..": ")
    -- This line prints empty string
    -- But I checked - server sends zipped data
    print(data)
    for k,v in pairs(headers) do print(k..": "..v) end
    
 
end

function fail(error)
    print("error")
end
3

Well data being returned is not nil. When I try to access it as a table it shows it’s a string. I’ll keep playing with it.

edit: It looks like there are options for reading gzip data. I’m not familiar with gzip so let me do some more research.

4

Well if I encode data into Base64 I get a printable string.

5

Briarfox, thanks!
It works, but it is unuseful.

How to unzip…

Btw, how you encoded that?
Has Codea standart functions for that?

6

I don’t think the Base64 helps. I was just trying to see the string that was there. I’m still working on this. Anyone else have for knowledge on compression?

7

Besides setting the Accept header on the request for just HTML : http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

There is also the (reasonably unpleasant :wink: ) option of implementing gzip/deflate in pure lua :slight_smile:
A quick google turned up a pure lua implementation (although may need some modification for codea) :

https://github.com/davidm/lua-compress-deflatelua
https://github.com/davidm/lua-compress-deflatelua/blob/master/lmod/compress/deflatelua.lua

and for interest heres a pure JavaScript implementation :slight_smile:

https://github.com/beatgammit/gzip-js
https://github.com/beatgammit/deflate-js

The specs may also be of interest :slight_smile:

DEFLATE Compressed Data Format Specification version 1.3
http://tools.ietf.org/html/rfc1951

GZIP file format specification version 4.3
http://tools.ietf.org/html/rfc1952

Edit: I also happened across this which you could possibly coax to decode it for you :slight_smile:
http://i-tools.org/gzip

or you could write a little Php script and use gzdecode to decode the data
http://php.net/manual/en/function.gzdecode.php

8

Thanks, XanDDemoX!

Please explain, how to use some of the choices you described.
First about Accept header. I tried to disallow that, but could not.
And that is the fastest way to solve the problem.

Second about pure lua. That will be slow, but enough for me, I think.
You were right. That code is not compiled by Codea…
I got in line
local crc32 = require “digest.crc32lua” . crc32_byte
error that that module not found.
How Can I add it to Codea?

How to use JavaScript fo my purpose?
I have no idea…

And finally yes, I have thoughs about writing main code in PHP
and then get Queries from Codea program and send to Codea the results only
from the php script.
But this does not seem to me a good Idea, because we need to organize special server - webpage for the program, and it will be difficult for many users…

And Thanks, again!

9

The accept header is probably your best bet for a simple solution. Setting it up like so should tell the server you want plain text or HTML instead of gzipped :slight_smile: (it actually says to the sever i only support/accept these formats of data)


local params ={ headers = {Accept="text/*, text/html"} }

local success = function() end
local fail = function() end

http.request("url",success,fail,params)

The other solutions i wouldn’t recommend unless you absolutely have to as there is considerably more work involved :). The pure lua one would need the “inflate” part of the algorithm to be extracted or its dependencies to be included with it in Codea. The JavaScript one is really just for a secondary reference. The same goes for the actual specs of gzip and the deflate algorithm, just for reference of how gzip works :).

10

Unfortunatelly, that header Accept does not help.

This code shows 200: without any symbols after…

-- Downloading data
function setup()
    textData = nil
    params={}
    headers={}
    headers["Host"]="new.mcko.ru"
    params["method"]="GET"
    headers["Accept"]="text/*, text/html"
    params["headers"]=headers

    --   Request some data
    http.request( 'http://new.mcko.ru',
     didGetData, fail, params)

end

-- Our callback function
function didGetData( data, status, headers )
    print(status..": ")
    print(data)
    for k,v in pairs(headers) do print(k..": "..v) end
end

function fail(error)
    print("error")
end
11

Having a look at this it seems to work just fine and the accept headers are completely unnecessary also. It actually seems to be print is not printing the data which is 26611 characters long. Try this for the didgetdata function :slight_smile: I’m just printing the first 255 characters.

function didGetData( data, status, headers )
    print(status..": ")

    local count =#data
    print(count)
    print(string.sub(data,1,255))
    for k,v in pairs(headers) do print(k..": "..v) end
    
end