What do I need to *send* to match an `http.request`?

I’ve just started investigating the possibilities of http.request to get my iPad to communicate with my computer. I just want to set up some sort of simple two-way communication between a Codea program and some program on my computer. To avoid confusion, I’m using lua for the computer program as well so installed the lua socket module. The test program worked quite well: I got a port number and when I used that with http.request then it connected and the request was sent as the next line. All fine and good, to get data from Codea to the computer I simply need to add something that parses http requests.

But I also want to be able to get data from the computer to Codea. So this needs to be sent back in some format that Codea is expecting. And that’s my question: what is Codea expecting? It certainly gets the status code back, but raw data doesn’t get through. I presume that I need to send data using http. What I’m hoping is that there is a simple template that I can wrap around my data so that I don’t have to load a module that constructs a valid http return. Is there? Or am I oversimplifying matters and I should just install a particular module and be content with that.

(Belay that, gotten it working now)

Hi @Andrew_Stacey I just replyed to @Codeslinger about the http function for sending and receiving information to a PC. Could you also give more info on using the http function. See post “Send file to a PC”.

.@dave1707 I can certainly keep a record of my experiments. They aren’t very sophisticated as yet.

I installed the lua socket library and the http.parser libraries on my Mac. I installed luasocket via luarocks directly (luarocks install luasocket) but you can also get it from http://w3.impa.br/~diego/software/luasocket/home.html. http.parser was a little trickier. I had to download a C library first, then install the parser. The link is: https://github.com/brimworks/lua-http-parser. What I did was:

cd local/include
git clone git://github.com/joyent/http-parser.git
export CFLAGS=-I$HOME/local/include
cd local/src
git clone https://github.com/brimworks/lua-http-parser.git
luarocks make CFLAGS=-g lua-http-parser-1.0-1.rockspec

I then took the sample TCP server program from the luasocket webpage and started adapting it to get something that would receive and send data to a Codea program. Here’s my current version:

-- debugging:
-- 1 everything
-- 2 warnings
-- 3 errors
debug_level = 1
debug_name = {
   "LOG",
   "WARNING",
   "ERROR"
}

function debug(e,l)
   if l >= debug_level then
      print(debug_name[l] .. ": " .. e)
   end
end

-- load namespace
local socket = require("socket")
local lhp = require("http.parser")
local connection_close = [[
HTTP/1.1 200 OK
Date: Wed, 02 Feb 2011 00:50:50 GMT
Connection: close

0123456789]]
connection_close = connection_close:gsub('\
', '\\r\
')

-- create a TCP socket and bind it to the local host, at any port
local server = assert(socket.bind("*", 50000))
-- find out which port the OS chose for us
local ip, port = server:getsockname()
-- print a message informing what's up
print("Please connect to " .. ip .. " on port " .. port)
-- loop forever waiting for clients
running = true
while running do
  -- wait for a connection from any client
  local client = server:accept()
  -- make sure we don't block waiting for this client's line
  client:settimeout(10)
  local rip, rport = client:getpeername()
  debug("Connection from " .. rip .. " on port " .. port,1)
  -- receive the line
  local get_headers = true
  local request = ""
  local headers = {}
  local body = ""
  local parser = lhp.request {
   on_url          = function(url)  print("URL: " .. url) end,
   on_header       = function(hkey, hval) headers[hkey] = hval end,
   on_body         = function(b)
      if b then
	 body = body .. b
      end
   end,

   on_message_begin    = function() print("Starting ...") end,
   on_message_complete = function() print("Finishing ...") end,
   on_headers_complete = function() get_headers = false end
			     }
  
  while get_headers do
     local line, err = client:receive()
     if err then
	debug(err,3)
	get_headers = false
     else
	parser:execute(line .. "\\r\
")
     end
  end
  if headers["Content-Length"] then
     local line, err = client:receive(headers["Content-Length"])
     if err then
	debug(err,3)
     else
	if line then
	   parser:execute(line)
	end
     end
  end
  print("Body: " .. body)
  client:send(connection_close)
  client:close()

  -- done with client, close the object
  -- client:close()
end

On the Codea side, I have:

function setup()
    http.request('http://<laptop-ip>:50000?hello',gotSomething,gotNothing,
        {
            method = "POST",
            data = "hello world",
        }
    )
end

function draw()
end

function gotSomething(d,s,h)
    print(s .. ":" .. d)
    for k,v in pairs(h) do
        print (k .. ":" .. v)
    end
end

function gotNothing()
    print("Got nothing")
end

Line endings were probably the most confusing part. When the socket receives data it can do so in one of three ways: get all the data, get a line, or get a fixed number of bytes. The first of these doesn’t work: the socket never returns. Getting lines is fine except that line endings get stripped. So I do this for headers and then read the given number of bytes (from Content-Length) for the body. This ensures that there’s no character changes on the body.

Andrew - the questions you’re asking (and some of the issues you’re having) are all answered by one thing: the HTTP protocol. What you’re doing above is reverse-engineering it. Don’t get me wrong - it’s fun - but you don’t have to do that if you don’t want to.

In particular - most webservers will do all of the parsing, and hand your program an array of all of the headers and their values, and the body as a string if it’s a post. They’ll also take your response, add appropriate headers (including Content-length), and get it back out to the browser. This is what we call CGI (Common Gateway Interface) - you can save yourself a lot of work if you google for “CGI LUA” and maybe run an apache server.

If you’re willing to use another language that’s similar to lua - and don’t want to mess about - I’d recommend ruby with sinatra. Here’s a full-on sinatra webserver:

require 'sinatra'

get '/hi' do
  "Hello World!"
end

http://www.sinatrarb.com/

Now I’m here, hopping around the threads about data exchange.

Yes, a readily available web server and a CGI script would be a perfect solution, there’s so much that can be done wrong when writing one from scratch. Anyway, I wrote a very simple one myself so I can do with to whatever I want.

And just to introduce some chaos I’ll continue my findings in the Codea 1.5 beta here instead of the thread where I was at first.

Look:


function postproject(pname)
    local tabs = listProjectTabs(pname)
    for i,tab in ipairs(tabs) do
        local data = readProjectTab(pname..":"..tab)
        post(tab, data)
    end
end

function post(filename, data)
    url = "http://192.168.178.25:8000/"..filename
    para = {}
    para["method"] = "POST"
    para["data"] = data
    http.request(url, success, fail, para)
end

function success() end
function fail() end

postproject("Cargo-Bot")

A few seconds later I got the full Cargo-Bot source on my PC. I think it’s about time to rewrite Ruilov’s github client.

A note for the adventurous: Since I’m sending to my own server I don’t need to care about some details, e.g. I’m always assuming binary encoding.

@Andrew_Stacey Thanks for the information you posted. Unfortunatly, that’s too much installing of stuff on my PC that I’m comfortable with. For the amount of information that I’ll send back and forth, I guess I’ll stick with my packing the data into a .PNG file. I have no trouble sending ASCII and Hex files both ways using my photo editor on the PC and Dropbox on the PC and iPad. The program on the iPad to read the .PNG file and convert it to a string of data is only 19 lines. On the PC, I just use existing programs to read the .RAW file. I also don’t need both the iPad or PC to be on at the same time. If I want to send a file to the PC, I just send it to the Dropbox and the next time I turn the PC on, the file’s there. Same with going from the PC to the iPad. If I knew more about file servers and all that other stuff, then the http function would be interesting to play with. But right now, it’s more complicated than I want to figure out.

.@Bortels Yes, I know. It’s the corollary to DRY: Don’t Repeat What Others Have Already Done (DRWOHA … shame that “Don’t Repeat What Have Others” doesn’t make sense).

But … and you should have predicted this … what I’m really aiming for is more complicated and a web server adds a step that I don’t want in the final set-up. More importantly, web servers tend to work reactively whereas what I want to do requires a proactive program that responds to data sent from the iPad.

Dave, if you want to transfer text (files), we’d better look into the Dropbox API. The only thing that I’m not proficient with is OAuth, which seems to be needed. Your images are a nice idea but nothing more than a bridge technology.

@Codeslinger That is just my way of doing it for now. I’m still waiting for the file.io support to be able to save/transfer files the right way. But on the good side, it’s a lot of fun trying to do work arounds. I already know how to use file.io, but I never had to work with .png or .raw photo files before.There’s always something new to learn when something you want to do isn’t supported.

I wrote the DropBox API code in Lua for Codea, that might help you guys out.

I’m sorry Andrew - that’s confusing. The suggestion to not re-implement what’s already been done is orthogonal to the protocol you use. I don’t understand how re-implementing the server side of http would give you anything you don’t already have (other than, again, doing it for education or fun). With your code or with Apache, you still must initiate the connection from Codea, talk, and then do it again (poll) if you’re waiting on some server-side event; with the callback interface Codea uses, there is no equivalent to non-parsed headers (ie. streaming). You can’t get away from http - it’s what Codea speaks here.

To answer your original question - a http response consists of two parts: a series of one-line key/value pairs, seperated by a colon (the “Headers”) containing metainformation such as the type of data being sent, followed by a blank line then the actual content of the web page (the “Body”). Codea will helpfully parse that all for you, delivering the body as a string and the headers as a table.

Here’s an example of a http session - I got this, by the way, by using “curl” - you can google for it and download it - as “curl -vvv https://home.bortels.us


> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: home.bortels.us
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 15 Dec 2012 09:33:58 GMT
< Server: Apache/2.2.22 (Ubuntu)
< Last-Modified: Fri, 25 May 2012 17:32:03 GMT
< ETag: "3a289a-d3e-4c0dfbf166ac0"
< Accept-Ranges: bytes
< Content-Length: 3390
< Vary: Accept-Encoding
< Content-Type: text/html
<
<!doctype html>
<html lang="en">
<head>
<title>Tom's Home Page</title>
... and the rest of the html of my home page

The first lines (“>”) are what the client sends to the server - a protocol line (“GET / HTTP/1.1”) that tells what method and URI, then a series of header lines with extra information, such as the useragent used and what domain I’m trying for, and what kind of data I can accept. For http 1.1, the only required lines are the first method line, and the Host: header.

The remaining lines (“<”) are the response (200 is “ok”), followed by a bunch of header lines with data about the page being returned, followed by a blank line, then the page content itself.

The real issue you will have is that there’s no way to make the server reach out and initiate a connection to Codea - you’ll need to poll the server. (You can have your server delay answering - so if you made a chat server, for example, it simply wouldn’t return the result until a new line was ready to be delivered.)