http request XML

Questions
1

Hey guys

i have this piece of code, where i try to load an eve-central API key (which is a server located XML file) and have it save it locally for further use. (i want to read from that XML, parse it / use values)

I have it now save the entire part of the XML system they use, based on the URL i give it, but then i have trouble loading the XML’s parameters (headers). Now the code i am building on seems a bit overdone, but it’s mainly for testing purposes, to see what my options are and what to built on. Question is: what are my options? End goal: download a LARGE amount of data, formatted in an XML order (eve central.com) and use parts of it for further calculations. (so for instance, i now ask for ID 34, stats from region 10000002. it gives me buy, sell and all volumes, min, max, median, etc and writes it to an xml file for later use)

let me note i am an amateur coder with limited C and JAVA knowledge and basically only know a bit about the structure of LUA and some basic coding. Almost totally new to codea and trying to learn as much as i can the next few weeks

code i use now:

-----------------------------------------------

local VALUE = 1
function setup()
    print("-- run check: loading eve central")
    market = nil
    http.request("http://api.eve-central.com/api/marketstat?&typeid=34&regionlimit=10000002", loadxml, failxml)
end
function loadxml(inp)
   market = inp
local file = os.getenv("HOME").."/Documents/market.xml"
local xml = inp
print("-- created file "..file)
writefiles(file,xml)
readFiles(file)
   end
function writefiles(file,xml)
    print("-- start write -- ")
    print("write in "..file)
    io.output(io.open(file,"w"))
    io.write(xml)
    io.close()
    print("-- write done --")
    end
function readFiles(file)
    print("--start read --")
    print("read from "..file)
    io.input(io.open(file,"r"))
    local li
    for i in io.lines() do print(i) end
    print("-- read done --")
    end
    function failxml()
        print("xml download failed")
        end
function draw()
    background(40, 40, 50)
    strokeWidth(5)
    if market ~= nil then
        text(market, WIDTH / 2, HEIGHT/2)
        fill(255)
        textWrapWidth(WIDTH)
        else
        text("loading", WIDTH / 2, HEIGHT/2)
end
end

---------------------------------------------
2

Not any experience with XML, but I think I know what you need. Function loadFiles can accept actually three parameters. First is data received from the website, second is status (200, 404, etc.), and third is headers. I believe the third is what you want. loadFiles should look like this, I think:

function loadxml(inp, status, headers)
    if inp ~= nil and status == 200 then
        market = inp
        local file = os.getenv("HOME") .. "/Documents/market.xml"
        local xml = inp
        print("-- headers:")
        for k, v in pairs(headers) do -- Loop through the headers and print all of them
            print("    " .. k .. ": " .. v)
        end
        print("-- created file " .. file)
        writefiles(file, xml)
        readFiles(file)
    else -- The page returned nothing, or gave an error.
        print("-- error getting xml, status: " .. status) -- Note that this kind of error is different from the failed function. The failed function is for something like a network error, like no WiFi. This kind of error is like 404 Not Found, where data is received, but the data tells you the page couldn't be found.
    end
end

Tested.

3

@Numenni - to make your code format nicely, put three ~ on a line before and after it. Also, use ’ for apostrophes if you don’t want weird text formatting.

Can you clarify whether your problem is getting the XML headers downloaded at all, or reading them using Codea?

4

Reading them using Codea after i created the xml file, but i rather have the direct approach opted by SkyTheCoder here. will test this afternoon and let you guys know!

it does not print anything between Headers: and Created file, but also gives back no error. does that have to do with the setup of the source file?

i love the additions though, status check is great for error handling, didn`t think of that one. but somehow the script now does not recognise the headers (i believe).

PS. you can CLICK the link in my original post to view what it reads.

Eventually i want to be able to call ID > BUY > VOLUME for instance or ID > SELL > MAX.

having trouble loading this in lua. in .NET it`s much easier, better XML integration and can’t get the XML parsers to work for codea, so figured to try and hardcode it myself, but i have to be honest and say it’s much harder than it looked at first.

@ignatz thank u, will do next time!

oh and can i delete a comment?

EDIT:
reading my code carefully, i think we are now reading from the created XML file. I think thats where i go wrong somehow, i might not be creating the xml properly, hence it does not find the headers? And would it then not be easier reading directly form the URL provided file?

5

it does not print anything between Headers: and Created file, but also gives back no error. does that have to do with the setup of the source file?

i love the additions though, status check is great for error handling, didn`t think of that one. but somehow the script now does not recognise the headers (i believe).

PS. you can CLICK the link in my original post to view what it reads.

6

Eventually i want to be able to call ID > BUY > VOLUME for instance or ID > SELL > MAX.

having trouble loading this in lua. in .NET it`s much easier, better XML integration and can’t get the XML parsers to work for codea, so figured to try and hardcode it myself, but i have to be honest and say it’s much harder than it looked at first.

7

@ignatz thank u, will do next time!

8

Woops! My bad. Should’ve used pairs, not ipairs. Edited the above code.

You can either update with the edited code, or remove the i in ipairs, so it’s just pairs.

Edit: Tested it, it works now.

9

Great. Now i need to find a way to have it read and display the subsections correctly.

10

Is there a way to use luaxml?

11

Or otherwise integrate string.match(i, “< min>(d-)< /min>”) into io.lines?
Like


I would then want to draw the d- number on screen, or for now at leaSt have it printed

Ps formatting fails for me
12

@Numennal I didn’t read the whole discussion but here is an xml lib that I’ve used created by @hyrovitalyprotago

XML = class()
--[[
-- @Author : Hyro Vitaly Protago
-- @Version : 1.1
 
-- This class is very simple to use :
 
--     - If you have an object, and you want to export as xml :
--         tostring(XML(obj, nameOfObj))
        
--     - If you have an xml, and you want to remake table :
--         XML(obj).racine
 
--     This doesn't support empty element like "<br />".
--     This doesn't support namespace.
--     This doesn't support attribute. (because is totally useless in this case)
 
--     Errors message are not really precise, but sometimes, it can help
--]]

XML.templates = {}
XML.templates.header = "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\" ?>"
XML.patterns = {}
XML.patterns.header = "<\\?xml[^\\?]*\\?>"
 
function XML:init(obj, name)
    if obj ~= nil and type(obj) == "table" then -- mode to xml
        self.xml = self:toXml(obj, 0, name)
    elseif obj ~= nil and type(obj) == "string" then -- mode to tab
        self:tableConstruct(obj)
    else
        error("IllegalArgumentException: @Param (1) obj must be a table or a string")
    end
end
 
function XML:toXml(tab, level, name)
    level = level or 0
    local str = ""
    if name then
        str = "<" .. name .. ">\
"
        level = level + 1
    end
    for k,v in pairs(tab) do
        for i = 1, level do
            str = str .. "\\t"
        end
        if type(v) == "table" then
            str = str .. "<" .. k .. ">\
"
            str = str .. self:toXml(v, level + 1)
            for i = 1, level do
                str = str .. "\\t"
            end
            str = str .. "</" .. k .. ">\
"
        else
            str = str .. "<" .. k .. ">" .. tostring(v) .. "</" .. k .. ">\
"
        end
    end
    if name then
        str = str .. "</" .. name .. ">\
"
    end
    return str
end
 
function XML:tableConstruct(xml)
    local _i, _j = string.find(xml, XML.patterns.header)
    if _i and _j then
        -- header manipulations (namespace)
        xml = string.sub(xml, _j+1)
        xml = xml:gsub(">[\\s\\b\\t\\r\
]*<", "><")
        xml = xml:gsub("^%s*(.-)%s*$", "%1")
    end
    
    self.racine = {}
    self.stack = {{name="racine", table=self.racine}}
    
    while string.len(xml) > 0 do
        local i, j = string.find(xml, "^<[^/<>]*>[^<>]*</[^<>]*>")
        if i ~= nil and i == 1 then
            xml = self:simpleNode(xml)
        else
            i, j = string.find(xml, "^</[^<>]*>")
            if i ~= nil and i == 1 then
                xml = self:endOfComplexNode(xml)
            else
                xml = self:complexNode(xml)
            end
        end
    end
    
    assert(#self.stack == 1, "Xml malformed !")
    
end
 
function XML:simpleNode(xml)
    local b,e,b2,e2,name,name2,value
    b, e = string.find(xml, "<[^<>]*>")
    name = string.sub(xml, b + 1, e - 1)
    assert(string.len(name) > 0, "Xml malformed ! you cannot use tag with no name...")
    b2, e2 = string.find(xml, "</[^<>]*>")
    name2 = string.sub(xml, b2 + 2, e2 - 1)
    assert(name == name2, "Xml malformed ! Opening and Closing Tag incorrect: ("..name..") ~= ("..name2..")...")
    value = string.sub(xml, e + 1, b2 - 1)
    self.stack[#self.stack].table[name] = value
    return string.sub(xml, e2 + 1)
end
 
function XML:complexNode(xml)
    local b,e,name
    b, e = string.find(xml, "<[^<>]*>")
    name = string.sub(xml, b + 1, e - 1)
    self.stack[#self.stack].table[name] = {}
    table.insert(self.stack, {name=name, table=self.stack[#self.stack].table[name]})
    return string.sub(xml, e + 1)
end
 
function XML:endOfComplexNode(xml)
    local b,e,name,name2
    b, e = string.find(xml, "</[^<>]*>")
    name = self.stack[#self.stack].name
    name2 = string.sub(xml, b + 2, e - 1)
    assert(name == name2, "Xml malformed ! Opening and Closing Tag incorrect: ("..name..") ~= ("..name2..")...")
    table.remove(self.stack, #self.stack)
    return string.sub(xml, e + 1)
end
 
function XML:__tostring()
    local toXmlString
    toXmlString = function(tab, level)
        level = level or 0
        local str = ""
        for k,v in pairs(tab) do
            for i = 1, level do
                str = str .. "\\t"
            end
            if type(v) == "table" then
                str = str .. k .. ":\
" .. toXmlString(v, level + 1)
            else
                str = str .. k .. ": " .. tostring(v) .. "\
"
            end
        end
        return str
    end
    if self.racine then
        return toXmlString(self.racine)
    elseif self.xml then
        return XML.templates.header .. "\
" .. self.xml
    end
end
13

Your class seems to handle, sort of, what i want. But i got it so that it works, but returns xml malformed error, line 121 in e xml class

14

There seems to be a rogue closer downloaded, /type. Have to remove that from the string first seems to fix the problem. Will have to try for different entries though

15

Mmmmm, I have write that code a “long” time ago. I’ll try to make something better in the near future (maybe tomorrow or this weekend).

16

Its workng my friend, on the database end it does < type id34> and closes with < type> which i think confuses the code. It wants < /marketstat> in that spot ( run my code or follow the url to see what i mean. Substracting the type level solved the problem, but its a bit messy that way.

17

Yeah… too messy for my taste :wink: I’ve a lot of ideas for support attributes, empty elements and others things, so, when i’ll found time, i upgrade it.

18

[Duplicate post]

19

@HyroVitalyProtago I had started re working your old code. I’m looking forward to seeing what you come up with.

20

Thanks @Briarfox, this is my first work on it (just for read) :

XML2 = {}

-- @Author : Hyro Vitaly Protago
-- @Version : 2.0

-- Warnings:
--  - this utilities have not been made for check if a xml is malformed or not, so, check before if it's possible.

-- Support: attributes

-- This doesn't support: namespaces, empty elements (for the moment), mixed elements

-- How use it ?
--  - XML2.toTableFromString(str) : return a table who contains the elements of the xml text (str)

-- Examples:
-- XML2.toTableFromString(...).marketstat.type.buy.volume
-- XML2.toTableFromString(...).__attribute.version
-- XML2.toTableFromString(...).__type == "simple" || "complex"

-- ToDo:
-- - empty (and mixed) elements
-- - toStringFromTable(xml)

local function trim(s)
  return (s:gsub("^%s*(.-)%s*$", "%1"))
end

local function startsWith(s, pattern)
    return not (not s:find("^"..pattern))
end

local mt
mt = {
    __index = function (tbl, key)
        if (startsWith(key, "__")) then
            tbl = rawget(tbl,key)
            return tbl[key]
        end
        assert(rawget(tbl,"__type") ~= "empty", "Empty elements have no value.")
        tbl = rawget(tbl,"__values")
        if rawget(tbl[key],"__type") == "simple" then
            return rawget(tbl[key],"__value")
        end
        return tbl[key]
    end,
    __newindex = function(table, key, value)
        error("Read-only table")
    end
}

local function removeXmlDefinition(xml)
    local _i, _j = string.find(xml, "<\\?xml.-\\?>")
    if _i and _j then
        -- header manipulations (namespace)
        xml = string.sub(xml, _j+1)
        xml = xml:gsub(">[\\s\\b\\t\\r\
]*<", "><")
        xml = xml:gsub("^%s*(.-)%s*$", "%1")
    end
    return xml
end

local function parse(xml, acc)
    xml = trim(xml)
    
    local i, j, element, elementName
    local space, endOfElement
    local attributes = nil
    
    i, j = string.find(xml, ".->")
    element = string.sub(xml, i, j)
    
    space = string.find(element, "%s") or j
    endOfElementName = math.min(space, j)
    
    elementName = string.sub(element, 2, endOfElementName-1)
    
    acc[elementName] = {}
    
    if space ~= j and space < j then -- attributes
        attributes = {}
        local attributesString = trim(string.sub(element, endOfElementName, j-1))
        local ii, jj = string.find(attributesString, '.-="')
        while (ii) do
            local attrName = string.sub(attributesString, ii, jj-2)
            local attributesString2 = string.sub(attributesString, jj+1)
            local iii,jjj = string.find(attributesString2, '"')
            local attrValue = string.sub(attributesString2, 1, jjj-1)
            attributes[attrName] = attrValue
            attributesString = trim(string.sub(attributesString, jj+jjj+1))
            ii, jj = string.find(attributesString, '.-="')
        end
        acc[elementName].__attributes = attributes
    end
    
    -- ToDo
    if string.sub(xml, j-1, j-1) ==  "/" then -- empty element
        -- acc[elementName].__type = "empty"
    end
    
    xml = string.sub(xml, j+1)
    
    i2, j2 = string.find(xml, "</.->")
    local nextEndBalise = string.sub(xml, i2, j2)
    local nextEndBaliseName = string.sub(nextEndBalise, 3, string.len(nextEndBalise)-1)
    
    if elementName == nextEndBaliseName then -- simple
        acc[elementName].__type = "simple"
        acc[elementName].__value = string.sub(xml, i, i2-1)
        setmetatable(acc[elementName],mt)
        return string.sub(xml, j2+1)
    else -- complex
        acc[elementName].__type = "complex"
        acc[elementName].__values = {}
        xml = parse(xml, acc[elementName].__values)
        while (xml and string.len(xml) > 0) do
            i2, j2 = string.find(xml, "</.->")
            local nextEndBalise = string.sub(xml, i2, j2)
            local nextEndBaliseName = string.sub(nextEndBalise, 3, string.len(nextEndBalise)-1)
            if nextEndBaliseName == elementName then
                setmetatable(acc[elementName],mt)
                return string.sub(xml, j2+1)
            else
                xml = parse(xml, acc[elementName].__values)
            end
        end
    end
end

function XML2.toTableFromString(xml)
    xml = removeXmlDefinition(xml)
    local racine = {}
    parse(xml, racine)
    setmetatable(racine,mt)
    return racine
end