http.request search function

I have an iPad 1 and so the forum search function doesn’t work for me anymore. I though I would write my own, but I have a few questions. What I have so far is a list of Titles and the Author of each discussion from the 1st page. My questions are: how do I read past the first page and how do I read the information contained in each discussion.


function setup()
    http.request("http://www.twolivesleft.com/Codea/Talk/discussions",gotData)
end

function gotData(data,status,headers)
    --print(data)
    find(data)
end

function find(str)
    s=1
    while true do
        -- find title
        s,e=string.find(str,'class="Title">',s)
        if s==nil then
            return
        end
        s1,e1=string.find(str,"<",e)
        print(string.sub(str,s+14,e1-1))
        s=e

        -- find author
        s,e=string.find(str,'class="Author">',s)
        if s==nil then
            return
        end
        s1,e1=string.find(str,"<",e)
        print(string.sub(str,s+15,e1-1))
        print()
        s=e        
    end
end

Ahhh snap @Ignatz :wink: beat me to it by a minute! :slight_smile:

@Ignatz Thanks, the /p2 worked like a charm. As for searches now, I use Google and keyword site:twolivesleft.com . I’m still unclear on the discussion contents. What’s in the “data” that I would use for the contents of the discussion. I didn’t see anything that looked useable.

If you look at the page source of this webpage, you’ll see the message text is contained in a div tag labelled message.

But if you were to search this page, I would just http.request it, which should give you the page source, then just search it as it is, without trying to separate message content from all the HTML and CSS coding.

Just to be absolutely clear - to see the discussions, you have to http.request each discussion thread separately. That means parsing the list of discussions to pull off the URL of each separate discussion, then using http.request with each one to get the detailed discussion (such as this page).

@Ignatz I currently have the code to get the title of every discussion on every page. I still can’t figure out how to get the discussion contents. For instance, if I wanted the contents of this discussion, what would I be requesting. I’m not sure what the ??? below should be. I’ve tried different things and either I don’t have the correct info or I have the wrong format because I haven’t seen anything that looks like what’s in this discussion. I haven’t had a need for http, so I’m not familiar with it.


http.request("http://www.twolivesleft.com/Codea/Talk/discussions/?????")

With your current approach @dave1707 to read the contents of each discussion and each page you would have to make a request for each one you would want to search.

Perhaps try looking at the page source and make your request specifically to the google custom search and parse the contents of that. Let google do the searching and you can just display the result :slight_smile:

If you look at the source for the main discussion page, you will see this buried in there.

NOTE I have replaced the angle brackets with square brackets, otherwise they get treated as HTML tags by the forum editor!

[div class=“ItemContent Discussion”]
[a href=“/Codea/Talk/discussion/3410/http-request-search-function#Item_8” class=“Title”]http.request search function[/a]

The URL you need is
http://http://twolivesleft.com PLUS the link in the source above, ie

http://twolivesleft.com/Codea/Talk/discussion/3410/http-request-search-function#Item_8

(NB don’t worry about the #Item_8, that is just the forum remembering which post I’d read last, and scrolling down to the new posts. You will still get the whole discussion downloaded regardless).

So what you need is a way of finding all these URL links. The text [div class=“ItemContent Discussion”] seems to do the job.

If you look at the URL, page 2 has the same URL as page 1, with “/p2” added, and so on for other pages.

To extract the discussion, you’d need to load each discussion thread separately, and if there are multiple pages per thread, load each one. That seems a huge amount of effort to go to and would take forever to search, unless you do it once and build your own keyword database!

Then of course, you could always just use google… Search “Codea forum KEYWORD”

@Ignatz Thanks, that worked like a charm. I’m just curious why it didn’t work when I tried it the first time. The only difference was I had #Item_7 instead of #Item_8. I must have had a typo somewhere when I did it and since I wasn’t sure what I was supposed to use, I didn’t check it. Thanks again.

EDIT: I just realized that I used discussions instead of discussion. I keyed in the information manually and just added it to one of the lines I already had.