Understanding touches

This is really just me thinking aloud, but hopefully if I think something stupid then someone’ll hear my thinking and put me out of my misery.

I’m trying to understand exactly how a touch works so that I can better handle multiple touches and taps. Here’s my code that I’m experimenting with:

function setup()
    x = WIDTH/2
    y = HEIGHT/2
    c = color(255,255,0,255)
    frame = 0
end

function draw()
    background(0, 0, 0, 255)
    frame = frame + 1
    fill(c)
    ellipse(x,y,100)
end

function touched(touch)
    x = touch.x
    y = touch.y
    local status, taps
    if touch.state == BEGAN then
        status = "began"
        c = color(255,0,0,255)
        frame = 0
    end
    if touch.state == MOVING then
        status = "moving"
        c = color(0,255,0,255)
    end
    if touch.state == ENDED then
        status = "ended"
        c = color(255,255,0,255)
    end
    if touch.tapCount == 1 then
        taps = ""
    else
        taps = "s"
    end
    print(status .. ", " .. frame .. " frames, " .. touch.tapCount .. " tap" .. taps)
end

What I see is that touch events are only registered when something happens. So a touch can begin and then wait a while before anything more happens - if you have a steady finger!

This makes life a little complicated if one is trying to track changes with two touches. The temptation is simply to write change = touchA.deltaX + touchB.deltaX. This is fine if both touches changed. If only one touch changed, then as the other one has not been updated, its delta is still the same. So when dealing with two touches, one wants an overlay object which gets updated every cycle no matter what.

Moreover, this testing suggests that the documentation is subtly wrong. It says that the deltaX and deltaY are the amount that the touch has moved since the previous frame. It appears that this is since the previous change to the touch. Namely, even if there are several frames between touch events, the delta is the difference between them.

With taps, it appears that the tap count follows the touch.id and is incremented if there was a “tap” at the same location (presumably within some plus-minus) within some time limit. A “tap” is a touch that has just a beginning and an ending. Because of what I said above, a tap can last a long time if you are very steady. I guess that this is the secret behind “long taps”. Again, to properly process these events one would need a more robust touch object.

So for a complicated project, I’m thinking that the following would be a reasonable way to cleanly work with touches.

  1. A new Touch class which was meant to deal with a whole touch, not just its events.
  2. In the touched function, we initiate, update, and destroy these objects according to the new data.
  3. At the start of the draw function, we process the current state of play vis-a-vis the Touches that are active.

Unfortunately this is how touches work in iOS. A stationary touch does not trigger events. The way to detect a long press, or a pinch gesture as you describe is to keep track of touch IDs until they end.

Good catch on the documentation though - that is incorrect. It should be since the last state change, not frame.

I am running out of laptop battery so apologies if this doesn’t answer your question completely.

EDIT: I think if we add support for Apple’s built-in (and very robust) gesture recognizers, some of these problems will be overcome.

The multi-touch demo has an example processing all touches, regardless of whether they are stationary:

function setup()
    touches = {}
end

function touched(touch)
    if touch.state == ENDED then
        touches[touch.id] = nil
    else
        touches[touch.id] = touch
    end
end

function processTouches()
    for k,touch in pairs(touches) do
        -- here you get every active touch, regardless of whether it has moved
        -- e.g. compute distance between touch pairs 
    end
end

function draw()
    processTouches()
end

You have brought up a lot of points here, so let me try to work through a few of them.

The first thing you should know is that the touch API is a pretty thin wrapper over the iOS touch API. You can find some useful documentation on Apple’s website

http://developer.apple.com/library/ios/#documentation/EventHandling/Conceptual/EventHandlingiPhoneOS/MultitouchEvents/MultitouchEvents.html

We just wrap the UITouch structure into a lua structure and pass it to touched whenever the iOS event handlers are called (touchBegan, touchMoved, touchEnded, touchCancelled). Which handler it was passed to is reflected in the state member (BEGAN, ENDED, MOVING, CANCELLED).

Given this:

  1. The delta members are based on where the touch was when it was last passed to an event handler and where it is now.
  2. Taps are handled by Apple’s (magical) algorithm for detecting taps (which it stores on the UITouch structure).
  3. The touch id is based on the UITouch object’s memory location which can be reused but is consistent between a BEGAN and ENDED state.

which can be reused

Yes! That caught me out in another program. I wanted each touch to spawn a new object, so I merrily created an array of objects indexed by their touch id. I was a little surprised when old objects vanished from view as new ones were created; printing the touch.id showed what was going on and I changed my code accordingly.

Given that the deltas are computed from the last state change, and not the last frame, it would be useful to extend the touch object to include a field that records the time since this state change occurred. I’m not sure if it would be most useful in terms of time or number of frames. Probably time as that’s harder to keep track of inside the script - especially if it can be actual time between the events, not just time between the frames in which the events occurred.

As an example, my advanced code in the touch tutorial computes a final velocity from the end of the touch and I’d written it assuming that the deltas corresponded to the frames.

Moved to iPad now that laptop is dead. Here’s a quick example that does pretty smooth pinch-to-zoom by tracking two touches, even if one is stationary. It uses the centre of the pinch to zoom around.

-- Use this function to perform your initial setup
function setup()
    print("Pinch to Zoom the view")
    touches = {}
    lastPinchDist = 0
    pinchDelta = 0
    pinchCenter = vec2(WIDTH/2,HEIGHT/2)
    zoom = 1
end

function touched(touch)
    if touch.state == ENDED then
        touches[touch.id] = nil
    else
        touches[touch.id] = touch
    end
end

function processTouches()
    touchArr = {}
    for k,touch in pairs(touches) do
        -- push touches into array
        table.insert(touchArr,touch)
    end
    
    if #touchArr == 2 then
        t1 = vec2(touchArr[1].x,touchArr[1].y)
        t2 = vec2(touchArr[2].x,touchArr[2].y)
        
        dist = t1:dist(t2)
        if lastPinchDist > 0 then 
            pinchDelta = dist - lastPinchDist
            pinchCenter = (t1 + t2)/2
        end
        
        lastPinchDist = dist
    else
        pinchDelta = 0
        lastPinchDist = 0
    end
end

function draw()
    -- This sets the background color to black
    background(0, 0, 0)
    -- compute pinch delta
    processTouches()
    -- scale by pinch delta
    zoom = math.max( zoom + pinchDelta/300, 0.2 )
    
    translate(pinchCenter.x,pinchCenter.y)
    scale(zoom,zoom)
    
    pinchDelta = 0

    -- Do your drawing here
    fill(212, 86, 86, 255)
    rect(132,130,200,200)
    fill(111, 140, 189, 255)
    ellipse(0,0,300,300)
end

RIght, so that illustrates what I was thinking. That when working with something that depends on changes in touch data, you have to store it yourself (lastPinchDist in your code) rather than relying on the .deltaX and .deltaY.

Now with taps it gets even more complicated. With a triple tap, say, we have six events: each tap generates a BEGAN and ENDED event, and each tap gets its own id (which may or may not be repeated … the guarantee is that concurrent touches get their own ids, not successive).

So in my program, I get a touch. If it has no moving parts, I can’t tell if it’s going to be part of a tap or not. Then I get another touch. Suppose it has a tap count of 2. Then that says that one of the previous touches has to be discarded. But to work out which, I have to look at the coordinates since there’s no other link between the pieces of the tap. Maybe I’ve been tapping away on my fun new virtual keyboard!

I can distinguish taps from touches by whether or not there’s a MOVING event between the BEGAN and ENDED. Then to work out what to do with a tap, I have to wait until I’m sure that there are no more coming (assuming that I’m coding something with support for multiple taps). If I want a double tap to do something, I need to wait until I’m sure it’s not a triple tap.

Woe betide me if I ever put in place a lock that requires one to tap out the blue Danube to release it!

This is the nature of handling touches on iOS. Gestures require complicated state machines to track. Apple’s gesture recognisers are really the answer to this - stand-alone state machines that identify a single parameterised gesture, and can be chained together and cancelled out of if necessary.

I think we should expose them via an API.

For example, to detect a triple tap you would use a tap gesture recogniser, set the number of taps to 3 and the number of fingers to 1. This would automatically perform the timing checks necessary to make sure it’s not a partial-4-tap before confirming the gesture.

I’ve been playing with this, though not intensive, and realized the potential problems of custom gesture detection. That’s why I proposed Codea devs to provide API for gesture detection which is more than just simple touch detection as currently provided. When one need to have a bit more complex interaction than just touch or tap, the code is beginning to get complicated. This is contradictive with Codea’s mission as tool for quick prototyping. :slight_smile:

I’m not holding my breath for the gestures API given how long Apple are taking over the current update, so although I agree that’s the way to go, I’m trying some elementary code in the meantime.

What I have at the moment is a bunch of classes. I have an extended Touch class which tracks a touch across all its events and saves some more information (such as starting time and coordinates). Then there’s a Touches class which organises Touches and figures out which other class is “handling” them. Each run, then it figures out the current active touches and passes them to whatever-it-is that is dealing with them. It groups the touches together into a Gesture, which is an array of touches, and passes that to whatever object has “claimed” the touches. The idea is that the object dealing with the touches can query the Gesture to see what type it is, or it can simply deal with the raw touches as they are. But that’s the bit I’ve still to code.

I think that the way I’ll deal with taps is to allow a Touch to persist beyond its natural lifetime if the handling object says it can. So if an object has a routine for 56 taps then when it gets a single tap it can say “Hold on to this one to see if another one is coming after it.” Then that touch (tap) doesn’t get deleted and gets passed again next time around.

It’s a bit of a balance as to which object is responsible for which bit. I feel as though handling objects shouldn’t have to deal with really complicated figuring out stuff and that that should be handled by the Gestures part. But then if the handler wants some really special input, then it should have freedom to be able to extend the gestures if it wants to.

(As with the first post in this thread, this is just me thinking aloud. Anyone’s welcome to join in, but no response is required!)

This is why I wrote my controllers library. I found deltaX/Y to be pretty useless (because of the misleading documentation I now realise), so wrote a library that let me write & compose objects that track and interpret touch gestures.

As it stands, Codea doesn’t expose enough events to let a program track touches correctly. If the user switches away from Codea, the running program doesn’t get ENDED events for active touches. So its state machines get out of sync with the OS.

Actually cancelled may be getting sent in the current version of Codea that you have. I believe we just haven’t exposed a global variable to reflect the CANCELLED state. Try print(touch.state) - I suspect cancelled shows up as an integer not covered by BEGAN / MOVING / ENDED. We have fixed this now, of course.

Yes, I saw events with a state of zero. That must be CANCELLED. Thanks.

I don’t understand how a touch gets into the CANCELLED state. What sequence of events leads up to that?

With one hand, start a touch gesture. Before you lift your fingers from the screen, with the other hand hit the home button.

Or, while a touch is happening, a push notification pops up.