how do you convert touch 2d to touch 3d?

[Andrew_Stacey]

I just played a bit more. I put camera(0,0,10,0,0,0,0,1,0) and perspective(40,HEIGHT/WIDTH) (or was it WIDTH/HEIGHT - whatever the standard was). I then drew a basic object on the screen and gave myself a parameter to vary its z-coordinate. As expected, when the z-coordinate gets to about 9 then it reaches the “front” of the screen and gets clipped. But the back is a long way off! In fact, I suspect that there is no back. I got up to z = -1000 and my object was still there (albeit as a single pixel). This semi-infinite ray gets squashed into the interval [-1,1] and so the inversion formula is quite bad.

I’ve yet to find any code for glUnProject that does anything any better than inverting the matrix. I had one idea which would be that since we’re using homogeneous coordinates, it’s always okay to work with a multiple of the matrix or the vector. So whenever there’s an operation which can introduce inaccuracies and which involves multiplication or division then it might be possible to avoid it. For example, instead of inverting the matrix you can use the cofactor matrix. But I don’t think that these fix things too well.

There are two steps needed:

1. Convert the touch to a ray for answering “Was this object touched?”
2. Convert the touch to a point at a specific z-level as specified by the object that claimed the touch so that it can deal with it in its own frame of reference.

For the first, we need two points and two that can be determined accurately. For that, I think we need to be near the front of the screen, but not too near. Good values are going to be ones that give a solid value for the fourth coordinate (since we end up dividing by that). I wonder if this is predictable.

For the second, we use the routine above with the specific z-value and have to live with the inaccuracy. Hopefully the object will be nicely in the frustum and so the inaccuracy won’t be too bad.

[aciolino]

I’ll have to examine my code more; I can’t seem to pass a z value outside of my above post and get any reasonable results…either the cube is ‘on’ the camera, or it’s in the distance if its in the above tiny range. I understand that ndc[3] has to be in -1 to 1. Maybe I need to move my camera target to the object before I rush to judgement on the values.

I did play with multiples instead of inversion, but found that I didn’t know what I was doing so I rolled back and decided to try to do a ray picker instead.

At the end of the day, the idea is still to touch an object on the screen and detect if I hit it. If so, I add a new object at the touch point (to prove I can), and I can steal z from the touched object.

I’m just trying to figure out why MY z’s can’t fit in the range you mention above (9 to -1000)…

[Andrew_Stacey]

Here’s my current testing code

function setup()
    --displayMode(FULLSCREEN_NO_BUTTONS)
    cube = mesh()
    corners = {}
    for l=0,7 do
        i,j,k=l%2,math.floor(l/2)%2,math.floor(l/4)%2
        table.insert(corners,{vec3(i,j,k),color(255*i,255*j,255*k)})
    end
    vertices = {}
    colours = {}
    local u
    for l=0,2 do
        for i=0,1 do
            for k=0,1 do
                for j=0,2 do
                    u = (i*2^l + ((j+k)%2)*2^((l+1)%3)
                        + (math.floor((j+k)/2)%2)*2^((l+2)%3)) + 1
                    table.insert(vertices,corners[u][1])
                    table.insert(colours,corners[u][2])
                end
            end
        end
    end
    cube.vertices = vertices
    cube.colors = colours
    parameter("x",-5,5,0)
    parameter("y",-5,5,0)
    parameter("z",-10,10,0)
    watch("mid")
    watch("mat")
    watch("dbg")
    cubepos = vec3(0,0,1)
    watch("cubepos")
end

function draw()
    background(40, 40, 50)
    noSmooth()
    noStroke()
    fill(0, 255, 238, 255)
    ellipse(CurrentTouch.x,CurrentTouch.y,15)
    resetMatrix()
    perspective(40, WIDTH/HEIGHT)
    camera(0,0,10,  0,0,0,  0,1,0)
    mat = modelMatrix()*viewMatrix()*projectionMatrix()
        pushMatrix()
        translate(x,y,z)
        cube:draw()
        popMatrix()
        pushMatrix()
        cubeMatrix = modelMatrix()*viewMatrix()*projectionMatrix()
        translate(cubepos.x,cubepos.y,cubepos.z-1)
        cube:draw()
        popMatrix()
        pushMatrix()
        mid = converttouch(0)
        translate(mid.x,mid.y,mid.z)
        scale(.01)
        --ellipse(0,0,2)
        popMatrix()
end

function touched(touch)
    local zl = getzlevel(cubepos,cubeMatrix)
    local tc = converttouch(zl,nil,cubeMatrix)
    dbg = tc
    if touch.state == BEGAN then
        if tc.x > cubepos.x and tc.x < cubepos.x + 1 and
            tc.y > cubepos.y and tc.y < cubepos.y + 1 then
                cubetouch = tc
        else
            cubetouch = false
        end
    else
        if cubetouch then
            local zc = cubepos.z
            cubepos = tc - cubetouch
            cubepos.z = zc
        end
    end
end

function converttouch(z,t,A)
    A = A or modelMatrix() * viewMatrix() * projectionMatrix()
    t = t or CurrentTouch
    z = z or 0
    local m = cofactor(A)
    local ndc = {}
    local a
    ndc[1] = (t.x/WIDTH - .5)*2
    ndc[2] = (t.y/HEIGHT - .5)*2
    ndc[3] = z -- 2.0 * 5 - 1.0 --touch.z , fake with lookAt.z
    ndc[4] = 1
    a = applymatrix(ndc,m)
    if (a[4] == 0) then return end
    a = vec3(a[1], a[2], a[3])/a[4]
    return a
end

function getzlevel(v,A)
    A = A or modelMatrix() * viewMatrix() * projectionMatrix()
    v = v or vec3(0,0,0)
    local u = applymatrix(vec4(v.x,v.y,v.z,1),A)
    if u[4] == 0 then return end
    return u[3]/u[4]
end

function applymatrix(v,m)
    local u = {}
    u[1] = m[1]*v[1] + m[5]*v[2] + m[09]*v[3] + m[13]*v[4]
    u[2] = m[2]*v[1] + m[6]*v[2] + m[10]*v[3] + m[14]*v[4]
    u[3] = m[3]*v[1] + m[7]*v[2] + m[11]*v[3] + m[15]*v[4]
    u[4] = m[4]*v[1] + m[8]*v[2] + m[12]*v[3] + m[16]*v[4]
    return u
end

function cofactor(m)
    local rm = matrix()
    local sgn,l
    local fm = {}
    for k=1,16 do
        fm = {}
        l = math.floor((k-1)/4) + 1 + 4*((k-1)%4)
        sgn = (-1)^(math.floor((k-1)/4))*(-1)^((k-1)%4)
        for j=1,16 do
            if j%4 ~= k%4 
                and math.floor((j-1)/4) ~= math.floor((k-1)/4)
                then
                    table.insert(fm,m[j])
            end
        end
        rm[l] = sgn*Det(fm)
    end
    return rm
end
                
function Det(t)
    return t[1]*t[5]*t[9]
         + t[2]*t[6]*t[7]
         + t[3]*t[4]*t[8]
         - t[3]*t[5]*t[7]
         - t[2]*t[4]*t[9]
         - t[1]*t[6]*t[8]
end

There’s probably a few bits of excess code there hung over from other experiments. One thing to note: in my previous experiment I had the touch test for the back of the cube. That worked, but it was a bit odd moving the cube by its back face. So I shifted it to test to the front. However, then the accuracy was low enough that the cube essentially moved continually backwards, hence the lines:

            local zc = cubepos.z
            cubepos = tc - cubetouch
            cubepos.z = zc

which save the z position and restore it after the touch.

There were a few other experiments I did that ended up with similar inaccuracies. So anything that uses this sort of code is going to have to build in some sort of robustness to ensure that small errors in the numbers don’t accumulate into large effects, and that “silly” numbers get dealt with properly.

[aciolino]

Ok, so now you’ve introduced cofactors. How is a cofactor better/different from inversion? I guess I could google that.

applymatrix() is could be called Matrix4xVector4, effectively, right? So why does line 94,

    a = applymatrix(ndc,m)

Work, or more importantly, needed? I realize a matrix has 4 Vec4’s but something seems weird to me to call it on two matrices.

Why did you write Det()? Isn’t it the same as matrix:determinant()? OIC, it’s a table vs. a matrix. Nice.

[Andrew_Stacey]

Technically it’s the “matrix of cofactors”: http://en.wikipedia.org/wiki/Cofactor_(linear_algebra) It is related to the inverse in that cof(a)/det(a) = inv(a) so it avoids dividing by the determinant.

In a = applymatrix(ndc,m) then ndc is really a 4-vector and m a 4x4 matrix so it is vector * matrix (order matters here).

In my library then I have more general matrix and vector handling code and once this lot is sorted out then I’ll add it to that, so I’m keeping things fairly general in this code. Plus several of the methods for matrices and vectors don’t exist as yet (such as multiplying a vector by a matrix).

[aciolino]

Re: avoid the divide by zero: neat!

[aciolino]

I think I will try to cake a modification so that touch gets the x/y/false z, and drag will allow the z to change based on screen Y, we basically push/pull by holding.

This would probably work for things that have to ‘fall’ to the ground after creation.