Screen position of 3D point? [Solved]

@LoopSpace @Ignatz Thanks, it works great! Sorry, I didn’t learn all the math symbols in school, and I can’t seem to find anything on Khan Academy. I understand math when it’s code, though.

@SkyTheCoder Are there any particular symbols or is it general maths?

If you understand math[s] when it’s code, just pretend it is always code. After all, there’s not a big difference.

@LoopSpace I just don’t understand what all the symbols mean, the lines, the placement of numbers and symbols, all that stuff. I can understand when someone tells it to me, but with all the symbols it’s like a different language (literally, it’s only that I can’t read it)

A cheat I’ve used in the past is to draw a second version of the scene with setContext, and in that version just do the objects as flat colours. If you colour each object differently then on touch you can get the colour of the touched point, and based on the colour know what was touched.

Or for a surface you need x,y for you can use a gradient colour to determine where the touch landed…

I use this technique for a plane in https://gist.github.com/sp4cemonkey/5208579 from thread http://codea.io/talk/discussion/2465/my-grass-simulation-with-vertex-shaders/p1 although looking at that old code now, it’s a bit obscene :wink:

@SkyTheCoder - assuming you are looking straight down on your plane, ie along the normal, then how about this for a solution.

  1. You can calculate the 2D screen position of any 3D point in world space, using the answer to your initial question above

  2. Your 3D plane can be seen as a rectangle defined by 4 corner points

  3. You can calculate the 2D screen position for (say) the bottom left and top right corners, which gives you width and height, and then

  4. interpolate your 2D touch position between the points, to get the position on the plane

No nasty math required…

Of course, if your plane is at an angle, straight interpolation won’t work. But there’s another way, using binary search.

  1. Start with a rectangle for the plane as before (before it is drawn)

  2. take the centre point of the rectangle, call it c

  3. set dx, dy = WIDTH/4, HEIGHT/4

  4. calculate the 2D screen position of c (when drawn in 3D, rotated etc), call it c2

  5. If touch.x > c2.x, c.x = c.x+ dx, otherwise c.x = c.x- dx

  6. do the same for y

  7. dx, dy = dx/2, dy/2

  8. go back to 4 and repeat until c2:dist(p) < 1 or dx<=1

This looks like a lot of steps, but they are all simple, and even step 3 is only a couple of multiplications, because the required matrix values can be pre-calculated.

I believe there is another method, too. Draw your plane to an image in memory, clipping it to the 1x1 pixel that you touched, creating texture coordinates that map from (0,0) to (1,1), and using a fragment shader that encodes the (x,y) texture coordinates into the pixel colours. Then read the touched pixel colour and decode the coordinates.

Both of these methods should give sufficiently exact answers for any purpose, and they should be guaranteed to work, unlike the usual solution of creating a ray by inverting matrices. (Also, I understand these methods, which I like :smiley: )

@spacemonkey I never thought about using a gradient shader and then getting the color from the touch’s position!

@Ignatz The main game I was using this for was actually not looking straight down, but sort of at an angle. Wouldn’t a “binary search” be a bit laggy at 60 FPS, though? I’d be willing to learn more advanced maths if it was faster.

@SkyTheCoder - if your plane is full screen, then a gradient shader will only give you an accuracy of within 4 pixels in the x direction, because there are only 256 colours and 1024 pixels. That’s why I suggested encoding the x and y values in colours, which sounds the same as a gradient shader, except you use the z value to provide additional accuracy for the x and y values. Also clip the image to 1x1.

I’ll demo my interpolation, no advanced math is necessary, I assure you.

@SkyTheCoder - try this. Touch the yellow plane.

It assumes the plane is XY, which is easily changed if necessary.

It takes about 0.00015 sec on my iPad3, which means it takes up just 1/110 of one draw cycle.

It works by starting in the middle, seeing where that point gets drawn, and adjusting the guess to get closer and closer to the touch point. You’ll see there is no fancy math, and no dangerous matrix inversions.

function setup()
    --plane settings
    plane={}
    plane.centre=vec3(0,0,0)
    plane.rotate=vec3(45,0,20)
    plane.size=vec2(70,100)
    --calculate factors needed for interpolation
    --set up camera and rotations in a function so we are sure the settings are the same here and in draw
    SetupPerspective()
    M=modelMatrix()*viewMatrix()*projectionMatrix()
    parameter.text("Touch","") --shows results
end

function draw()
    background(50)
    translate(plane.centre.x,plane.centre.y,plane.centre.z)
    SetupPerspective() --camera and rotation
    --draw plane
    fill(255,255,0)
    rect(-plane.size.x/2,-plane.size.y/2,plane.size.x,plane.size.y)
    --draw touch to show it worked
    if p then 
        translate(p.x-plane.size.x/2,p.y-plane.size.y/2,1)
        fill(255,0,0)
        ellipse(0,0,5)
    end
end

function SetupPerspective()
    perspective()
    camera(0,0,200,0,0,-1)
    rotate(plane.rotate.x,1,0,0)
    rotate(plane.rotate.y,0,1,0)
    rotate(plane.rotate.z,0,0,1)
end

function touched(t)
    if t.state==ENDED then 
        p=GetTouchPosition(t) 
        Touch=tostring(p)
    end
end

--binary iteration
function GetTouchPosition(t)
    --start with centre of plane
    local c=vec3(plane.centre.x,plane.centre.y,plane.centre.z)
    --set initial step size to 1/4 size
    local d=vec2(plane.size.x/4,plane.size.y/4)
    --iterate until we get down to 1 pixel
    while d.x>1 do
        --get 2D position of 3D point
        local m=M:translate(c.x,c.y,c.z)
        local x=(m[13]/m[16]+1)*WIDTH/2
        --if it is greater than x, we need to go left, so adjust c by step size, and vice versa
        if x>t.x then c.x=c.x-d.x else c.x=c.x+d.x end
        --same for y
        local y=(m[14]/m[16]+1)*HEIGHT/2
        if y>t.y then c.y=c.y-d.y else c.y=c.y+d.y end
        --halve step size
        d=d/2
    end
    --calculate position of point on plane 
    return c-plane.centre+vec3(plane.size.x/2,plane.size.y/2,0)
end




@Ignatz It works pretty good, but seems to be a bit jumpy and unprecise, such as in this image (white circle is my touch) Any idea why?

Image

@Loopspace Out of curiosity, could you comment your code? I tried to understand it, but got stuck at all the cofactor functions with all the exponents and modulus.

@SkyTheCoder - I see what you mean. Try this amended function. It runs slower, but still only takes 1/10 of a draw cycle.

function GetTouchPosition(t)
    --start with centre of plane
    local c=vec3(plane.centre.x,plane.centre.y,plane.centre.z)
    local s=0.7 --step
    local d=vec2(plane.size.x*s*s,plane.size.y*s*s)
    --iterate until we get down to 1 pixel
    local error
    while true do
        local m=M:translate(c.x,c.y,c.z)
        local x=(m[13]/m[16]+1)*WIDTH/2
        local y=(m[14]/m[16]+1)*HEIGHT/2
        --if it is greater than x, we need to go left, so adjust c by step size, and vice versa
        if x>t.x then c.x=c.x-d.x else c.x=c.x+d.x end
        --same for y        
        if y>t.y then c.y=c.y-d.y else c.y=c.y+d.y end
        --halve step size
        d=d*s
        error=vec2(x,y):dist(vec2(t.x,t.y))
        if error<.2 or d.x<0.05 then break end
    end
    --calculate position of point on plane, and error
    return c-plane.centre+vec3(plane.size.x/2,plane.size.y/2,0),error
end

@SkyTheCoder - here’s one that does it with an image and shader, about as fast as the code immediately above (1/600 second)

function setup()
    --plane settings
    plane={}
    plane.centre=vec3(0,0,0)
    plane.rotate=vec3(45,0,20)
    plane.size=vec2(500,500,0)
    --set up mesh, needed for shader
    plane.mesh=mesh()
    local s=plane.size
    local x1,y1,x2,y2,z=-s.x/2,-s.y/2,s.x/2,s.y/2,s.z
    plane.mesh.vertices={vec3(x1,y1,z),vec3(x2,y1,z),vec3(x2,y2,z),vec3(x2,y2,z),vec3(x1,y2,z),vec3(x1,y1,z)}
    plane.mesh.texCoords={vec2(0,0),vec2(1,0),vec2(1,1),vec2(1,1),vec2(0,1),vec2(0,0)}
    plane.mesh:setColors(color(255,255,0))
    parameter.text("Touch","") --shows results
end

function draw()
    background(50)
    pushMatrix()
    SetupPerspective()
    --draw plane
    plane.mesh:draw()
    --draw touch to show it worked
    if p then 
        translate(p.x-plane.size.x/2,p.y-plane.size.y/2,1)
        fill(255,0,0)
        ellipse(0,0,20)
    end
    popMatrix()
end

function SetupPerspective()
    perspective()
    camera(0,0,900,0,0,-1)
    translate(plane.centre.x,plane.centre.y,plane.centre.z)
    rotate(plane.rotate.x,1,0,0)
    rotate(plane.rotate.y,0,1,0)
    rotate(plane.rotate.z,0,0,1)
end

function touched(t)
    if t.state==ENDED then
        p=GetPlaneTouchPoint(t)
        Touch=p
    end
end

function GetPlaneTouchPoint(t)
    local img=image(WIDTH,HEIGHT)
    setContext(img)
    pushMatrix()
    SetupPerspective()
    clip(t.x-1,t.y-1,3,3)
    plane.mesh.shader=shader(PlaneShader.v,PlaneShader.f)
    plane.mesh:draw()
    setContext()
    popMatrix()
    local x2,y2,x1y1=img:get(t.x,t.y)
    local y1=math.fmod(x1y1,16)
    local x1=(x1y1-y1)/16
    local x,y=(x1+x2*16)/4096*plane.size.x,(y1+y2*16)/4096*plane.size.y
    plane.mesh.shader=nil
    return vec2(x,y)
end

PlaneShader = {

v = [[
uniform mat4 modelViewProjection;
attribute vec4 position;
attribute vec2 texCoord;
varying highp vec2 vTexCoord;

void main()
{
    vTexCoord = texCoord;
    gl_Position = modelViewProjection * position;
}
]],
f = [[
precision highp float;
varying highp vec2 vTexCoord;

void main()
{
    highp vec2 T = vTexCoord * 4096.0;
    highp float x1=mod(T.x,16.0);
    highp float x2=(T.x - x1) / 16.0;
    highp float y1=mod(T.y,16.0);
    highp float y2=(T.y - y1) / 16.0;
    gl_FragColor = vec4(x2/255.0,y2/255.0,(x1*16.0+y1)/255.0,1.0);
}
]]}

@Ignatz But now, using it too much causes Codea to crash because it runs out of memory…

instead of

function GetPlaneTouchPoint(t)
    local img=image(WIDTH,HEIGHT)
    setContext(img)

try

local img=image(WIDTH,HEIGHT)
function GetPlaneTouchPoint(t)
    setContext(img)
    background(0)

it may do it.

@Jmv38 - good idea. You don’t need the background command, though.

Actually, I think the best may be to define img as a global in setup. I was able to do many touches without problems.

NB @SkyTheCoder - the image shader provides high resolution accuracy up to a screen size of 4096x4096 because the z value is used to extend the x and y colour values by a factor of 16 each. That’s what all the formulae are for.

(commenting to get notifications on this topic)

@Ignatz Now it’s working when img is in global, but why was it a little bit off when it was local?

Image

@matkatmusic you can also bookmark the thread, by pressing the little star icon next to it.

No idea

@SkyTheCoder - even better, start by creating an image with all the pixel positions coded, then you only have to do it once, and each touch is a simple lookup.

Run this from setup

function Preset()
    img=image(WIDTH,HEIGHT)
    setContext(img)
    pushMatrix()
    SetupPerspective()
    plane.mesh.shader=shader(PlaneShader.v,PlaneShader.f)
    plane.mesh:draw()
    setContext()
    popMatrix()
    plane.mesh.shader=nil
end

Then getting the touch is simply a lookup, super fast

function GetPlaneTouchPoint(
    local x2,y2,x1y1=img:get(t.x,t.y)
    local y1=math.fmod(x1y1,16)
    local x1=(x1y1-y1)/16
    local x,y=(x1+x2*16)/4096*plane.size.x,(y1+y2*16)/4096*plane.size.y
    return vec2(x,y)
end