Solution for identifying what object was touched on a 3D scene

Code Sharing
21

Reading in more detail the discussing above, i’ve noticed a very sensible remark from Andrew: ‘you need information of only one pixel’. This is very correct. And we dont want to have to think about it (my sensible input). This is also correct (i usually agree with myself. After a lot of debate.). So, maybe, a simple way to mix both would be to

  • have a very small rendering image 10x10,
  • translate all the scene so that the point we are touching is in the middle of this image.
  • and let the GPU do the hard work by using setContext and draw().
    The GPU will be much faster because it kind of know the pixels ‘out of the field’ and doesnt draw them (i think). I’ve got to try that one!
22

After doing a few changes, i think a part of the problem comes from the shader declaration. I think it could be faster if the shader could be there all the time, and only activated on demand, with a boolean for instance. Not added on demand, as curently. I dont know how to do that (i dont know much about shaders).

23

I can fix the shader, if you share what youvedone

24

@Ignatz perhaps this might give some insight into matrix multiplication :slight_smile:

Using the example from @Andrew_Stacey’s tutorial on matrices:
http://loopspace.mathforge.org/discussion/13/matrices-in-codea

-- rotate y axis
rotate(90,0,1,0)
mb = modelMatrix()
resetMatrix()

-- rotate x axis
rotate(90,1,0,0)
ma = modelMatrix()
resetMatrix()

-- combined transform
rotate(90,0,1,0)
rotate(90,1,0,0)
mc = modelMatrix()

print("Rotate")

print("mc=")
print(mc)
print("ma*mb")
print(ma*mb)
print("mb*ma")
print(mb*ma)

This does the same multiplication manually and outputs what’s happening at each iteration to show how the result is calculated to get to (ma*mb)=mc :slight_smile:

-- matrix A equivalent to (ma)
-- resetMatrix()
-- rotate(90,1,0,0)
-- ma = modelMatrix()
mta={{1.00, 0.00, 0.00, 0.00},
 {0.00, -0.00, 1.00, 0.00},
 {0.00, -1.00, -0.00, 0.00},
 {0.00, 0.00, 0.00, 1.00}}

-- matrix B equivalent to (mb)
-- resetMatrix()
-- rotate(90,0,1,0)
-- mb = modelMatrix()
mtb={{-0.00, 0.00, -1.00, 0.00},
 {0.00, 1.00, 0.00, 0.00},
 {1.00, 0.00, -0.00, 0.00},
 {0.00, 0.00, 0.00, 1.00}}

-- Result matrix (mc)
-- will be equivalent to (ma*mb) or (mb*ma) (in this case)
mr={{0,0,0,0},
{0,0,0,0},
{0,0,0,0},
{0,0,0,0}}

-- output vars
local mtstr =""
local index =""
local map =""

for row=1,4 do
    
    for col =1,4 do
        -- create output 
        index = index.."mr["..row.."]["..col.."]="..mr[row][col].."\
"
        for cur = 1,4 do
            -- capture before for output
            local before = mr[row][col]
            
            -- multiply: for each row in Martix A multiply left side across with each column from the top down in Matrix B and add to the current result / row and column being calculated
            mr[row][col] = mr[row][col]+(mta[row][cur]*mtb[cur][col])
            
            -- more output
            index = index .. "mr["..row.."]["..col.."]+=(mta["..row.."]["..cur.."]*mtb["..cur.."]["..col.."])"
            index = index .. "=\
"..before.."+("..mta[row][cur].."*"
            ..mtb[cur][col].."="..(mta[row][cur]*mtb[cur][col])..")="..mr[row][col].."\
"
        end
        
        -- more output
        mtstr=mtstr..string.format("%.2f",tostring(mr[row][col]))..","
        map = map.."mr["..row.."]["..col.."] "
        index = index.."mr["..row.."]["..col.."]="..mr[row][col].."\
\
"
        
        -- column moves one down
    end
    -- more output
    index = index.."\
"
    mtstr = mtstr.."\
"
    map = map.."\
"
    
    -- row moves one down
end


print("Mapping:\
"..map)
print(index)
print("Results:")
print(mtstr)
print("==")
print(mc)
25

@XanDDemoX - :smiley:

26

@Jmv38 - I fixed up the shader and the image definition. Code is here

https://gist.github.com/dermotbalson/5906898

It seems to run pretty fast on my iPad 1

27

@ignatz.
I tried your new code (with my bouncing girl added, you should try it).
=> no lag at all.
=D> =D> =D> =D> =D> =D>
So the problem was really the shader definition, not the image size.
YOU’VE MADE IT!!!
Real time 3d touch detection, effortless for the developper.
Congrats.

28

@Jmv38- that’s cool. I’ll still try the math approach, but at least we have a solution that is practical and fast enough (and pixel perfect if we need it).

I’ll tidy it up so we can share it.

PS What is your bouncing girl? Share!

29

That 'bouncing girl is simply the 5 lines of code i mentionned in my first post above in this discussion, and add at the end of your draw loop. It really help to see the lag (although in this last version you’ll see almost nothing).
This code of you is going to be really useful to people who want to do 3d games!

30

@Jmv38 - generalised code is here (long press into Codea, 2 tabs)

https://gist.github.com/dermotbalson/5907888

I split all the touch code into a separate tab. It only requires 3 extra lines of code in the normal code, all marked with asterisks.

Improvements welcome!

31

I made my own version for Codeler, and it’s working, but for some reason when I touch on something everything disappears. I can still touch in the spot where it was and it select it, but I can’t see what I’m touching. I used setContext() to set the rendering target back to the screen, too. My code (simplified):

function touched(touch)
    if touch.state == BEGAN then
        local i = image(WIDTH, HEIGHT)
        setContext(i)
        background(0)
        for k,v in ipairs(boxes) do -- Iterate table of cubes
            resetMatrix()
            perspective(65)
            camera(5, 5, 5, 0, 0, 0)
            translate(0, 0, 0)
            local model = mesh()
            model.vertices = v.cverts -- Cube vertices
            model.texture = TextureName
            model.texCoords = v.tverts -- Cube texture coordinates
            model.shader = shader(vS, fS)
            model.shader.id = k
            model:draw()
        end
        resetMatrix()
        setContext()
        
        local r, g, b, a = i:get(touch.x, touch.y)
        if a ~= 0 and r + g + b ~= 0 then
            SelectedBox = r + g + b
        end
    end
end

vS = [[
//
// A basic vertex shader
//

//This is the current model * view * projection matrix
// Codea sets it automatically
uniform mat4 modelViewProjection;

//This is the current mesh vertex position, color and tex coord
// Set automatically
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;

//This is an output variable that will be passed to the fragment shader
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
    //Pass the mesh color to the fragment shader
    vColor = color;
    vTexCoord = texCoord;
    
    //Multiply the vertex position by our combined transform
    gl_Position = modelViewProjection * position;
}
]]

fS = [[
//
// A basic fragment shader
//

//Default precision qualifier
precision highp float;

//This represents the current texture on the mesh
uniform lowp sampler2D texture;

//The interpolated vertex color for this fragment
varying lowp vec4 vColor;

//The interpolated texture coordinate for this fragment
varying highp vec2 vTexCoord;

uniform float id;

void main()
{
    //Sample the texture at the interpolated coordinate
    lowp vec4 col = texture2D( texture, vTexCoord ) * vColor;
    
    lowp vec4 idColor = vec4(0.0, 0.0, 0.0, 1.0);
    
    if (id > 255.0)
    {
        if (id > 510.0)
        {
            idColor = vec4(1.0, 1.0, id / 255.0, 1.0);
        }
        else
        {
            idColor = vec4(1.0, id / 255.0, 0.0, 1.0);
        }
    }
    else
    {
        idColor = vec4(id / 255.0, 0.0, 0.0, 1.0);
    }
    
    //Set the output color to the texture color
    if (col.a >= 0.75)
    {
        gl_FragColor = idColor;
    }
    else
    {
        gl_FragColor = vec4(0.0, 0.0, 0.0, 0.0);
    }
}
]]

Help?

32

@SkyTheCoder - I can’t see an obvious problem, or an endless loop. What does your draw function look like?

33

@SkyTheCoder It’s a lot easier to test and debug if you post the full code, or at least code that can be run without guessing the rest of it.

34

@Jmv38 - I edited the code so that when you touch, and draw to an image, it is clipped to the few pixels around the touch point, which should cut down a lot on the GPU load.

Same link as before, see what you think
https://gist.github.com/dermotbalson/5907888

35

@SkyTheCoder - I’ve been discussing the math approach with @Andrew_Stacey, and he provided some nice code to let you pick - and then drag - cubes around the screen. The code “decides” which face of each cube is going to be touch sensitive.

Note that because a touch is 2D, you can only drag the cube in two dimensions x and y, ie z stays the same. Also that more code will be needed if you want to touch and drag other shapes, like spheres.

All of which confirms my feeling that 3D is hard, really hard.

Anyway, here is Andrew’s code: https://gist.github.com/dermotbalson/5914883

36

@ignatz i have tested your last version: i cannot see any slow down in this one, on my ipad1!. So i think your solution for 3d touch is just great: simple to use, quick, no head-ache… Thanks for sharing. Make sur you put it in your tutos so it doesnt get lost in the forum.

37

@Jmv - I have done a tutorial, thanks. I also tested the effect of clip, and it was pretty small. Basically, the hidden image takes the same time to draw as it would on the screen, so the cost is basically one frame.