Screen position of 3D point? [Solved]

LoopSpace · January 5, 2015, 4:29pm

@SkyTheCoder There’s a commented version at https://gist.github.com/loopspace/5937499.

Ignatz · January 5, 2015, 10:19pm

@LoopSpace - there are shader compilation errors, because you can’t define shaders before setup runs, in the latest version of Codea. You need to wrap your cube and sphere setup code in functions.

Ignatz · January 6, 2015, 12:57am

@LoopSpace - can you please confirm the way you choose the rotation for the objects? Specifically - you choose a rotation vector

local th = 2*math.pi*math.random()
local z = 2*math.random() - 1
local r = math.sqrt(1 - z*z) 
rotation = vec4(r*math.cos(th),r*math.sin(th),z,360*math.random())

and then rotate

mm:rotate(self.rotation.w,self.rotation.x,self.rotation.y,self.rotation.z)

[which is a little confusing at first, because the components of the
rotation vector (x,y,z,w) are used in a different order (w,x,y,z) ]

So the x,y,z components define a normalised axis. z is first given a random value, and the length of (x,y,z) must equal 1, so the total length of the x,y components is 1-z*z, which explains the choice of r.

Then the x component = r*math.cos(th) and y = r*math.sin(th), which is the way you calculate x,y values for angle th and radius r. The squares add up to r*r = 1 - z*z, as required.

So it looks to me as though, having chosen a random direction for z, you then choose a further random rotation on the xy axis such that the overall xyz axis is normalised.

Can you please confirm (or if not, please clarify)?

Ignatz · January 6, 2015, 1:24am

@LoopSpace - apologies for all the questions, but these are really valuable functions, so I’m looking at the code very carefully.

I notice that you test for a touch by working through a list of objects, and the first object to claim a touch gets it. If one object is partly in front of another, and you touch the first one, the back object would claim the touch if it was polled first. You provide a solution by pushing a touched object to the back of the list, so if you touch again, you should get the front object this time.

This would be ok if you’re just arranging objects on the screen and don’t mind a bit of toggling, but if you’re playing a fast moving game, touch needs to be both consistent and accurate. If you touch the front object, you expect to always get the front object, and toggling between front and back objects would be a player nightmare.

So I don’t see how the code could be used in a game, as it stands. Could you perhaps poll all objects, and if you have multiple claimants, pick the touch point closest to the camera?

Ignatz · January 6, 2015, 2:22am

I think there is a bug in line 105, you pass t through to isTouchedBy, but the touch parameter is touch, not t, which is nil. The code still works, I think because your functions have a line t = t or CurrentTouch().

Ignatz · January 6, 2015, 3:23am

Should the addRect for the Picture be defined with x,y = 0,0 rather than 0.5, 0.5, since addRect centres the rectangle on the x,y values?

LoopSpace · January 6, 2015, 3:39pm

@Ignatz

Shader: Thanks, I hadn’t tried running this code with the latest Codea so hadn’t spotted that.
Rotations: Yes, that’s about it. The vec4 is a rotation using the angle-axis encoding. Since the axis is a 3-vector, it felt best to use the x, y, z components of the vec4 for it, leaving the w component for the angle. However, if you define a vec4 as r = vec4(1,2,3,4) then the order of the components is x,y,z,w. Hence the w component gets swapped from last when defined to first when fed into the rotation. The rotation is chosen “at random” by choosing the axis and the angle each at random (this isn’t actually the best way to choose a random rotation, but for this code cheap-and-cheerful is good enough). I don’t know how much detail you want me to go into here, but the way I choose the x,y,z components ensures that the axis is chosen properly “at random”.
Touch Order: You have to remember where I’m coming from here. I think I can lay claim to writing the first 3D program in Codea, and it was a shape explorer where you defined a shape by vertices and edges and could move those around. In that situation, it is common to want to select an object that happens to be behind another so the z-ordering was never the “right” way to decide which object to query first. If you want to ensure that you always pick the front object, then there are various ways you could modify the querying routine but I’d want to do a bit of testing as to which was fastest before deciding which is best.
Touch (Bug): Yup, that’s a bug. Thanks.
Picture: I’m consistent, that’s the main thing here. The PIcture’s internal coordinates are set so that it is in the unit square, and that’s used when testing touches. If I want the position of the Picture to correspond to its centre, rather than its lower-left corner, then I’d also need to adjust the touch test. But that would be straightforward. I think that the Cube actually has the same property: it is the “unit cube” so the position is actually the position of the lower-left-back corner.

No worries on the questions.

Ignatz · January 6, 2015, 4:31pm

@LoopSpace - thanks for all of that.

A couple of suggestions.

Your classes expect an axis-angle rotation parameter, but (yourself excluded) I doubt that very many Codea users would know how to create that. The alternative of a set of Euler angles, or maybe the current model matrix (which means the user can make the rotation they want, then call the class) will make the code more accessible.
Your code allows the user to move shapes along the touched plane. While interesting, this is unlikely to be a common requirement. More common is going to be the need to identify which object was touched, and in some cases, where it was touched. You have a class property which identifies the touch point, so this would not be hard to do. I would add a class function to fetch this value.
I do think users will always expect - and want - the front object to be touched. I can’t think of any situations where this wouldn’t be the case. (If you want to move stuff around, the common behaviour in every program I’ve ever worked with, is that touch always affects the front object, and if you want to to get at objects behind, you move the front object out of the way).
A greatly simplified example that shows how to identify touched objects and touch locations, will make it much easier for other people to use the classes.

Finally, I have a couple of questions on the utility functions at the end.

Why is applymatrix needed, when Codea already allows a 4D matrix x vector operation?

Is there a way to explain how any of _planetoscreen, screentoplane and screenframe work, in other than purely mathematical terms?

For example, I explained some of the code in your matrix post in a comment above like this -

"So what the matrices do is to convert a 3D point to its final position in front of the camera.

Then in the last part, m[13] is the x translation, which is scaled for distance by dividing it by the factor in m[16], then the same is done for y."

That’s the kind of explanation that might help me and others to understand these functions.

LoopSpace · January 6, 2015, 5:17pm

@Ignatz

Well, ideally I’d have it work with quaternions … I chose angle-axis because that’s how Codea handles rotations with the rotate function and the matrix:rotate method. I didn’t want to add unnecessary code beyond what I wanted to demonstrate. I agree that this code is fairly rudimentary, but it was demonstration code rather than sample code, if you see what I mean.
The point here was to make the point that you can’t go from the 2D screen to 3D world space. Rather, you should define a “plane of movement” in the 3D world and confine yourself to that; then it is possible to cleanly transfer a touch from the screen to that plane. It is for the object to define that plane, though. The Cube ensures that it is parallel to one of its faces while the Sphere uses the plane orthogonal to the camera.
That may be the common behaviour, but it’s really really annoying behaviour. I don’t like the fact that to get to the object behind then I have to move all the others out of the way! But as I said, this is easy to modify.
“Touch location” isn’t necessarily well-defined due to the 2d-3d problem.
Re “applymatrix”. I suspect that it is because when I originally wrote that code, then Codea didn’t have this function.
I’ll have a think and see if I can come up with an explanation.

Ignatz · January 8, 2015, 8:33am

@SkyTheCoder

I believe this minimal code - plus @LoopSpace’s code - will give you the (x,y) touch point as a fraction of width,height, eg (0.4,0.6)

Assuming you set up your plane with Euler angles, I’ve provided a function that converts to an axis angle vector as required by LoopSpace.

You’ll need to make the shader correction I mentioned above. I think you already know about this problem.

function setup()
    plane={}
    plane.pos=vec3(0,0,-250)
    plane.size=vec3(700,700,1)
    plane.rotation=vec3(45,0,20)
    --create axis/angle vector from x,y,z Euler rotations
    r=AxisAngleFromEuler(plane.rotation)
    p=Picture(plane.pos,plane.size,r,"Cargo Bot:Starry Background")
    light=vec3(-0.717,0,0.717)
    ambient=0.5
end

function draw()
    background(50)
    perspective()
    camera(0,100,1400,0,0,-1)
    p:draw()
end

function touched(t)
    if t.state==ENDED then
        p:isTouchedBy(t) 
        touchPoint=vec2(p.starttouch.x,p.starttouch.y)
        print(touchPoint)
    end
end 

function AxisAngleFromEuler(r)
    pushMatrix()
    rotate(r.x,1,0,0) --reorder these if you want
    rotate(r.y,0,1,0)
    rotate(r.z,0,0,1)
    local m=modelMatrix()
    popMatrix()
    local a=math.deg(math.acos((m[1]+m[6]+m[10]-1)/2))
    local d=math.sqrt((m[10]-m[7])^2+(m[3]-m[9])^2+(m[2]-m[5])^2)
    local x=(m[10]-m[7])/d
    local y=(m[3]-m[9])/d
    local z=(m[2]-m[5])/d
    return vec4(x,y,z,a)
end

Ignatz · January 8, 2015, 8:41am

By the way, I did a speed test comparison of

the picture-based approach I suggested earlier, using a coded picture created at startup, so that each touch only requires a single pixel lookup, and
the LoopSpace approach

The clear winner is LoopSpace, running about 7 x faster.

But since even the “slow” picture approach can do 1,000,000 lookups in about 13 seconds, ie one lookup takes 0.08% of a draw cycle, I doubt that it is going to matter either way.

SkyTheCoder · January 8, 2015, 1:23pm

@Ignatz It says I’m missing a Picture class. Do you know what that is?

Ignatz · January 8, 2015, 6:24pm

@SkyTheCoder - yes, that’s part of LoopSpace’s code, which you will need to include as well (and fix his shader)

SkyTheCoder · January 8, 2015, 7:08pm

@Ignatz All right, I got it working. Is it printing the texture coordinate of where I touched?

(Also, here’s a tip: add the code tween.delay(0, function() and setup() end) around the code, it makes it wait one frame before it runs the code, and by doing so, lets it initiate the OpenGL frame first so the shaders work)

Ignatz · January 8, 2015, 7:15pm

Yes, so multiply by texture width and height for pixel coords

Re your tip - I think I’d rather define my shader as a string, because I often share my code, and that tweak would just confuse anyone who saw it. I was taught a long time ago - “don’t be clever, keep it simple”. (But thank you anyway, it’s interesting).

SkyTheCoder · January 8, 2015, 7:33pm

@Ignatz I see how getting texture coordinates would be useful, I might use that in a game.

Re the tip, I meant if you’re just running some older code from someone else and need it to work with the OpenGL frame in Codea 2.1, just add those lines, I find it easier. (I don’t mean to add those lines of code in all your projects and then share that code with others)

Ignatz · January 9, 2015, 4:02am

@LoopSpace - I’ve cut your code to the minimum I think will calculate the touch position for a rotated plane. This code will print the (x,y) position of a touch on the picture

https://gist.github.com/dermotbalson/286f386438d427d3c721

I’ve tried to make it as simple and intuitive as possible, so it is easy to use. I got rid of the functions that slid the planes based on touch, and handling of multiple objects of different types, and lighting. This meant classes weren’t needed, either.

As I have it now, users can handle the rotations as they normally would, and pass the position through, to store the matrix needed for touch handling.

The unit square approach does require two oddities, though - doing translation last because of the way you build the model matrix, and drawing the image as 1x1.

The real benefit is that now I have to add just two lines of code into a normal project, to trap touch positions on a plane. This is a very light footprint, making it really easy to include touch capabilities without disrupting the code that is already there.

LoopSpace · January 9, 2015, 2:50pm

@Ignatz Nice!

I still completely disagree about rotations (you keep referring to “Euler angles” as if they were a single thing, but in fact there are a number of varieties of Euler angles) but I guess that’s one we’ll have to agree to disagree about. Same with the location of the rectangle.

The inbuilt multiplication of a matrix by a vec4 doesn’t work (@Simeon - take note) so you still need my applymatrix4 function (indeed, the gist refers to it but doesn’t include it).

You can do the translation first. Your code is equivalent to:

    mm = mm:translate(p.x,p.y,p.z)
    mm = mm:scale(s.x,s.y,s.z)

I’m not sure that you’ve got the final position of the touch quite right. The picture is centred at the origin and is of height and length 1, so the final touch position should be of the form (x,y,0) with x and y in the range [-.5,.5]. But the values that I get are of the form (x,y,.5) with x and y in the range [0,1].

Here’s a modified version of that part:

function PictureIsTouchedBy(t,pos)
    local tc = screentoplane(t, vec3(0,0,0), vec3(1,0,0), vec3(0,1,0), Matrix)
    -- Was the picture touched?
    if math.abs(tc.x) > .5 or math.abs(tc.y) > .5 then return nil end
    -- The picture was touched, just need to decide our plane of movement
    local n = screenframe(t,Matrix)
    local plane
    if n.z*n.z > n.x*n.x + n.y*n.y then
    -- More "face on" so move in the plane containing the picture
        return tc
    else
    -- More "side on" so move in the plane with the line orthogonal to the picture 
    --and the line in the picture which is orthogonal to the viewer
        return screentoplane(t, vec3(0,0,0), vec3(0,0,1), n:cross(vec3(0,0,1)), Matrix)
    end 
end

(I also need to explore what that last part is doing. I’ve not figured that out yet.)

Ignatz · January 9, 2015, 5:21pm

Thank you for that. I would still love to understand these screen - plane conversions…

If the bit you haven’t figured out is my change at the end of PictureIsTouchedBy, I noticed you were calculating tc twice the same way if the default face applied, so I simply modified it so that it calculates tc just once, and then does the alternative calculation if the face is different. I haven’t modified any of your logic because I don’t understand it.

Ignatz · January 9, 2015, 5:40pm

Re Euler angles, I don’t want to get hung up on that. All I’m really trying to do is let the user determine for themselves how to do the rotations.

The approach I’m taking is that this code is a small add on utility to a user’s much larger project, therefore it shouldn’t dictate things like rotation methods, ie it shouldn’t say " you need to do A this way, and B that way". It should minimise its demands on the user code.

So I don’t care how users do rotations, as long as this code can deal with whatever it is. I mentioned Euler because most Codea users would use some form of it.