Making holes in meshes (eg see-through windows or doors in walls)

I thought this might interest some people (and, as always, maybe there’s a better way!)

http://instagram.com/p/Zy-WGgBHU4/

I wanted to make a building with windows and doors you could see through, ie I needed to make holes in the walls.

One way is to make a set of rectangle meshes that carefully leave gaps, but this requires you to figure out the rectangles yourself, for each new building, especially if you have more than one hole. I wanted to make it automatic.

Another solution is to use a fragment shader to test if the current pixel is in a hole area, and if so, discard the pixel. This does what I want (see picture), but the only problem is that OpenGL treats the holes as solid, and doesn’t draw anything behind them (once your wall is drawn). So you need to sort your meshes so you draw from furthest to nearest - something I do anyway.

My only concern is speed, but I’m still testing that.

My fragment shader is very simple, I just pass through a vec4 with start and end xy of the hole, and do a test

if (vTexCoord.x<hole1.x || vTexCoord.x>hole1.z || vTexCoord.y<hole1.y || vTexCoord.y>hole1.w) //then draw, else discard

Perhaps instead of discarding it you can make it transparent?

My understanding was the order matters if you were using transparency (alpha), but if you use discard you should be able to draw the meshes in any order.

@spacemonkey - yes, I’m sure you’re right, and my tests confirmed this, which is why I was discarding unwanted pixels.

However, in my project, it doesn’t seem to be the case. I think it’s because I’m putting a window frame image over the hole, and that image has transparent pixels, so OpenGL isn’t filling in anything behind them. I’m sorting the meshes anyway, so I can deal with that.

I thought I would add to this that I have used a simple shader to delete transparent pixels (where alpha is very small) in tree images, in the hope this would avoid the need to sort on z order (OpenGL treats transparent pixels as solid, so they block out images behind).

This is the transparent pixel problem if you don’t sort on z order
http://instagram.com/p/Z6rXV2BHbZ/

It almost works.
http://instagram.com/p/Z6rdN1hHbi/

The problem is OpenGl anti aliases the edges, blending them with whatever background is there when they are dawn. So many of my trees end up with an edge of sky pixels, even though another tree is drawn behind them later on. There doesn’t seem to be a way to turn this off.

I tried giving the images a solid background so OpenGl wouldn’t mix sky with the edge of the image, and then discarding that background in the shader. Nope. OpenGL blends the edge of the image with the background in that case, giving you an outline of whatever colour you chose for the background.

So I’m back to sorting on z order, to get the result I want

http://instagram.com/p/Z6rjkFBHbt/

Just one question, because I’m not 100% OpenGL does antialiasing by default. Are you sure your textures don’t have antialiased endges up front?

It’s possible, but when I filled the transparent pixels with a solid colour like red, and used the shader to discard that colour, I ended up with a blended red edge to all the images, so OpenGL is doing some blending of its own.

Yeah, but potentially the same thing, when you gave them solid background, did your image editing software antialias the solid background with the tree edge?

I will have to test that!

But something else I noticed is that when I draw trees in my 3D scene, they have blue outlines if they were against the sky when they were drawn, and green outlines if they were against other trees. You can see this in the 2nd pic I posted above. That’s not the paint program.

@spacemonkey

The test below shows OpenGL does anti alias images.

I create a white circle in a square, otherwise transparent image, and put it into 2 meshes which I draw next to each other, onto a red background.

However the second mesh has a shader that discards transparent pixels.

Finally I draw a yellow rectangle just behind both circle images.

The left hand circle retains an opaque red background because OpenGL thinks transparent cells are solid and doesn’t bother drawing the yellow rectangle behind them.

The right hand circle has no background any more, so the yellow rectangle is drawn behind it, but what is striking is the red edge on the circle, showing that as OpenGL drew the circle on the red background, it anti aliased the edge, and this edge remained when the yellow rectangle was drawn behind.

http://instagram.com/p/Z8HTMnBHQK/

Having seen what OpenGL does to solid images, I noticed that my tree images tend to have partly transparent edges, which just encourages OpenGL even more (if that is possible!) to blend them with the current background at time of drawing.

So I’m not sure there’s any alternative to z order sorting.

Code follows


--# Main
function setup()
    --make white ball on transparent background
    img=image(150,150)
    setContext(img)
    fill(255,255,255,255)
    ellipse(75,75,75)
    setContext()
    
    --create 2 identical meshes
    m1=mesh()
    m1.texture=img
    u=m1:addRect(0,0,img.width,img.height)
    m1:setRectTex(u,0,0,1,1)
    m1:setColors(color(255))
    
    --give the second one a shader that discards transparent pixels
    m2=mesh()
    m2.texture=img
    u=m2:addRect(0,0,img.width,img.height)
    m2:setRectTex(u,0,0,1,1)
    m2:setColors(color(255))
    m2.shader = shader(TransparentShader.vertexShader, TransparentShader.fragmentShader)
    
    --create a yellow image to go behind our shaded image to see if Codea anti aliased it
    img3=image(500,500)
    setContext(img3)
    fill(255, 255, 0)
    rect(1,1,500,500)
    setContext()
    m3=mesh()
    m3.texture=img3
    u=m3:addRect(0,0,img3.width,img3.height)
    m3:setRectTex(u,0,0,1,1)
    m3:setColors(color(255,255,0))
end

-- This function gets called once every frame
function draw()
    background(255,0,0)
   perspective()
   camera(0,20,200,0,0,0)
   pushMatrix()
   translate(-65,0,-120)
   m1:draw()
   popMatrix()
   pushMatrix()
   translate(65,0,-120)
   m2:draw()
   popMatrix()
   pushMatrix()
   translate(-65,0,-125)
   m3:draw()
   popMatrix()
end


--# Shader
TransparentShader = {
vertexShader = [[
//
// A basic vertex shader
//

//This is the current model * view * projection matrix
// Codea sets it automatically
uniform mat4 modelViewProjection;

//This is the current mesh vertex position, color and tex coord
// Set automatically
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;

//This is an output variable that will be passed to the fragment shader
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
    //Pass the mesh color to the fragment shader
    vColor = color;
    vTexCoord = texCoord;

    //Multiply the vertex position by our combined transform
    gl_Position = modelViewProjection * position;
}

]],
fragmentShader = [[
//
// A basic fragment shader
//

//Default precision qualifier
precision highp float;

//This represents the current texture on the mesh
uniform lowp sampler2D texture;

//The interpolated vertex color for this fragment
varying lowp vec4 vColor;

//The interpolated texture coordinate for this fragment
varying highp vec2 vTexCoord;

void main()
{
    lowp vec4 col = texture2D(texture, vTexCoord);
    if(col.a<0.05) discard;
    else gl_FragColor = col * vColor;
}

]]}

I think it’s still the texture, drawing your elipse in there gives fuzzy edges maybe. Anyway, play with the parameter in this, it’s your exact code, I just made the discard threshold tunable. Definitely the behaviour implies the antialiasing is in the texture lookup or the texture itself.



--# Main
function setup()
    --noSmooth()
    --make white ball on transparent background
    img=image(150,150)
    setContext(img)
    fill(255,255,255,255)
    ellipse(75,75,75)
    setContext()

    --img = readImage("Cargo Bot:Condition Blue")

    --create 2 identical meshes
    m1=mesh()
    m1.texture=img
    u=m1:addRect(0,0,img.width,img.height)
    m1:setRectTex(u,0,0,1,1)
    m1:setColors(color(255))

    --give the second one a shader that discards transparent pixels
    m2=mesh()
    m2.texture=img
    u=m2:addRect(0,0,img.width,img.height)
    m2:setRectTex(u,0,0,1,1)
    m2:setColors(color(255))
    m2.shader = shader(TransparentShader.vertexShader, TransparentShader.fragmentShader)

    --create a yellow image to go behind our shaded image to see if Codea anti aliased it
    img3=image(500,500)
    setContext(img3)
    fill(255, 255, 0)
    rect(1,1,500,500)
    setContext()
    m3=mesh()
    m3.texture=img3
    u=m3:addRect(0,0,img3.width,img3.height)
    m3:setRectTex(u,0,0,1,1)
    m3:setColors(color(255,255,0))
    
    parameter.number("threshold", 0.0, 1.0, 0.05)
end

-- This function gets called once every frame
function draw()
    --noSmooth()
    background(255,0,0)
   perspective()
   camera(0,20,200,0,0,0)
   pushMatrix()
   translate(-65,0,-120)
   m1:draw()
   popMatrix()
   pushMatrix()
   translate(65,0,-120)
   m2.shader.threshold = threshold
    m2:draw()
   popMatrix()
   pushMatrix()
   translate(-65,0,-125)
   m3:draw()
   popMatrix()
end


--# Shader
TransparentShader = {
vertexShader = [[
//
// A basic vertex shader
//

//This is the current model * view * projection matrix
// Codea sets it automatically
uniform mat4 modelViewProjection;

//This is the current mesh vertex position, color and tex coord
// Set automatically
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;

//This is an output variable that will be passed to the fragment shader
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
    //Pass the mesh color to the fragment shader
    vColor = color;
    vTexCoord = texCoord;

    //Multiply the vertex position by our combined transform
    gl_Position = modelViewProjection * position;
}

]],
fragmentShader = [[
//
// A basic fragment shader
//

//Default precision qualifier
precision highp float;

//This represents the current texture on the mesh
uniform lowp sampler2D texture;
uniform lowp float threshold;
//The interpolated vertex color for this fragment
varying lowp vec4 vColor;

//The interpolated texture coordinate for this fragment
varying highp vec2 vTexCoord;

void main()
{
    lowp vec4 col = texture2D(texture, vTexCoord);
    if(col.a<threshold) discard;
    else gl_FragColor = col * vColor;
}

]]}

Googling on this, I think it depends on how your texture has been bound to the mesh. (which is done by Codea and outside our control) When it’s bound it will either do interpolation in the lookups or not, and it seems it is doing that, hence the blurring.

Fiddling with your parameter didn’t seem to change anything for me

I separately found something interesting. If you load a tree image with fuzzy edges and simply make a copy of it with img:copy, then sprite the two images side by side, the copy has harder edges. It looks a bit quantum like - observing changes the system state.

Hmm, I assumed the example was showing the issue on the right hand circle with red fringes? pushing the threshold for the alpha test in the shader for discarding up to about 0.95 removed the fringes, putting it at 1.0 shrunk the circle a bit…

Yes, the issue was the red fringe.

I would expect that a high alpha value would get rid of the fringe, but I can’t set it that high in practice or I won’t have any tree left!

I don’t think there’s any reliable solution except z order sorting. And that seems to be confirmed by what I see on internet forums.

Yep, the issue is the textures are bound by Codea with Bilinear filtering, so it mixes the nearest 4 texture points for each pixel, hence the edge blur. :frowning:

aha, thanks for that

while we’re talking, I am building a shared library of interesting/useful shaders. I was wondering you if have any interesting shaders to share?

Sure, all my shaders are on the forum in one place or another, some are too complex and or just curiosities, but the useful ones (in my opinion) are:

Particle effect: http://twolivesleft.com/Codea/Talk/discussion/2516/particle-effect-in-a-shader

Sprite Sheet Shader: http://twolivesleft.com/Codea/Talk/discussion/2857/animation-with-png#Item_8

Lighting shader: http://twolivesleft.com/Codea/Talk/discussion/2379/ads-lighting-class#Item_1

And the tiling texture one you already have.

Thanks!

I’m VERY interested in the lighting class and dying to try it with my 3D tabletop, I just have to get to understand it and modify it to include the tiling texture shader as well

(How on earth did you learn all this stuff?)

I had never seen a shader before until Codea 1.5 came out, then everything I learnt from google. But they are not that difficult, it’s just a case of tinkering away and seeing where you get to. I also bought a book http://www.amazon.co.uk/gp/product/0321502795/ref=oh_details_o03_s00_i00?ie=UTF8&psc=1 which was quite helpful, even though a fair bit of it is the openGL end that Codea does for you.

The lighting one is probably a bit heavy as it stands, and will be too slow for your scene I think. I found a really interesting article about speeding up lighting shaders, but I haven’t had time to digest it yet.