Squeezy sponge - deforming a cube in the shader

I’ve been experimenting with deforming shapes in the shader.

sponge3

sponge1

Drag on the left and right sides of the screen to stretch and twist the left and right sides of this cube. It’s like squeezing out a sponge. It’s done by uploading two modelMatrices to the shader, one for each side of the sponge, and then interpolating between the positions and normals generated by the matrices using mix and smoothstep.

sponge2

Edit: sponginess adjusted.

--# Main
-- Cube Deform

local tArray = {{x=0,y=0, mm = matrix(), id = 0},{x=0,y=0, mm = matrix(),id=0}} --coords and matrices for 2 bones/ touch points

function setup()
    m = mesh() 
    m.texture = readImage("Cargo Bot:Game Area")
    m:setColors(color(255))
    m.vertices, m.normals, m.colors, m.texCoords = cube(5, 4) 
    m.shader = shader(Deform3.vert, Deform3.frag)
    m.shader.ambient = 0.4
    m.shader.lightColor = color(223, 216, 145, 255)
    m.shader.light = vec4(10,5,15,1)
    cam = {pos = vec3(5,-10,10)}
    centre = vec2(WIDTH, HEIGHT)/2
    print("Drag on the left and right sides of the screen to squeeze out the sponge!")

end

function draw() 
    background(40, 40, 50)
    perspective()
    camera(cam.pos.x, cam.pos.y, cam.pos.z,0,0,0)
    
    for i,v in ipairs(tArray) do
        if v.id == 0 then
            v.x = v.x *  0.97 --damping/ return sponge to form
            v.y = v.y *  0.97
        end
        pushMatrix()
        translate(v.x*0.02,v.y*0.02,0)
        rotate(v.x, 0,1,0)
        rotate(-v.y, 1,0,0)
        v.mm = modelMatrix()
        popMatrix()
    end
    
    m.shader.modelMatrix1 = tArray[1].mm
    m.shader.modelMatrix2 = tArray[2].mm --load matrices into shader
    --nb draw at the identity matrix
    m.shader.eye = cam.pos
    m:draw()
end

local sc = 0.02
function touched(t)
    if t.state == BEGAN then 
        if t.x<centre.x then --bone 1 on the left
            tArray[1].id = t.id
        else
           tArray[2].id = t.id -- bone 2 on the right       
        end
    else
        for i,v in ipairs(tArray) do
            if t.id == v.id then  --find out which side this touch originated on
                v.y = v.y + t.deltaY * 0.3
                v.x = v.x + t.deltaX * 0.3
                if t.state == ENDED then v.id = 0 end 
                --[[
                if i==1 then --only right-side touches influence lighting
                    m.shader.light = vec4((t.x - centre.x)*sc, (t.y-centre.y)*sc, 7, 1)
                end
                  ]]
            end
        end      
    end
    
end

Deform3={
vert = [[
const float size = 2.5;
uniform mat4 modelViewProjection;
uniform mat4 modelMatrix1; //bone1
uniform mat4 modelMatrix2; //bone2

attribute vec4 position;
attribute vec4 color;
attribute vec3 normal;
attribute vec2 texCoord;

varying lowp vec4 vNormal;
varying lowp vec4 vPosition;
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
    float bl = smoothstep(-size, size, position.x); 
    mat4 mm = modelMatrix1 * (1.-bl) + modelMatrix2 * bl;
    vPosition = mm * position; 
    vNormal = normalize(mm * vec4( normal, 0.0 )); 

    vColor = color;
    vTexCoord = texCoord;
    gl_Position = modelViewProjection * vPosition; 
}

]],

frag = [[

precision highp float;

uniform lowp sampler2D texture;
uniform float ambient; // --strength of ambient light 0-1
uniform vec4 light; //--directional light direction (x,y,z,0)
uniform vec4 lightColor; //--directional light colour
uniform vec4 eye; // -- position of camera (x,y,z,1)

varying lowp vec4 vNormal;
varying lowp vec4 vPosition;
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
    lowp vec4 pixel= texture2D( texture, vec2( fract(vTexCoord.x), fract(vTexCoord.y) ) ) * vColor; 
    lowp vec4 ambientLight = pixel * ambient;
    lowp vec4 norm = normalize(vNormal);
    lowp vec4 lightDirection = normalize(light - vPosition * light.w);
    lowp vec4 diffuse = pixel * lightColor * max( 0.0, dot( norm, lightDirection ));

    //specular blinn-phong
    vec4 cameraDirection = normalize( eye - vPosition );
    vec4 halfAngle = normalize( cameraDirection + lightDirection );
    float spec = pow( max( 0.0, dot( norm, halfAngle)), 8. );//last number is specularPower, higher number = smaller highlight
    lowp vec4 specular = lightColor  * spec * 64. ;// add optional shininess at end here

    vec4 totalColor = ambientLight + diffuse + specular;
    totalColor.a=vColor.a;
    gl_FragColor=totalColor;
}

]]}

--# Cube
function cube(s, depth)
    -- all the unique vertices that make up a cube
    local vertices = {
    vec3(-0.5, -0.5,  0.5), -- Left  bottom front
    vec3( 0.5, -0.5,  0.5), -- Right bottom front
    vec3( 0.5,  0.5,  0.5), -- Right top    front
    vec3(-0.5,  0.5,  0.5), -- Left  top    front
    vec3(-0.5, -0.5, -0.5), -- Left  bottom back
    vec3( 0.5, -0.5, -0.5), -- Right bottom back
    vec3( 0.5,  0.5, -0.5), -- Right top    back
    vec3(-0.5,  0.5, -0.5), -- Left  top    back
    }
    
    -- 6 quadiralateral faces (anticlockwise)
    local faces = {
    -- Front
    {vertices[1], vertices[2], vertices[3], vertices[4]},
    -- Right
    {vertices[2], vertices[6], vertices[7], vertices[3]},
    -- Back
    {vertices[6], vertices[5], vertices[8], vertices[7]},
    -- Left
    {vertices[5], vertices[1], vertices[4], vertices[8]},
    -- Top
    {vertices[4], vertices[3], vertices[7], vertices[8]},
    -- Bottom
    {vertices[5], vertices[6], vertices[2], vertices[1]},
    }
    
    --recursive subdivide function
    local origin = vertices[5]
    local tOri = vec2(0.5,0.5)
    local verts, norms, cols, texC = {}, {}, {}, {}
    local rnd = math.random
    
    local function subdivide(f, depth, n, c, t)
        if depth == 0 then --triangulate
            local tc = {vec2(f[1][t.x], f[1][t.y])+tOri, vec2(f[2][t.x], f[2][t.y])+tOri,vec2(f[3][t.x], f[3][t.y])+tOri,vec2(f[4][t.x], f[4][t.y])+tOri}
           -- local c = color(rnd(128)+128, rnd(128)+128, rnd(128)+128)
            table.move({f[1]*s,f[2]*s,f[3]*s, f[1]*s,f[3]*s,f[4]*s}, 1, 6, #verts+1, verts) --triangulate
            table.move({tc[1],tc[2],tc[3], tc[1],tc[3],tc[4]}, 1, 6, #texC+1, texC)
            table.move({n,n,n,n,n,n},  1, 6, #norms+1, norms)
            table.move({c,c,c,c,c,c},  1, 6, #cols+1, cols)        
            return --end recursion
        end
        --else, do some more subdividing
        local mid = {}
        for i = 1, 4 do --4 mid points
            local j = (i % 4) + 1
            mid[i] = (f[i] + f[j]) * 0.5
        end
        local cent = (f[1] + f[2] + f[3] + f[4]) * 0.25 --centre point
        for i = 1, 4 do --4 sub-faces
            local j = (i % 4) + 1
            subdivide ({f[j], mid[j], cent, mid[i]}, depth - 1, n, c, t)        
        end    
    end
    --start subdivision

    for i = 1,6 do --6 faces. 
        --establish norm, color, and axis of texCoords for each face
        local norm = (faces[i][2]-faces[i][1]):cross(faces[i][3]-faces[i][1])
        local col = color(rnd(100)+155, rnd(100)+155, rnd(100)+155) --color(rnd(255), rnd(255), rnd(255))
        local plane = (faces[i][3]-origin)-(faces[i][1]-origin) --measure diagonal
        local tc = {x="x",y="y"}
        if plane.x==0 then tc.x="z" --if no x dimension...
        elseif plane.y==0 then tc.y="z" --or no y dimension, then remap texCoord onto Z
        end
        subdivide(faces[i], depth, norm, col, tc)
    end
    
    print ("recursions:", depth, "verts:", #verts) -- = 4 corners ^ depth * 6 faces * 6 tri points
    return verts, norms, cols, texC
end

Step 1 - squeezy sponge

Step 2 - Spongebob :))

it’s quite relaxing to play with. I wonder if you could game-ify it… A very odd variation on Geometry Wars maybe??

This is what I love about Codea, just that combo of incredible graphics power plus multitouch… An idea pops into your head, and you never quite know what’s going to pop out a few hours later.

Btw, how do you roll your own version of modelViewProjection in the vertex shader? Or rather, I want just viewProjection.

You can see in the code above, after deforming the position I have to convert it back to local coords (by multiplying by the inverse of the model matrix) to multiply it by modelViewProjection. The back conversion step would be unnecessary if I could create a viewProjection matrix. This nearly worked:

projectionMatrix * viewMatrix * modelMatrix * position but the perspective was weird, as if the object was getting bigger, rather than receding with distance, and the object was dark, as if the normals were all facing the wrong way. The projection, view, and model matrices were all just loaded in from Codea. I tried making various ones inverse, transpose, inverse():transpose() etc, but I couldn’t get a combination that worked. Anyone know how to do this?

wow! you’ve become our current locomotive! Thanks for that.

You’re welcome. It’s when there’s something else really important that I should be doing that I most often end up procrastinating!

wonderful, sponge cube

@yojimbo2000 - you know, if you forget that inverse matrix stuff and just write

gl_Position = modelViewProjection * vPosition;

you get very similar results.

Looking at the Codea draw code, and specicially, only the code that affects modelMatrix…

function draw() 
    <snip>
    for i,v in ipairs(tArray) do  --two passes
        <snip>
        resetMatrix() --create 2 matrices
        translate(v.x*0.02,v.y*0.02,0)
        rotate(v.x, 0,1,0)
        rotate(-v.y, 1,0,0)
        v.mm = modelMatrix()
    end
    <snip>
    m:draw()
end

So the modelMatrix that is used by the shader is the modelMatrix created by the second pass through the loop. Is this intended?

Why not reset the matrix after the second pass, then modelMatrix is just the unit matrix and there is no need for any inverse transformation?

Really impressive. Reminds me of an accordion - might be nice to use the corner positions to modify a sound - a sort of squeezy theremin

@Ignatz this is actually a proof of concept of doing bone deformation/ matrix interpolation on the shader. So the two sides of the cube are the bones, and the vertex weight/ skin weight is set by feeding local position.x into the smoothstep curve.

The idea is that modelMatrix / mmInverse represents the “root”/ parent bone in world space, and modelMatrix2 is a child-bone in local space. I admit that it’s confusing!

if you forget that inverse matrix stuff and just write gl_Position = modelViewProjection * vPosition; you get very similar results.

That doesn’t work because whatever transformation is applied to the left side of the cube gets doubled.

But your suggestion about resetting the matrix is a really interesting one. I didn’t think of this, but I guess what you’re saying is that one way of getting just the viewProjection matrix that I was asking for, is to make sure the modelMatrix is the identity matrix when you draw. So then the two matrices for the bones would both need to be in world space. That’s probably an easier way of doing it. I’ll post some updated code in a bit.

In the shader, I also came up with an alternate way of interpolating between the two matrices directly (instead of producing two position vectors and two normal vectors and interpolating between those vectors):

mat4 mm = modelMatrix * (1.-bl) + modelMatrix2 * bl;

You then calculate a single position and normal with the interpolated matrix.

@West all mods are welcome! It’s like a Rorschach. I saw a sponge, @Ignatz saw SpongeBob, you saw an accordion.

OK, I’ve changed the code above to incorporate @Ignatz 's suggestion to draw at the identity matrix and with the 2 bone matrices now both in world space. I think it’s easier to understand now.

Very nice…