Camera and shaders

I’m new to shaders (using @Ignatz ever-excellent articles as a jump off point) and am trying to do a simple difference between two images. Here is the code, but I get an odd effect where the top image seems to switch between the current and an older (stale) image. I suspect this may be down to the CAMERA source rather than the shader but any pointers would be appreciated (including if you can replicate the issue). There also appears to be issues with initializing the camera but think I’ve got a workaround included.

To use tap on the screen to update one of the two pictures. The picture which updates switches between the top and bottom with each tap.

-- Camera Compare

-- Use this function to perform your initial setup
function setup()
    cameraSource(CAMERA_FRONT)
    --without the following line the program fails
    snap=image(CAMERA)
    frame={}
    frame[1]=image("Cargo Bot:Codea Icon") --dummy initial value - if you set to CAMERA you get an error
    frame[2]=image("Cargo Bot:Clear Button")
    
    f=2
    m=mesh()
    m.shader=shader(DiffShader.vertexShader, DiffShader.fragmentShader)
    --allow user to set threshold level
    m.shader.thresh=0.2
    m.shader.texture=frame[1] --forehround
    m.shader.texture2=frame[2]--background
    u=m:addRect(3*WIDTH/4,HEIGHT/2,WIDTH/4,HEIGHT/4)
    m:setRectTex(u,0,0,1,1)
end

-- This function gets called once every frame
function draw()
    
    -- This sets a dark background color
    background(44, 44, 117, 255)
    sprite(frame[1],WIDTH/4,3*HEIGHT/4,WIDTH/4, HEIGHT/4)
    sprite(frame[2],WIDTH/4,2*HEIGHT/4,WIDTH/4, HEIGHT/4)
    m.shader.texture=frame[1]
    m.shader.texture2=frame[2]
    m:draw()
end

function touched(t)
    if t.state==ENDED then
        takephoto(f)
        f=f+1
        if f>2 then f=1 end
    end
end

function takephoto(fin)
    frame[fin]=image(CAMERA)
end

--Difference shader - compare the rgb elements of two images and display the differences
DiffShader = {
vertexShader = [[
//
// A basic vertex shader
//

//This is the current model * view * projection matrix
// Codea sets it automatically
uniform mat4 modelViewProjection;

//This is the current mesh vertex position, color and tex coord
// Set automatically
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;

//This is an output variable that will be passed to the fragment shader
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
//Pass the mesh color to the fragment shader
vColor = color;
vTexCoord = texCoord;

//Multiply the vertex position by our combined transform
gl_Position = modelViewProjection * position;
}


]],
fragmentShader = [[
//
// A basic fragment shader
//

//Default precision qualifier
precision highp float;

//This represents the current texture on the mesh
uniform lowp sampler2D texture;
uniform lowp sampler2D texture2;

//The interpolated vertex color for this fragment
varying lowp vec4 vColor;

//The interpolated texture coordinate for this fragment
varying highp vec2 vTexCoord;

uniform lowp float thresh;

void main()
{
//Sample the texture at the interpolated coordinate

lowp vec4 col1 = texture2D( texture, vTexCoord );
lowp vec4 col2 = texture2D( texture2, vTexCoord );

if (abs(col2.r-col1.r)<thresh && abs(col2.g-col1.g)<thresh && abs(col2.b-col1.b)<thresh) discard; else gl_FragColor = col2;
}
]]}

For me, the top left pic flickers between two images.

After making a series of experimental changes, I’ve established that it flickers if you use a table of stock images (ie not due to the camera), and also if you change the pic using a timer instead of a touch, and also if you simplify the shader so it only returns the first pic given to it. If you use meshes instead of sprites it makes no difference

If you don’t draw image2 (bottom left) it doesn’t flicker, but I don’t think image1 updates properly

If you don’t use a shader, it works properly.

Very peculiar. :s

Ah, it is the camera. This line is enough to cause all the problems

snap=image(CAMERA)

The flickering is what I get. Both elements of the flickering image are sent into the shader.

If you remove that line then I get an error in line 30 - bad argument #1 to sprite - codeaimage expected, got nil

I wondered if it was something to do with using the same source as both a texture and a sprite and with something under the hood associated with CAMERA. In the documentation the example is:

img=image(CAMERA)
sprite(CAMERA,x,y)

local m=mesh()
m.texture=CAMERA

The first takes a snapshot, the second a live video feed. If you feed the object stored in the table into both instances (as I’m doing), does this cause the error?

Finally, on a related note with a camera feed and a shader. Do you know when to use m.texture and m.shader.texture? I wondered if this was a contributing factor.

Many thanks for helping out.

PS: I also wondered if it was something to do with frame buffering. Back in the days of programming on an AMIGA you would complete all the screen updates on a hidden image and only once complete would you swap it into view then start the next update on a new image, to prevent flickering.

The image command captures a static photo, so it should be no different to opening a picture stored on disk. I’ve not had trouble working with the camera before.

I realise you need a dummy camera image command in setup to make it work, but what I mean when I said this caused the problem, was that if you start by loading in a set of pictures from one of the built in image libraries, and use this set instead of the camera (ie rotate through this set rather than take camera pics), you still get flickering as long as that camera line is there in setup AND the shader image is drawn. In other words, that single line, without any other reference to camera images, is enough to cause the problem. If you remove the camera image line in setup, OR you don’t draw the shader image, the flickering stops.

I don’t think it’s frame buffering because if it were, the flickering would occur only for one frame when the images change, but it is continuous.

Go figure.

Wrt texture, m.texture is shorthand for m.shader.texture, so that’s not a problem.

Thanks for the additional info, especially on the m.texture call.

A bit of digging, it appears the term I was looking for is double or triple buffering

https://en.m.wikipedia.org/wiki/Multiple_buffering#Double_buffering_in_computer_graphics

It just feels like this is going - if you pause the program only one of the images in the flicker draws - it almost seems as though the camera is dumping out its live stream, but only to a single buffer. Doesn’t explain why it doesn’t occur in the second image though. @Simeon any ideas - this seems like a bug

No, when I substitute a table of library images, ie I don’t use camera images at all - except for the dummy call in setup - the image flickers between two of the table images.

To allow the devs to see if there is a problem, we need to strip this program to the bare essentials, as I’ve done below.

@Simeon - The program below uses a shader to compare two images, and draw a third image which only contains pixels which are different, ie it shows the change from one image to the next. It is intended for use with the camera, to track movement.

However, there is a flicker problem, as demonstrated in this test rig - which has been cut to the minimum to make diagnosis easier. The test uses library images, whereas in the full version, camera images would be used, but this test shows the flickering occurs even if library images are used.

To begin with, the iPad camera is initialised with an image statement (which seems to be necessary if you are going to use camera images). Although our test program doesn’t use camera images, this line has been left in, because it helps to cause the flickering. (We will need it in the original program, so it can work with camera images).

The program reads in two library images and puts them in a table pix, and also into two meshes m1 and m2. These are simply drawn on the screen at left, without alteration - but they flicker.

A third mesh m has a shader attached, which will display the changed pixels to the right of screen. The shader accepts the two images as uniforms, along with a toggle which can disable the comparison of the two images and simply returns the first image. This allows you to dynamically test the effect of using a null shader (ie that does nothing).

If you run the program below, you should see that one of the left hand images flickers between the two images, even though it has clearly been set to show just one of them. The shader image at right correctly shows the pixels which are different between the two images.

So what is causing this? We can isolate it.

There are two parameter toggles. The first disables the shader, so the image at right simply displays one of the pictures. This stops the flickering.

The second toggle “nullShader”, tells the shader to simply return the first texture given to it, without doing any comparison, ie a null shader. This doesn’t stop the flickering.

If you comment out the camera image line in setup, this will also stop the flickering.

So the combination of a camera image command, used just once, and a very simple shader, cause flickering in unrelated image meshes. Why?

function setup()
    
    parameter.boolean("UseShader",true)
    parameter.boolean("NullShader",false)
    
    --commenting out this line stops the flickering
    snap=image(CAMERA)
   
    pix={readImage("Planet Cute:Character Princess Girl"),readImage("Planet Cute:Character Pink Girl")}
    
    --set up two meshes at left of screen to hold pictures
    m1=mesh() m1:addRect(WIDTH/4,3*HEIGHT/4,WIDTH/4, HEIGHT/4)
    m1.texture=pix[1] 
    m2=mesh() m2:addRect(WIDTH/4,2*HEIGHT/4,WIDTH/4, HEIGHT/4)
    m2.texture=pix[2]
    
    --this mesh at right of screen will have a shader
    --that shows the difference between the two pics at left
    m=mesh()
    m:addRect(3*WIDTH/4,HEIGHT/2,WIDTH/4,HEIGHT/4)
    m.texture=pix[1]
end

function draw()
    --change image in the frame table alternately, every 2 seconds 
    if UseShader then 
        m.shader=shader(DiffShader.vertexShader, DiffShader.fragmentShader)
        m.shader.texture2=pix[2]
        m.shader.nullShader=NullShader --toggles between a comparative shader and one that does nothing
    else m.shader=nil
    end       
    
    background(44, 44, 117, 255)       
    m1:draw()    
    m2:draw()    
    m:draw()
end

--Difference shader - compare the rgb elements of two images and display the differences
DiffShader = {
vertexShader = [[
uniform mat4 modelViewProjection;
attribute vec4 position;
attribute vec2 texCoord;
varying highp vec2 vTexCoord;

void main()
{
vTexCoord = texCoord;
gl_Position = modelViewProjection * position;
}

]],
fragmentShader = [[
precision highp float;
uniform lowp sampler2D texture;
uniform lowp sampler2D texture2;
uniform bool nullShader;
varying highp vec2 vTexCoord;

void main()
{

lowp vec4 col1 = texture2D( texture,  vTexCoord );
lowp vec4 col2 = texture2D( texture2, vTexCoord );
if (nullShader) gl_FragColor = col1;
else {
    if (abs(col2.r-col1.r)<0.1 && abs(col2.g-col1.g)<0.1 && abs(col2.b-col1.b)<0.1) discard; 
    else gl_FragColor = col1;
    }
}
]]}

@Ignatz - thanks for the clarity in narrowing down the issue

Let’s hope our dev wizards can figure it out

@John @Simeon - have you had a chance to look into this? Also are there plans to open up the manual camera configurations in codea?

http://www.imore.com/camera-api-ios-8-explained

I made these changes and I don’t get the flickering and it appears to work correctly. I have the camera being used. A lot of the changes were formatting just to make it easier for me to read. I don’t know shaders that well and I’m not sure what I changed to make it work.

EDIT: One thing I noticed in both versions is the white cross in the pink hair doesn’t show in the diff image.

function setup()
    pix={readImage("Planet Cute:Character Princess Girl"),
            readImage("Planet Cute:Character Pink Girl")}
    snap=image(CAMERA)
    m=mesh()
    m:addRect(3*WIDTH/4,HEIGHT/2,WIDTH/4,HEIGHT/4)
    m.texture=pix[1]
    m1=mesh() 
    m1:addRect(WIDTH/4,3*HEIGHT/4,WIDTH/4, HEIGHT/4)
    m1.texture=pix[1] 
    m2=mesh() 
    m2:addRect(WIDTH/4,2*HEIGHT/4,WIDTH/4, HEIGHT/4)
    m2.texture=pix[2]
end

function draw()
    background(44, 44, 117, 255)       
    m.shader=shader(DiffShader.vertexShader, DiffShader.fragmentShader)
    m.shader.texture2=pix[2]
    m:draw()
    m1:draw()    
    m2:draw()    
end

DiffShader = {
vertexShader = [[
    uniform mat4 modelViewProjection;
    attribute vec4 position;
    attribute vec2 texCoord;
    varying highp vec2 vTexCoord;
    void main()
    {   vTexCoord = texCoord;
        gl_Position = modelViewProjection * position;
    }    
]],
fragmentShader = [[
    precision highp float;
    uniform lowp sampler2D texture;
    uniform lowp sampler2D texture2;
    uniform bool nullShader;
    varying highp vec2 vTexCoord;    
    void main()
    {   lowp vec4 col1 = texture2D( texture,  vTexCoord );
        lowp vec4 col2 = texture2D( texture2, vTexCoord );
        if (abs(col2.r-col1.r)<0.1 && 
                abs(col2.g-col1.g)<0.1 && 
                abs(col2.b-col1.b)<0.1) 
            discard; 
        else 
            gl_FragColor = col1;
    }
]]}

This version seems to work a lot better. I changed gl_FragColor=col1 to be gl_FragColor=abs(col1-col2). This now shows the cross in the hair.

function setup()
    pix={readImage("Planet Cute:Character Princess Girl"),
            readImage("Planet Cute:Character Pink Girl")}
    snap=image(CAMERA)
    m=mesh()
    m:addRect(3*WIDTH/4,HEIGHT/2,WIDTH/4,HEIGHT/4)
    m.texture=pix[1]
    m1=mesh() 
    m1:addRect(WIDTH/4,3*HEIGHT/4,WIDTH/4, HEIGHT/4)
    m1.texture=pix[1] 
    m2=mesh() 
    m2:addRect(WIDTH/4,2*HEIGHT/4,WIDTH/4, HEIGHT/4)
    m2.texture=pix[2]
end

function draw()
    background(44, 44, 117, 255)       
    m.shader=shader(DiffShader.vertexShader, DiffShader.fragmentShader)
    m.shader.texture2=pix[2]
    m:draw()
    m1:draw()    
    m2:draw()    
end

DiffShader = {
vertexShader = [[
    uniform mat4 modelViewProjection;
    attribute vec4 position;
    attribute vec2 texCoord;
    varying highp vec2 vTexCoord;
    void main()
    {   vTexCoord = texCoord;
        gl_Position = modelViewProjection * position;
    }    
]],
fragmentShader = [[
    precision highp float;
    uniform lowp sampler2D texture;
    uniform lowp sampler2D texture2;
    uniform bool nullShader;
    varying highp vec2 vTexCoord;    
    void main()
    {   lowp vec4 col1 = texture2D( texture,  vTexCoord );
        lowp vec4 col2 = texture2D( texture2, vTexCoord );
        if (abs(col2.r-col1.r)<0.1 && 
                abs(col2.g-col1.g)<0.1 && 
                abs(col2.b-col1.b)<0.1) 
            discard; 
        else 
            gl_FragColor = abs(col1-col2);
    }
]]}

Come on dave, surely you noticed that you swapped the draw sequence. :wink:

Drawing the shader image first removes the flickering. I wonder why?

@Ignatz Like I said, I don’t know shaders that well. What I meant above was, other than changing thing to make it easier for me to read, I don’t know what changed to make it work. I know I changed the order of drawing the meshes, but I don’t see why that mattered as you also wondered. The drawing order matters if objects overlap, but these didn’t.

Fantastic- thanks for the workaround @dave1707 ! Much appreciated