GLES 3.0, instancing, and other changes to shaders in Codea 2.3.2

General
1

Until now Codea’s shaders have used GLES 2.0

Starting with 2.3.2 beta build 56, optional support for GLES 3.0 is added.

This is only available on devices with an A7 chip or newer (iPad Air onwards).

(and is currently only available to Codea beta testers).

GLES 3.0 is optional, and can be added on a shader-by-shader basis (ie you can mix 2.0 and 3.0 in the same program) by placing #version 300 es in the string of the vertex and fragment shaders, so you should be able to add it gradually, if it’s needed for your code.

This Apple page is quite a handy cheat sheet to the changes you need to make to the shader language : https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/AdoptingOpenGLES3/AdoptingOpenGLES3.html#//apple_ref/doc/uid/TP40008793-CH504-SW18

And regardless of which GLES version you use, you can never read this page enough times:

https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/BestPracticesforShaders/BestPracticesforShaders.html

The most useful resource is probably the quick reference card (pages 4 to 6 is the relevant section):

https://www.khronos.org/files/opengles3-quick-reference-card.pdf

And this is the bible:

https://www.khronos.org/registry/gles/specs/3.0/GLSL_ES_Specification_3.00.4.pdf

Here’s some very simple sample code:

-- test of GLES 300 Matrix Mesh Buffer Test
--

function setup()
    m = mesh()
    m:addRect(0,0,200,200)
    m:setColors(color(255,200,0))
    m.shader = shader(vert, frag)

    dummy = m:buffer("dummy") --test vec4 attribute
    transform = m:buffer("transform") --test mat4 attribute
    local mat = matrix():translate(0,100,0) --should move rectangle up
    for i=1,m.size do
        dummy[i] = vec4(60,0,0,0) --works, moves rectangle to right
        transform[i] = mat
    end

end

function draw()
    background(40, 40, 50)
    perspective()
    camera(0,0,400, 0,0,0)

    m:draw()
end

vert=[[
#version 300 es

uniform mat4 modelViewProjection;

in mat4 transform;
in vec4 position;
in vec4 color;
in vec4 dummy;

out lowp vec4 vColor;

void main()
{
    vColor = color;
    gl_Position = modelViewProjection * (transform * position + dummy);
}
]]

frag=[[
#version 300 es

in lowp vec4 vColor; //NB varying becomes "in" in fragment shader
out lowp vec4 fragColor;

void main()
{
    fragColor = vColor;
}
]]
2

One issue to be aware of:

in 3.0, the old texture2D and texture3D commands have been replaced with a single texture command. The problem is that this clashes with the uniform texture variable that the Codea Mesh API passes to the shader.

The workaround is to define your own custom texture variable eg:

frag = [[
#version 300 es

uniform lowp sampler2D textureMap; //custom texture variable
in highp vec2 vTexCoord;
out lowp vec4 fragColor;

void main()
{
  fragColor = texture(textureMap, vTexCoord); //the new texture command
}
]]

Then, in the Codea code, replace

myMesh.texture = "Dropbox:grass"

with:

myMesh.shader.textureMap = "Dropbox:grass"

(note the .shader).

3

How to convert a 2.0 shader to 3.0:

  1. In the vertex shader:

    • add #version 300 es to start
    • change attribute to in
    • change varying to out

  2. In the fragment shader:

    • add #version 300 es to start
    • change varying to in (nb different from the vertex shader, you can’t just do a global find/replace on varying)
    • there is no built-in gl_FragColor variable anymore. Instead, you define any variable as an out eg out lowp vec4 fragColor;, out lowp vec4 fragmentColor; etc
    • the texture2D and texture3D functions have been replaced with a single texture function. Note that this clashes with the texture uniform passed by the Codea mesh API, so you will have to use a custom variable, eg uniform lowp sampler2D textureMap; which you access from Codea with myMesh.shader.textureMap instead of myMesh.texture (see post immediately above this one).
4

Also, even if you don’t use GLES 3.0, the GLES 2.0 shader specifications are now more fully supported by the Codea 2.3.2 mesh API, particularly with regard to matrices:

  • You can now have an array of uniform mat4s eg: uniform mat4 modelMatrices[20];
  • Attributes/buffers can now be matrices: eg attribute mat4 modelMatrix;
5

added a link to the first post to the quick reference card

(love the Khronos quick reference cards. Everything should have a quick reference card):

https://www.khronos.org/files/opengles3-quick-reference-card.pdf

Start your engines! B-)

6

Initial thoughts:

there’s obviously an enormous amount to test here, and many features may require changes to the Codea API if they’re going to be accessible.

Things that GLES 3.0 supports that I’m going to try when I have a spare minute:

textures on the vertex shader!

Could be a great way of creating various vertex displacement effects, displacement mapping, height mapping with a texture, very cheap noise effects ( compared to the relatively expensive ashima arts pnoise/ snoise functions), providing data for instanced drawing, building a voxel engine etc

7

looking forward to trying this soon, thanks for your efforts, much appreciated :bz

8

Ok, first test of a unique-to-GLES 3.0 feature: textures on the vertex shader! (Here being used to displace vertices).

It works!

But, when you animate it, it’s jerky. Haven’t worked out quite why that is yet. I thought it was to do with the resolution of the texture, but when I increased it, it stayed jittery. Anyone have any ideas how to make the animation smoother?

(Take two texture samples and interpolate between them somehow?)

-- test of GLES 300 
-- textures on the vertex shader for vertex displacement
--

function setup()
   spriteMode(CORNER)

    -- create a mesh of 10x10 quads. nb using addRect for a "Z-Up" world
    m = mesh()
    local w,h = 20,20
    for x=1,10 do
        for y=1,10 do
            local r = m:addRect((x-5.5)*w,(y-5.5)*h,w,h)
            m:setRectTex(r, (x-1)*0.1,(y-1)*0.1,0.1,0.1)
            m:setRectColor(r, math.random(255), math.random(255), math.random(255))
        end
    end
  
    -- create a noise image that will displace these vertices
    img = image(256,256)
    local fine = 10
    local seed = math.random() * 5000
    local seed2 = math.random() * 3000
    for x=1,img.width do
        for y = 1,img.height do
            local n1 = noise(x/fine, y/fine, seed)+1
            local n2 = noise(x/fine, y/fine, seed2)+1
            img:set(x,y, math.ceil(n1 * 20),math.ceil(n2*60),math.ceil( n1*128) )
            --Bigger value in blue because we're displacing more along Z axis (ie up)
        end
    end

    --set up the shader
    m.shader = shader(vert, frag)
    m.shader.heightMap = img
    offset = vec2(0,0)
    rot = 0
end

function draw()
    background(40, 40, 50)
    sprite(img, 0,0) --show a preview of the displacement map in the corner
    perspective()
    camera(0,-200,180, 0,0,30, 0,0,1) --Z up

    rot = (rot + 0.2)%360
    rotate(rot)
    
    offset = offset + vec2(0.0002, -0.0002) --why does this produce jerky motion?
    m.shader.offset = offset
    m:draw()
end

vert=[[
#version 300 es

uniform mat4 modelViewProjection;
uniform sampler2D heightMap; //texture in vert shader! 3.0 only
uniform highp vec2 offset;

in highp vec2 texCoord;
in highp vec4 position;
in lowp vec4 color;

out lowp vec4 vColor;

void main()
{
    vColor = color;
    vec3 displace = texture(heightMap, fract(texCoord + offset )).rgb;
    gl_Position = modelViewProjection * (position + vec4(displace * 100., 0.));
}
]]

frag=[[
#version 300 es

in lowp vec4 vColor; //NB varying becomes "in" in fragment shader
out lowp vec4 fragmentColor; 

void main()
{
    fragmentColor = vColor;
}
]]
9

Instancing is coming! While we wait, here’s an interesting article with Android code examples and fps comparisons with and w/o instancing:

https://software.intel.com/en-us/articles/opengl-es-30-instanced-rendering

10

With regard to the vertex displacement shader (2 posts above this one). The animation is even jerkier if you set noSmooth(), so it is something to do with the way the texture is read. The spec says “level of detail [for texture lookup] is not implicitly computed for vertex shaders” (p93). Hmmm. I think someone needs to invest in a “GLES 3.0 programming recipes” book :-??

11

Here’s a very quick (and not terribly exciting) adaptation of @Simeon 's 2D instanced example, using 3D cubes instead of rects (GLES 2.0).

function setup()
    m = mesh()

    m.vertices, m.texCoords = addCube()
    m:setColors(color(255,200,0))
    m.shader = shader(vert, frag)
    m.texture = readImage("Planet Cute:Wall Block")
    numInstances = 100

    dummy = m:buffer("dummy") --test vec4 attribute
    transform = m:buffer("transform") --test mat4 attribute

    -- Here we resize the transform buffer to
    -- the number of instances, rather than vertices
    -- note that the buffer can be any size:
    --  if you render N instances with a buffer sized to M
    --  where N > M then the instances will use a divided
    --  version of the buffer
    --
    -- E.g. I render 10 instances with 5 transforms then
    --  instance 1,2 use transform 1
    --  instance 3,4 use transform 2
    --  instance 5,6 use transform 3
    --  ... etc
    transform:resize(numInstances)

    -- We have to tell Codea to treat this buffer as an
    -- instanced buffer, that is, it is not per-vertex
    transform.instanced = true

    for i=1,m.size do
        -- Set per vertex
        dummy[i] = vec4(0,0,0,0)
    end

    for i=1,numInstances do
        -- Sets one transform *per instance*
        local x = ((i-1)%10) - 5.5
        local y = math.floor((i-1)/10) - 5.5
        transform[i] = matrix():translate(x * 2, y * 2, 0)
    end
    rot = 0
    pos = vec3(0,0,0)
end

function draw()
    background(40, 40, 50)

    perspective()
    camera(30,40,20, 0,0,0, 0,0,1)
    -- mesh:draw can now take a number of instances
    -- it draws this many, instanced buffers can be
    -- used to differentiate each instance within a
    -- shader
    rot = (rot + 0.2%360)
    pos = pos + vec3(-0.02,0,0)
    translate(pos:unpack())
    rotate(rot)
    m:draw(numInstances)
end

vert=[[
uniform mat4 modelViewProjection;

attribute mat4 transform;
attribute vec4 position;
attribute vec4 color;
attribute vec4 dummy;
attribute mediump vec2 texCoord;

varying lowp vec4 vColor;
varying mediump vec2 vTexCoord;

void main()
{
    vColor = color;
    vTexCoord = texCoord;
    gl_Position = modelViewProjection * (transform * position + dummy);
}
]]

frag=[[
uniform sampler2D texture;
varying lowp vec4 vColor;
varying mediump vec2 vTexCoord;
void main()
{
    gl_FragColor = texture2D(texture, vTexCoord) * vColor;
}
]]

function addCube()
        local vertices = {
      vec3(-0.5, -0.5,  0.5), -- Left  bottom front
      vec3( 0.5, -0.5,  0.5), -- Right bottom front
      vec3( 0.5,  0.5,  0.5), -- Right top    front
      vec3(-0.5,  0.5,  0.5), -- Left  top    front
      vec3(-0.5, -0.5, -0.5), -- Left  bottom back
      vec3( 0.5, -0.5, -0.5), -- Right bottom back
      vec3( 0.5,  0.5, -0.5), -- Right top    back
      vec3(-0.5,  0.5, -0.5), -- Left  top    back
    }


    -- now construct a cube out of the vertices above
    local cubeverts = {
      -- Front
      vertices[1], vertices[2], vertices[3],
      vertices[1], vertices[3], vertices[4],
      -- Right
      vertices[2], vertices[6], vertices[7],
      vertices[2], vertices[7], vertices[3],
      -- Back
      vertices[6], vertices[5], vertices[8],
      vertices[6], vertices[8], vertices[7],
      -- Left
      vertices[5], vertices[1], vertices[4],
      vertices[5], vertices[4], vertices[8],
      -- Top
      vertices[4], vertices[3], vertices[7],
      vertices[4], vertices[7], vertices[8],
      -- Bottom
      vertices[5], vertices[6], vertices[2],
      vertices[5], vertices[2], vertices[1],
    }

    -- all the unique texture positions needed
    local texvertices = { vec2(0.03,0.24),
                          vec2(0.97,0.24),
                          vec2(0.03,0.69),
                          vec2(0.97,0.69) }
                
    -- apply the texture coordinates to each triangle
    local cubetexCoords = {
      -- Front
      texvertices[1], texvertices[2], texvertices[4],
      texvertices[1], texvertices[4], texvertices[3],
      -- Right
      texvertices[1], texvertices[2], texvertices[4],
      texvertices[1], texvertices[4], texvertices[3],
      -- Back
      texvertices[1], texvertices[2], texvertices[4],
      texvertices[1], texvertices[4], texvertices[3],
      -- Left
      texvertices[1], texvertices[2], texvertices[4],
      texvertices[1], texvertices[4], texvertices[3],
      -- Top
      texvertices[1], texvertices[2], texvertices[4],
      texvertices[1], texvertices[4], texvertices[3],
      -- Bottom
      texvertices[1], texvertices[2], texvertices[4],
      texvertices[1], texvertices[4], texvertices[3],
    }
    return cubeverts, cubetexCoords
end
12

@yojimbo2000 Is this worth looking thru.

http://ptgmedia.pearsoncmg.com/images/9780321933881/samplepages/0321933885.pdf
13

The GitHub repo for that book looks good:

https://github.com/danginsburg/opengles3-book/

14

in the book i read

Flat/smooth interpolators—In OpenGL ES 2.0, all interpolators were implicitly linearly interpolated across the primitive. In ESSL 3.00, interpolators (vertex shader outputs/fragment shader inputs) can be explicitly declared to have either smooth or flat shading.

maybe this is related to jerkiness?

15

Yeah I think so. I think this corresponds to smooth/ noSmooth in the Codea API, eg a low res texture in the frag shader displays pixelated with noSmooth, and blurred/smoothed out with smooth, and in this example, if you set noSmooth, you can clearly see the animation has “steps” as it goes from pixel to pixel. The problem is, it also has a jittery quality in smooth mode too (as if it’s going forwards and backwards over each part of the animation instead of just smoothly moving forwards). There are loads of new texture sampling functions in GLES 3.0 so maybe I need to try one of those instead of texture. I’ll ask on stack exchange, as there are people there with lots of 3.0 experience.

16

I tried playing around with cloth/hair simulation, here’s what I came up with

-- Cloth Sim

-- Use this function to perform your initial setup
function setup()
    displayMode(FULLSCREEN)
    numStrands = 300
    steps = 20
    vS = [[
//
// A basic vertex shader
//

//This is the current model * view * projection matrix
// Codea sets it automatically
uniform mat4 modelViewProjection;

//This is the current mesh vertex position, color and tex coord
// Set automatically
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;
attribute mat4 model;
attribute mat4 bend;

//This is an output variable that will be passed to the fragment shader
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
    //Pass the mesh color to the fragment shader
    vColor = color;
    vTexCoord = texCoord;
    
    //Multiply the vertex position by our combined transform
    mat4 mvp = modelViewProjection * model;
    for (float i = 0.0; i < abs(position.y - 0.5); i += 0.5) {
        mvp = mvp * bend;
    }
    gl_Position = mvp * position;
}
]]
    fS = [[
//
// A basic fragment shader
//

//Default precision qualifier
precision highp float;

//This represents the current texture on the mesh
uniform lowp sampler2D texture;

//The interpolated vertex color for this fragment
varying lowp vec4 vColor;

//The interpolated texture coordinate for this fragment
varying highp vec2 vTexCoord;

void main()
{
    //Sample the texture at the interpolated coordinate
    //lowp vec4 col = texture2D( texture, vTexCoord ) * vColor;
    
    //Set the output color to the texture color
    gl_FragColor = vColor;
}
]]
    strand = mesh()
    strand.shader = shader(vS, fS)
    local vertices = {}
    local cols = {}
    local len = 0.5
    local wid = 0.05
    for i = 0, steps do
        table.insert(vertices, vec3(-wid, (-0.5 - i) * len, 0.0))
        table.insert(vertices, vec3(wid, (-0.5 - i) * len, 0.0))
        table.insert(vertices, vec3(wid, (0.5 - i) * len, 0.0))
        table.insert(vertices, vec3(-wid, (-0.5 - i) * len, 0.0))
        table.insert(vertices, vec3(wid, (0.5 - i) * len, 0.0))
        table.insert(vertices, vec3(-wid, (0.5 - i) * len, 0.0))
        local mult = math.random() * 0.2 + 0.9
        local col = color(84 * mult, 71 * mult, 58 * mult, 255)
        for j = 1, 6 do
            table.insert(cols, col)
        end
    end
    strand.colors = cols
    strand.vertices = vertices
    model = strand:buffer("model")
    model.instanced = true
    model:resize(numStrands)
    for i = 1, numStrands do
        model[i] = matrix():translate(i * 0.05, 0, 0)
    end
    bends = {}
    bend = strand:buffer("bend")
    bend.instanced = true
    bend:resize(numStrands)
    for i = 1, numStrands do
        bend[i] = matrix()
    end
    cam = vec2(-10, -10)
    timeSpeed = 1
end

-- This function gets called once every frame
function draw()
    -- This sets a dark background color 
    background(191, 240, 255)

    -- This sets the line thickness
    strokeWidth(5)

    -- Do your drawing here
    cam = cam:rotate(math.rad(DeltaTime * timeSpeed * 22.5))
    camera(cam.x + numStrands * 0.025, 0, cam.y, numStrands * 0.025, -3.75, 0, 0, 1, 0)
    perspective(70)
    for i = 1, numStrands do
        bend[i] = matrix():rotate(
            (noise(ElapsedTime * timeSpeed, i * 0.05, 0) * 0.25
            + noise(ElapsedTime * timeSpeed, i * 0.025, 0) * 0.5
            + noise(ElapsedTime * timeSpeed, i * 0.0125, 0)) / 1.75
        * 2, 1, 0, 0):rotate(
            (noise(ElapsedTime * timeSpeed, i * 0.05, 16) * 0.25
            + noise(ElapsedTime * timeSpeed, i * 0.025, 16) * 0.5
            + noise(ElapsedTime * timeSpeed, i * 0.0125, 16)) / 1.75
        * 1, 0, 0, 1)
    end
    strand:draw(numStrands)
end

It can render 300 strands and keep a pretty stable 60 FPS…Instances seem pretty cool, and very powerful :smiley:

17

@SkyTheCoder impressive! Looks great.

18

Just one note (don’t have a recent enough iPad for GLES 3), but textures in vertex shaders came along I think in IOS 7/8 I did some stuff with it reasonably successfully a long time ago. They are very handy, but certainly not a ES3 only capability.

19

Gosh, @spacemonkey is right. I didn’t realised that that had been enabled. Here’s the above vertex displacement code for GLES 2.0. @spacemonkey do you know why the animation is jittery?

-- test of GLES 300 
-- textures on the vertex shader for vertex displacement
--

function setup()
    spriteMode(CORNER)
  --  noSmooth()
    m = mesh()
    local w,h = 20,20
    for x=1,10 do
        for y=1,10 do
            local r = m:addRect((x-5.5)*w,(y-5.5)*h,w,h)
            m:setRectTex(r, (x-1)*0.1,(y-1)*0.1,0.1,0.1)
            m:setRectColor(r, math.random(255), math.random(255), math.random(255))
        end
    end
    img = image(256,256)
    local fine = 10
    local seed = math.random() * 5000
    local seed2 = math.random() * 3000
    for x=1,img.width do
        for y = 1,img.height do
            local n1 = noise(x/fine, y/fine, seed)+1
            local n2 = noise(x/fine, y/fine, seed2)+1
            img:set(x,y, math.ceil(n1 * 20),math.ceil(n2*60),math.ceil( n1*128) )
        end
    end
    m.shader = shader(vert, frag)
    m.shader.heightMap = img
    offset = vec2(0,0)
    rot = 0
end

function draw()
    background(40, 40, 50)
    sprite(img, 0,0)
    perspective()
    camera(0,-200,180, 0,0,30, 0,0,1)

    rot = (rot + 0.2)%360
    rotate(rot)
    
    offset = offset + vec2(0.0002, -0.0002) --why does this produce jerky motion?
    m.shader.offset = offset
    m:draw()
end

vert=[[

uniform mat4 modelViewProjection;
uniform sampler2D heightMap;
uniform highp vec2 offset;

attribute highp vec2 texCoord;
attribute highp vec4 position;
attribute lowp vec4 color;

varying lowp vec4 vColor;

void main()
{
    vColor = color;
    vec3 displace = texture2D(heightMap, fract(texCoord + offset )).rgb; //textureOffset(heightMap, texCoord, offset).rgb
    gl_Position = modelViewProjection * (position + vec4(displace * 100., 0.));
}
]]

frag=[[

varying lowp vec4 vColor; 

void main()
{
    gl_FragColor = vColor;
}
]]
20

@SkyTheCoder good work!