GLSL talk and demo

General
1

So, GLSL. It’s been in the buzz, mentioned in various threads - I’ve certainly been frothing about it. TLL has mentioned it may show up as a feature. What is it, and should you care? Let’s find out.

First, a link to a free app: http://itunes.apple.com/us/app/paragraf/id422685475?mt=8&ls=1

That’s “Paragraf”, an app that lets you write GLSL stuff, much as Codea let’s you write Lua stuff. Rather than wasting your time describing things, i suggest you download it and mess around for 5 minutes, looking at the examples. I hate their color scheme, but I love what they’ve done, and free is downright generous. 5 stars.

In a nutshell - GLSL let’s you write special programs called “shaders” that execute on the GPU instead of the CPU. The GPU is optimized to do this sort of stuff very, very fast, massively in parallel, so some graphics effects can be thousands of times faster than they would be implemented on the CPU in any normal language. “shader” is a historic term - they used to be only for shading, but are now general purpose, usable for all kinds of graphical manipulation, including 3d transforms, filters, and even oddball stuff like Conway’s Life game, which blows me away to this day. The Pargraf app examples are a good overview of the variety of effects you can achieve.

GLSL is insanely powerful, and profoundly crippled at the same time; it does one thing - graphics manipulation and effects - hugely well, and other things (like general purpose computing, or simple input/output) poorly or not at all. This is why a marriage of Codea and GLSL is so exciting - each can cover for the others shortcomings.

Fair warning - if there ever was an “advanced users” feature, this is it; I won’t pretend I even understand how most of the examples work! But - learning is fun, and I’m perfectly ok swiping bits of code while I do.

Anyway - found this while spelunking, wanted to share. It’s not in Codea, but I think it’s firmly in the “desirable and likely” category - it was first suggested by @Dylan, one of the Codea devs. Plus, I’ve been raving like a lunatic about it - go look and see for yourself if I’m blowing it out of proportion.

Best of all - I can think of no conceivable reason why Apple would balk at its inclusion; there’s even a sample iOS project up in github (easy to google for, but let me know if I should post a link), so it’s hopefully straightforward to implement as part of Codea as well.

2

I’m not against the inclusion of GLSL, but there’s a lot of editor UI stuff to handle, and exactly how we allow you to communicate with GLSL shaders from the Codea API:

How do you set uniforms and attributes? How do you tell a shader to use an image as a uniform sampler2d? How do you multi-pass. And so on. The shader API (not the language) in OpenGL is esoteric, passing it off directly inside Codea seems ugly. I think there is an elegant solution, but it’s one we will probably explore after some of the more requested features.

3

Mr. Bortels I thank you for the pointer to Paragraf. This, dear sir, is trés cool. Love the camera input!

Edit: @Simeon, if anyone can do elegant, you/TLL/Codea can.

4

I’d be comfortable simply passing them a string with the GLSL code. If you wanted to get fancy, special-handle a GLSL tab. As I said - advanced feature. But this stuff being hard doesn’t make it bad - it makes it FUN, especially given the power it adds. Not everyone wants to ski the bunny slopes.

I would suggest quick-and-dirty to begin with - if you call glsl(string), it won’t dirty up things for people who don’t want to use it. Then once we have some experience with it, a more refined implementation can be introduced. I like to think the various font libraries people wrote fed into your design decisions for native font support - a “preview” implementation may give you data for how to implement it elegantly down the line.

Disclaimer: when I ski for real, it’s always the bunny slopes… I am teh wimp.

5

Yes, but then how do you get data into your shaders? If you want to run your shader on an image, you need a way of telling OpenGL to link the texture object to a uniform in the fragment shader. This would have to be represented somehow in the Codea API. I’m sure it can be represented, but we would have to design that.

We can do a basic version first if you like, one that loads strings. But we’d still need to design the API to provide input to the shader. And this would still come after the features we’ve already promised.

6

Agree with @simeon. I think 2LL should focus first on features that can be used by EVERY users instead of features that can only be used by SOME (advance) users. :wink:

7

Hm… while it is undoubtedly true that (as users) our concrete desires can’t all be met at once… it seems to me that (as users) we are best served when we are encouraged in our excitement and dreams.

Edit: I read once how Mr. Jobs disliked ‘focus groups of users’ because he maintained that ‘people don’t know what they want.’

Something to keep in mind when we love (and kill) our visionaries/crazy ones.

PS: But yes, if I had to state my own druthers, I’d want to see text, text input and more graphic primitives first… but only because I currently have clear (albeit limited) dreams of how I would put them to use. Recognizing my limitations I’m very grateful to Mr. Bortels for his effort in introducing something I didn’t know about and broaching the subject with Simeon.

8

@Blanchot most of the features I’d like to include first are features that I personally want (physics, custom sprites, text). GLSL is nice, too. And we’ll get there. Most of the delays are due to Apple’s review times anyway.

9

@Simeon good to hear that you are “stepping to the music” which you yourself hear! At least that’s what’s most important in my book. Cheers!

10

paragraf looks nice. They could learn from the Codea guys with regard to the editor, though. I found myself looking for the cursor keys … say no more!

Being able to interleave this with lua would be great.

I notice that paragraf has code sharing - you can email a shader (not sure about import as yet).

11

That was pretty cool… :slight_smile: leaves one a bit dizzy though! It would be a great feature down the track to allow the camera or other arbitrary input data to feed a shader. Thanks, @Bortels for explaining it.

12

Yes - Codea has crippled me keyboard-wise, I habitually look for cursor control characters that aren’t there in other apps. They seem to be using the bog-standard text input box, which we all know sucks for coding. (total aside - if you don’t already have it, get the free wolfram alpha app - check out THAT keyboard.)

Paragraf is also crashy, unfortunately. I don’t know if that’s inherent in GLSL or it’s the app; but I’m inclined to point at the app, since I’m just trying their “help” examples. Still - you get more than you paid for it, so judge lightly.

How to get an image to a shader? Presumably you pass a pointer (or more exactly, an image() object) to it, along with the GLSL code, with nil meaning the screen? More interesting to me is output - presumably for maximum flexibility, you’d render back to another image(), not just to the screen - you could then have multiple GLSL pipelines for multiple effects. I liked the idea you had in the “we want video” thread, about having the camera live feed or a video decoding into an image, that you could then sprite() as usual. Can GLSL do that (support multiple pipelines)? I suspect so, I can run multiple quartz composer projects on the mac in different windows, but I’m by no means sure - thus the research and this thread, where I hope to nail down details and expand mindshare (aieeeeee I sound like my work). Point is - how we pass in and out data are not necessarily things I’d get to pick - they depend on architectural details of the internals of Codea, and lua, and GLSL, none of which I can speak to with authority.

But I’ll tell you this - right now, the only thing I found on the app store that let you do GLSL was Paragraf, and it’s a toy. If you added GLSL, especially in a manner like the above (multiple inputs and outputs, ooh chained maybe) you’d be all alone in a hot, hot niche - this technology is really new, and a lot of graphics-heads miss quartz composer on the iPad, and this would blow QC away in terms of flexibility and ease-of-use. (One of QCs biggest assets was also its biggest flaw -visual programming. Big projects get complex, and look like spaghetti - trying to keep code legible is hard enough with text, with pictures it gets much worse)

As for when - I won’t tell you this should bump things like native fonts or camera access (just the opposite!) - but I also know we’re mostly waiting on apple anyway. I merely suggest that, done right, this could make Codea something more than simply self-hosted lua with some useful libraries and a nice editor. Those are sufficient and good, but lua and a good editor don’t make people say “holy crap, how’d they do THAT?”. I’d prioritize that higher than some of the other things suggested, myself.

I think this could have game changing potential. VisiCalc single-handed ly made the Apple II a requirement for business. I think this has the potential to be that important to some people. Of all the features suggested, this has crazy “bang for the buck”.

13 
But I'll tell you this - right now, the only thing I found on the app store that let you do GLSL was Paragraf...

There is also an app named RenderDuck (also free, and also, like Paragraf, filed in my appstore under ‘Education’).

14

Neat - good find on RenderDuck.

15

Aaaggghhh …

Too many years of Perl, PHP, TeX, and now Lua have ruined what little C skills I ever had. I’ve lost count of the number of times paragraf has complained about me writing stuff like if (a > 1) {...} where a is a float. And I’ve only written one shader so far.

16

Wow! Paragraf even runs on the iPhone. That is amazing.

17

I’ve spent today’s commutes writing GLSL code on my phone with Paragraf. Bloody amazing app! I’m now sold on Bortel’s idea of GLSL tabs in Codea. I want it!

I wonder if GLSL input/output variables could be bound to Lua globals in the same way that Codea’s watches and parameters are at the moment. That could make for some really cool effects.

18

That’s the idea, kinda. You wouldn’t want them all bound, but a mechanism by which you can either pass in or bind variables would be important - Note, for example, Paragraf makes the current touch (up to 3 multitouch, actually, I think) available. As well as the image from the camera, of course.

I’d also like to see outbound binding as well. Thinking out loud, assume we have a glsl() routine that accepts as parameters some code and an input context (ie. an image). You might do something like this:


sepiatone = "some GLSL to do a sepia tone shading"
lens = "some GLSL to do a lens effect"
sprite(glsl(sepiatone, glsl(lens, glsl(camera))), WIDTH/2, HEIGHT/2)

Idea there being you have a “camera” bit of glsl (or hardcoded stuff) that simply returns the current camera image. To apply a GLSL effect, you call glsl with the code, and the input - so you can nest them to chain them. glsl returns an image - that’s either used as input to another GLSL, or composited to the screen.

We’d want some way to refer to a few “magic” contexts - “frontcamera”, “backcamera”, and “screen” come to mind. (we could use setContext(), but I can see it being messy, not that the above isn’t).

19

Bortels (or anyone else): How does one do an array in GLSL? I’ve been trying all sorts of things but I can’t get paragraf to accept any of my attempts. I really want an array of vec2 objects, but I’d settle for two arrays of floats.

20

http://www.pouet.net/topic.php?which=7102