So I've done a fair amount of programming with the OpenGL fixed-function pipeline: setup projection & lighting, render lots of triangles, rinse and repeat. But now I'd like to start experimenting with some screen-space techniques for non-photorealistic rendering.

Here's a mockup I did in GIMP: The idea is to take a scene (first image), render a black and white mask of it to a texture or FBO or whatever (second image), blur it (third image), and use the blurred version to modulate the scene (last image):



Yes, this example is kinda crappy and contrived, but the effect I want depends on using a blurred mask of the scene as a texture to modulate the final output.

My understanding of how I would do this goes like:
1) Render scene to FBO #1, no lighting/shading/texturing. Objects white. Background black.
2) Do a gaussian blur by rendering a quad into FBO #2 that does texture lookups in FBO #1
3) Render the (lit and colored) scene to the actual framebuffer, using FBO #2 as a screen-space texture

Ok, so here's the actual questions:
* Does the plan sound valid?
* Can FBO #1 and FBO #2 be created as rectangular (non-power-of-two) textures?
* Do I need to do anything clever to make sure that the pixel coordinates in the final scene correspond to the texture coordinates of FBO #2?

Pointers to sample code are appreciated!

* Yes, that's valid. Depending on your scene (opaque only?) you can do step 1) in the alpha channel of your regular scene draw.
* FBOs can be 2D, RECT, or NPOT 2D textures, depending on your hardware. On the desktop, your best (i.e. most compatible across all hardware) choice is RECT textures.
* TexCoords will either be a subrect of [0..1], or [0..N], or [0,1] depending what texture target you choose.

You should research separable gaussian blurs, as there are a couple ways to do this, with varying quality/speed tradeoffs. You should also establish your minimum hardware requirement, as that influences the choices you make.

Cool. I'm plenty familiar with seperable blurs. I'm hoping to get a GLUT or SDL version of this simple cube example coded up tomorrow... Obviously, after voting

Thanks!

It works! I'm using the blurred mask to erode corners and edges of geometry. Well at least on this cube. Before and after:



There's a movie at http://www.cs.cmu.edu/~mzucker/code/npr/smoothcube.mp4 - smoothing kicks in halfway through.

That's pretty sweet.