## Rendering 2D Shadows for Convex Shapes

Sage
Posts: 1,066
Joined: 2004.07
Post: #1
Now that I've finally gotten figuring out the shadow area, what is an effective way of creating the shadows? My current implementation is to use a triangle strip and extrude points out of the shape using a constant value. This works great for many shapes, but for some it does not. For instance, this triangle's points get extruded outword but not away from the shape so the shadow ends on screen:

And here's that code:

Code:
```void renderShadow( const Shape &s ) {     vec2f p1, p2;     vec2f a, scaledA;          if( !findPoints( s, p1, p2 ) )         return;          bool started = false, finished = false;          glColor4f( .2f, .2f, .2f, .7f );     glBegin( GL_TRIANGLE_STRIP );          while( !finished ) {         for( int i = 0; i < s._points.size(); i++ ) {             a = s._points.at( i );                          if( !started && a == p2 )                 started = true;             else if( started && a == p1 )                 finished = true;                          if( started ) {                 scaledA = ( ( a - mousePosition ).normalize() ) * SHADOW_SCALE + a;                                  glVertex3f( a.x, a.y, 1.0f );                 glVertex3f( scaledA.x, scaledA.y, 1.0f );             }                          if( finished )                 break;         }                      if( !started )             break;     }          glEnd();      //    glPointSize( 5 ); //    glBegin( GL_POINTS ); //    glColor4f( 1, 0, 0, 1 ); //    glVertex2f( p2.x, p2.y ); //    glColor4f( 0, 1, 0, 1 ); //    glVertex2f( p1.x, p1.y ); //    glEnd(); }```

So any suggestions on some changes to ensure that the above image doesn't happen again?

Edit: The square in the middle does not cast a shadow, so that's not a problem. However, if anyone has pointers to using a stencil buffer (in OpenGL) so that only the parts of the square not in shadow appear, I'd appreciate it .
Sage
Posts: 1,066
Joined: 2004.07
Post: #2
I still have the problem with my shadow lengths, but I did figure out the stencil buffer to only draw certain objects when in light. It's an effect I want for some things. In this picture the blue shapes are all drawn no matter what (with the shadows over them using alpha blending), but the crate is only drawn in the light.

So now in addition to figuring out how to properly scale the shadows, does anyone know how to take this up a notch and use multiple lights? I've given a few things a try with inverting the stencil buffer, but it didn't work out quite right. My ideal solution would make the shadows additive and still hide parts of the pink square that lie in either shadow.
Sage
Posts: 1,487
Joined: 2002.09
Post: #3
I was going to use this effect for a side-scroller that some friends and I wanted to make. Ultimately we canned it because none of us had the necessary art/level design skills to make something as large as we wanted.

I did have pretty cool shadows working though. Hard shadows are really easy.
1.) Clear the screen to the ambient light color. The framebuffer is going to be used as a lightmap that you build before blending the scene with it.

2.) Stencil quads for each back facing segment is projected off to infinity from the light. I used homogenous coordinates for this, but you could project the shadows only a certain length for effect.

3.) Blend the light's color onto the framebuffer, masked out by the stencil. You can even use a luminance texture to shape the light.

4.) Clear the stencil buffer, and repeat 2 & 3 for all lights.

5.) Draw the scene while multiplying it with lightmap. Keep in mind that you have to draw things front to back for this to work, and you won't be able to use alpha blending. If you need to do anything fancier, you'll have to render the scene or the light map to a texture and apply it afterwards.

I have (Ruby) code for all this if you would like to see it. (70 lines or so) Unless you have thousands of line segments that you want to shadow, you will probably be limited by fill rate. Don't use 10k x 10k resolutions, and don't use hundreds of lights as you are clearing and drawing the stencil buffer for every light.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Sage
Posts: 1,066
Joined: 2004.07
Post: #4
Hm. Thanks for the tip. I'm going to see what I can do. After a few tries my lightmap render-to-texture seems to just be a white box, so I have some more work to do .
Sage
Posts: 1,066
Joined: 2004.07
Post: #5
Ok. I really can't see any blatantly obvious problems in my code so far. Anyone have any idea why my resulting texture is blank white?

Code:
```GLuint EmptyTexture( int width, int height ) {     GLuint txtnumber;     unsigned int* data;          data = ( unsigned int* )new GLuint[ ( ( width * height )* 4 * sizeof( unsigned int ) ) ];          glGenTextures( 1, &txtnumber );     glBindTexture( GL_TEXTURE_2D, txtnumber );     glTexImage2D( GL_TEXTURE_2D, 0, 4, width, height, 0,                  GL_RGBA, GL_UNSIGNED_BYTE, data );     glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );     glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );          delete [] data;          return txtnumber; } ... vec2f size( 800, 600 ); GLuint lightTexture = EmptyTexture( size.x, size.y ); ... //clear the color and depth buffers     glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );          vector< shared_ptr< Light > >::iterator litr = lights.begin(), lend = lights.end();     vector< Shape >::iterator itr, end;     while( litr != lend ) {         shared_ptr< Light > light = ( *litr );                  //clear the stencil buffer         glClear( GL_STENCIL_BUFFER_BIT );                  //turn off the color and depth buffers         glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );         glDepthMask( GL_FALSE );                  //render the shadows to the stencil buffer setting all         //0s to all non-affected pixels         glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFF );         glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );                  itr = shapes.begin(), end = shapes.end();                  glColor4f( 0, 0, 0, 1 );         while( itr != end ) {             Shape shape = ( *itr );                          if( !shape.isPointInShape( light->_pos ) )                 renderShadow( shape, light->_pos );                          ++itr;         }                  //turn on the color and depth buffers         glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );                  //render our light color where there is no shadow         glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );         glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );                  light->_color.set();         glBegin( GL_QUADS );         glVertex2f( 0, 0 );         glVertex2f( 0, size.y );         glVertex2f( size.x, size.y );         glVertex2f( size.x, 0 );         glEnd();                  ++litr;     }          glEnable( GL_TEXTURE_2D );     //copy out our light texture     glBindTexture( GL_TEXTURE_2D, lightTexture );     glCopyTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, 0, 0, size.x, size.y, 0);     glDisable( GL_TEXTURE_2D ); //draw scene ...     glDisable( GL_STENCIL_TEST );          glPushMatrix();     glEnable( GL_TEXTURE_2D );     glBindTexture( GL_TEXTURE_2D, lightTexture );     glTranslatef( 0, 0, 10 );     glColor4f( 1, 1, 1, 1 );     glBegin( GL_QUADS );     glTexCoord2f( 0, 0 );    glVertex2f( 0, 0 );     glTexCoord2f( 0, 1 );    glVertex2f( 0, size.y );     glTexCoord2f( 1, 1 );    glVertex2f( size.x, size.y );     glTexCoord2f( 1, 0 );    glVertex2f( size.x, 0 );     glEnd();     glDisable( GL_TEXTURE_2D );     glPopMatrix();          glEnable( GL_STENCIL_TEST );          SDL_GL_SwapBuffers();```
Luminary
Posts: 5,143
Joined: 2002.04
Post: #6
TEXTURE_2D textures must be power-of-two*?

* actually, it's not quite that simple these days... just ask for the full rant
Sage
Posts: 1,066
Joined: 2004.07
Post: #7

* actually, it's not quite that simple these days... just ask for the full rant

Ah crap. Forgot. Is there any way to do non-power of 2 through extensions or anything? And I'd read the whole rant if you typed it
Sage
Posts: 1,487
Joined: 2002.09
Post: #8
GL_TEXTURE_RECTANGLE_EXT or one of its cousins.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Sage
Posts: 1,066
Joined: 2004.07
Post: #9
Skorche Wrote:GL_TEXTURE_RECTANGLE_EXT or one of its cousins.

Thanks. Now at least I have a yellow box .

So I think I'm having some error in my code for rendering the light. Right now I only have one light, but I don't see any shadows at all (just one big yellow box). Anything seem out of the ordinary?

Code:
```glEnable( GL_STENCIL_TEST );          //clear the color buffer     glClear( GL_COLOR_BUFFER_BIT );          vector< shared_ptr< Light > >::iterator litr = lights.begin(), lend = lights.end();     vector< Shape >::iterator itr, end;     while( litr != lend ) {         shared_ptr< Light > light = ( *litr );                  //clear the stencil buffer         glClear( GL_STENCIL_BUFFER_BIT );                  //turn off the color and depth buffers         glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );         glDepthMask( GL_FALSE );                  //render the shadows to the stencil buffer         glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFF );         glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );                  itr = shapes.begin(), end = shapes.end();                  glColor4f( 0, 0, 0, 1 );         while( itr != end ) {             Shape shape = ( *itr );                          if( !shape.isPointInShape( light->_pos ) )                 renderShadow( shape, light->_pos );                          ++itr;         }                  //turn on the color and depth buffers         glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );         glDepthMask( GL_TRUE );                  //render our light color where there is no shadow         glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );         glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );                  light->_color.set();         glBegin( GL_QUADS );         glVertex2f( 0, 0 );         glVertex2f( 0, size.y );         glVertex2f( size.x, size.y );         glVertex2f( size.x, 0 );         glEnd();                  ++litr;     }```
Luminary
Posts: 5,143
Joined: 2002.04
Post: #10
In the beginning, there was TEXTURE_2D. It has texture coordinates that run 0..1,0..1 ("normalized"), allows a variety of wrapping modes (most usefully REPEAT and CLAMP_TO_EDGE_SGIS), and allows mipmapping.

Around the time of the GeForce 2, NVidia's hardware started supporting non-power-of-two textures, in a limited kind of way. They made an extension, NV_texture_rectangle, to expose this functionality. NV_texture_rectangle introduced a new texture target (TEXTURE_RECTANGLE_NV), disallowed certain wrap modes (notably REPEAT), disallowed mipmapping, and used pixel coordinates rather than normalized coordinates, because that's what their hardware does internally.

Roll forward a few years, and in the 10.1 timeframe Apple's looking for a good way to upload things like movies to the video card. NV_texture_rectangle looks like a good fit for the job, but they need something for ATI hardware as well. It turns out that ATI Radeon cards actually do support the same functionality, so Apple rebrands the extension EXT_texture_rectangle, and implements it for both NVidia and ATI hardware. So far so good.

Roll forward a few years, and NVidia releases the GeForce 6 series of GPUs. These support unrestricted non-power-of-two texturing, and so a new extension is called for. ARB_texture_non_power_of_two is very simple conceptually, it says "it's not an error for your TEXTURE_2D texture to have non-power-of-two dimensions any more". Note that texture coordinates are still normalized, all wrap modes are still supported, and most importantly, mipmapping is still allowed.

Shortly after that, the OpenGL ARB were specifying OpenGL 2.0, and ARB_texture_non_power_of_two caught their eye as a useful kind of thing to add, seeing as how it removes a lot of "unnecessary" restrictions. Add it they did, very much against ATI's wishes -- whose hardware didn't support it at all.

Now, here's where things get interesting -- ATI can't very well sit back and say "well, um, we don't support OpenGL 2.0" whilst NVidia's out there trumpeting their OpenGL 2.0 support from the rooftops... but they can't get around the fact that their hardware doesn't support mipmapping (or REPEAT, I think) on non-power-of-two textures.

So they do something a little bit sneaky... they implement a software fallback for the parts they don't support, and claim "OpenGL 2.0 support". However, they don't export the ARB_texture_non_power_of_two extension, to indicate that one shouldn't expect this to run in hardware. In my mind, this is all fair enough -- there are plenty of parts of OpenGL 1.2 even that *no* hardware supports, and this isn't really any different.

Shortly after, somebody realizes that it'd be useful to use the old EXT_texture_rectangle textures in GLSL, and the new ARB_texture_rectangle extension is born, which is just EXT_texture_rectangle + some new bits in GLSL to allow access to these textures.

As if all this isn't bad enough, in 10.4.8, Apple comes to implementing OpenGL 2.0 (at last!). For whatever peculiar and unhelpful reasons of their own, they choose to export the ARB_texture_non_power_of_two extension on all hardware that exports OpenGL 2.0, including all the ATI hardware that doesn't really support the extension. That means that the extension check on Mac OS X is *not* a useful way to determine hardware support for the feature, and we must fall back to mac-specific ways to detect whether rendering is hardware-accelerated.

All-in-all, a right mess:

NVidia on Windows/Linux:
* supports NV_texture_rectangle
* supports ARB_texture_rectangle
* supports ARB_texture_non_power_of_two
* supports OpenGL 2.0

ATI on Windows/Linux:
* supports EXT_texture_rectangle
* supports OpenGL 2.0 (in software)

NVidia on Mac OS X:
* supports EXT_texture_rectangle
* supports ARB_texture_rectangle
* supports ARB_texture_non_power_of_two
* supports OpenGL 2.0

ATI on Mac OS X:
* supports EXT_texture_rectangle
* supports ARB_texture_rectangle
* supports ARB_texture_non_power_of_two (in software)
* supports OpenGL 2.0 (in software)

Notice that there's no single cross-platform way to use rectangle textures (best bet is to try ARB_texture_rectangle [everywhere except ATI/PC] and fall back to EXT_texture_rectangle).

Notice that there's no single cross-platform way to use non-power-of-two textures with GLSL (best bet is to try ARB_texture_rectangle [everywhere except ATI/PC] and fall back to OpenGL 2.0)

Notice that there's no single cross-platform way to check if ARB_texture_non_power_of_two textures are hardware-accelerated (best bet is to test for ARB_texture_non_power_of_two [everywhere except Mac OS X], and then on Mac OS X, test if it's actually accelerated using the Apple-specific APIs)

If that's not enough to make you cry, you're stronger than I am.
Sage
Posts: 1,487
Joined: 2002.09
Post: #11
Your stencil function looks the same as mine. Are you sure you aren't missing some other important state that you set?

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Sage
Posts: 1,066
Joined: 2004.07
Post: #12
I never would've guessed it was that complicated.

I wound up just using a power of two texture larger than my screen and using just the portion necessary. My screen is 800x600 and the texture is 1024x1024 so I'm not wasting too much memory. At least it works now...
Sage
Posts: 1,066
Joined: 2004.07
Post: #13
Skorche Wrote:Your stencil function looks the same as mine. Are you sure you aren't missing some other important state that you set?

I'm not sure, but I do have it working now. However, I'm having trouble with using light color now. I can't figure out a good way to get the two light's colors to blend over each other and each other's shadows without blending with their own shadows. Here's my total rendering code as is. Any ideas?

Code:
```void render() {     glLoadIdentity();          vec3i size = Application::application()->size();          glEnable( GL_STENCIL_TEST );     glDisable( GL_TEXTURE_2D );          //clear the color buffer     glClearColor( 0, 0, 0, 0 );     glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );          vector< shared_ptr< Light > >::iterator litr = lights.begin(), lend = lights.end();     vector< Shape >::iterator itr, end = shapes.end();     int layer = 0;     while( litr != lend ) {         shared_ptr< Light > light = ( *litr );                  //clear the stencil buffer         glClear( GL_STENCIL_BUFFER_BIT );                  //turn off the color and depth buffers         glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );         glDepthMask( GL_FALSE );                  //render the shadows to the stencil buffer         glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFF );         glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );                  itr = shapes.begin();                  glColor4f( 0, 0, 0, 1 );         while( itr != end ) {             Shape shape = ( *itr );                          if( !shape.isPointInShape( light->_pos ) )                 renderShadow( shape, light->_pos );                          ++itr;         }                  //turn on the color and depth buffers         glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );         glDepthMask( GL_TRUE );                  //render the shadows themselves         glStencilFunc( GL_EQUAL, 1, 0xFFFFFFFF );         glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );                  glColor4f( 0, 0, 0, .9f );         glBegin( GL_QUADS );         glVertex3f( 0, 0, layer );         glVertex3f( 0, size.y, layer );         glVertex3f( size.x, size.y, layer );         glVertex3f( size.x, 0, layer );         glEnd();                  //render our light color where there is no shadow         glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );         light->_color.set();         glBegin( GL_QUADS );         glVertex3f( 0, 0, layer );         glVertex3f( 0, size.y, layer );         glVertex3f( size.x, size.y, layer );         glVertex3f( size.x, 0, layer );         glEnd();                  ++litr;         ++layer;     }          //copy out our light texture     glEnable( GL_TEXTURE_2D );     glBindTexture( GL_TEXTURE_2D, lightTexture );     glCopyTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, 0, 0, 1024, 1024, 0);     glDisable( GL_TEXTURE_2D );          //clear the color and depth buffers     glClearColor( 1, 1, 1, 1 );     glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );      //    //render our "dynamic shapes" which only //    //appear outside of shadows     glDisable( GL_STENCIL_TEST ); //    glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );     glColor4f( 1, 1, 1, 1 );     glEnable( GL_TEXTURE_2D );     glBindTexture( GL_TEXTURE_2D, box->id );     glBegin( GL_QUADS );     glTexCoord2f( 0, 0 );    glVertex2f( 400, 300 );     glTexCoord2f( 0, 1 );    glVertex2f( 400, 350 );     glTexCoord2f( 1, 1 );    glVertex2f( 450, 350 );     glTexCoord2f( 1, 0 );    glVertex2f( 450, 300 );     glEnd();     glDisable( GL_TEXTURE_2D );           //    //draw our background image     glPushMatrix();     glEnable( GL_TEXTURE_2D );     glBindTexture( GL_TEXTURE_2D, ground->id );     glTranslatef( 0, 0, -2 );     glColor4f( 1, 1, 1, 1 );     glBegin( GL_QUADS );     glTexCoord2f( 0, 0 );    glVertex2f( 0, 0 );     glTexCoord2f( 0, 1 );    glVertex2f( 0, size.y );     glTexCoord2f( 1, 1 );    glVertex2f( size.x, size.y );     glTexCoord2f( 1, 0 );    glVertex2f( size.x, 0 );     glEnd();     glDisable( GL_TEXTURE_2D );     glPopMatrix();          //render static geometry that is drawn whether in-     //or outside of a shadow     itr = shapes.begin();     glColor4f( .5f, .3f, 1.0f, 1.0f );     while( itr != end ) {         Shape shape = ( ( *itr ) );                  shape.draw();                  ++itr;     }          glPushMatrix();     glTranslatef( 0, 0, 10 );     glColor4f( 1, 1, 1, 1 );     litr = lights.begin();     glPointSize( 5 );     glBegin( GL_POINTS );     while( litr != lend ) {         glVertex2f( ( *litr )->_pos.x, ( *litr )->_pos.y );         ++litr;     }     glEnd();     glPopMatrix();          glDisable( GL_STENCIL_TEST );         glPushMatrix();     glEnable( GL_TEXTURE_2D );     glBindTexture( GL_TEXTURE_2D, lightTexture );     glTranslatef( 0, 0, 10 );     glColor4f( 1, 1, 1, 1 );     glBegin( GL_QUADS );          glTexCoord2f( 0, ( float )size.y / 1024.0f );     glVertex2f( 0, 0 );          glTexCoord2f( 0, 0 );     glVertex2f( 0, size.y );          glTexCoord2f( ( float )size.x / 1024.0f, 0 );     glVertex2f( size.x, size.y );          glTexCoord2f( ( float )size.x / 1024.0f, ( float )size.y / 1024.0f );     glVertex2f( size.x, 0 );          glEnd();     glDisable( GL_TEXTURE_2D );     glPopMatrix();          SDL_GL_SwapBuffers(); }```

Edit: Here's a picture of the problem. Though this is only using white lights, it shows the problem. The one light gets rendered first and the shadows are cast. Then the next light lightens those shadows a bit and casts shadows of it's own. Unfortunately, these shadows then are not lit by the first light. Any ideas?

Sage
Posts: 1,487
Joined: 2002.09
Post: #14
Are you alpha blending the light values? Just add them.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Luminary
Posts: 5,143
Joined: 2002.04
Post: #15
Code:
```render scene to depth, and ambient light to color buffer for each light     clear the stencil buffer     render the shadow volumes to the stencil buffer     render the scene to the color buffer, lit by the light, with         * depth test set to equal; and         * depth writes turned off; and         * additive blending enabled```