Rendering 2D Shadows for Convex Shapes

Sage
Posts: 1,066
Joined: 2004.07
Post: #1
Now that I've finally gotten figuring out the shadow area, what is an effective way of creating the shadows? My current implementation is to use a triangle strip and extrude points out of the shape using a constant value. This works great for many shapes, but for some it does not. For instance, this triangle's points get extruded outword but not away from the shape so the shadow ends on screen:

[Image: shadow_screen1.png]

And here's that code:

Code:
void renderShadow( const Shape &s ) {
    vec2f p1, p2;
    vec2f a, scaledA;
    
    if( !findPoints( s, p1, p2 ) )
        return;
    
    bool started = false, finished = false;
    
    glColor4f( .2f, .2f, .2f, .7f );
    glBegin( GL_TRIANGLE_STRIP );
    
    while( !finished ) {
        for( int i = 0; i < s._points.size(); i++ ) {
            a = s._points.at( i );
            
            if( !started && a == p2 )
                started = true;
            else if( started && a == p1 )
                finished = true;
            
            if( started ) {
                scaledA = ( ( a - mousePosition ).normalize() ) * SHADOW_SCALE + a;
                
                glVertex3f( a.x, a.y, 1.0f );
                glVertex3f( scaledA.x, scaledA.y, 1.0f );
            }
            
            if( finished )
                break;
        }
            
        if( !started )
            break;
    }
    
    glEnd();
    
//    glPointSize( 5 );
//    glBegin( GL_POINTS );
//    glColor4f( 1, 0, 0, 1 );
//    glVertex2f( p2.x, p2.y );
//    glColor4f( 0, 1, 0, 1 );
//    glVertex2f( p1.x, p1.y );
//    glEnd();
}


So any suggestions on some changes to ensure that the above image doesn't happen again?

Edit: The square in the middle does not cast a shadow, so that's not a problem. However, if anyone has pointers to using a stencil buffer (in OpenGL) so that only the parts of the square not in shadow appear, I'd appreciate it Wink.
Quote this message in a reply
Sage
Posts: 1,066
Joined: 2004.07
Post: #2
I still have the problem with my shadow lengths, but I did figure out the stencil buffer to only draw certain objects when in light. It's an effect I want for some things. In this picture the blue shapes are all drawn no matter what (with the shadows over them using alpha blending), but the crate is only drawn in the light.

[Image: shadow_screen2.png]

So now in addition to figuring out how to properly scale the shadows, does anyone know how to take this up a notch and use multiple lights? I've given a few things a try with inverting the stencil buffer, but it didn't work out quite right. My ideal solution would make the shadows additive and still hide parts of the pink square that lie in either shadow.
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #3
I was going to use this effect for a side-scroller that some friends and I wanted to make. Ultimately we canned it because none of us had the necessary art/level design skills to make something as large as we wanted.

I did have pretty cool shadows working though. Hard shadows are really easy.
1.) Clear the screen to the ambient light color. The framebuffer is going to be used as a lightmap that you build before blending the scene with it.

2.) Stencil quads for each back facing segment is projected off to infinity from the light. I used homogenous coordinates for this, but you could project the shadows only a certain length for effect.

3.) Blend the light's color onto the framebuffer, masked out by the stencil. You can even use a luminance texture to shape the light.

4.) Clear the stencil buffer, and repeat 2 & 3 for all lights.

5.) Draw the scene while multiplying it with lightmap. Keep in mind that you have to draw things front to back for this to work, and you won't be able to use alpha blending. If you need to do anything fancier, you'll have to render the scene or the light map to a texture and apply it afterwards.

I have (Ruby) code for all this if you would like to see it. (70 lines or so) Unless you have thousands of line segments that you want to shadow, you will probably be limited by fill rate. Don't use 10k x 10k resolutions, and don't use hundreds of lights as you are clearing and drawing the stencil buffer for every light.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Sage
Posts: 1,066
Joined: 2004.07
Post: #4
Hm. Thanks for the tip. I'm going to see what I can do. After a few tries my lightmap render-to-texture seems to just be a white box, so I have some more work to do Smile.
Quote this message in a reply
Sage
Posts: 1,066
Joined: 2004.07
Post: #5
Ok. I really can't see any blatantly obvious problems in my code so far. Anyone have any idea why my resulting texture is blank white?

Code:
GLuint EmptyTexture( int width, int height ) {
    GLuint txtnumber;
    unsigned int* data;
    
    data = ( unsigned int* )new GLuint[ ( ( width * height )* 4 * sizeof( unsigned int ) ) ];
    
    glGenTextures( 1, &txtnumber );
    glBindTexture( GL_TEXTURE_2D, txtnumber );
    glTexImage2D( GL_TEXTURE_2D, 0, 4, width, height, 0,
                 GL_RGBA, GL_UNSIGNED_BYTE, data );
    glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
    glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
    
    delete [] data;
    
    return txtnumber;
}

...
vec2f size( 800, 600 );
GLuint lightTexture = EmptyTexture( size.x, size.y );

...

//clear the color and depth buffers
    glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );
    
    vector< shared_ptr< Light > >::iterator litr = lights.begin(), lend = lights.end();
    vector< Shape >::iterator itr, end;
    while( litr != lend ) {
        shared_ptr< Light > light = ( *litr );
        
        //clear the stencil buffer
        glClear( GL_STENCIL_BUFFER_BIT );
        
        //turn off the color and depth buffers
        glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
        glDepthMask( GL_FALSE );
        
        //render the shadows to the stencil buffer setting all
        //0s to all non-affected pixels
        glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFF );
        glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );
        
        itr = shapes.begin(), end = shapes.end();
        
        glColor4f( 0, 0, 0, 1 );
        while( itr != end ) {
            Shape shape = ( *itr );
            
            if( !shape.isPointInShape( light->_pos ) )
                renderShadow( shape, light->_pos );
            
            ++itr;
        }
        
        //turn on the color and depth buffers
        glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );
        
        //render our light color where there is no shadow
        glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );
        glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
        
        light->_color.set();
        glBegin( GL_QUADS );
        glVertex2f( 0, 0 );
        glVertex2f( 0, size.y );
        glVertex2f( size.x, size.y );
        glVertex2f( size.x, 0 );
        glEnd();
        
        ++litr;
    }
    
    glEnable( GL_TEXTURE_2D );
    //copy out our light texture
    glBindTexture( GL_TEXTURE_2D, lightTexture );
    glCopyTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, 0, 0, size.x, size.y, 0);
    glDisable( GL_TEXTURE_2D );

//draw scene

...

    glDisable( GL_STENCIL_TEST );
    
    glPushMatrix();
    glEnable( GL_TEXTURE_2D );
    glBindTexture( GL_TEXTURE_2D, lightTexture );
    glTranslatef( 0, 0, 10 );
    glColor4f( 1, 1, 1, 1 );
    glBegin( GL_QUADS );
    glTexCoord2f( 0, 0 );    glVertex2f( 0, 0 );
    glTexCoord2f( 0, 1 );    glVertex2f( 0, size.y );
    glTexCoord2f( 1, 1 );    glVertex2f( size.x, size.y );
    glTexCoord2f( 1, 0 );    glVertex2f( size.x, 0 );
    glEnd();
    glDisable( GL_TEXTURE_2D );
    glPopMatrix();
    
    glEnable( GL_STENCIL_TEST );
    
    SDL_GL_SwapBuffers();
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #6
TEXTURE_2D textures must be power-of-two*?

check your OpenGL errors Rasp

* actually, it's not quite that simple these days... just ask for the full rant Wink
Quote this message in a reply
Sage
Posts: 1,066
Joined: 2004.07
Post: #7
OneSadCookie Wrote:TEXTURE_2D textures must be power-of-two*?

check your OpenGL errors Rasp

* actually, it's not quite that simple these days... just ask for the full rant Wink

Ah crap. Forgot. Is there any way to do non-power of 2 through extensions or anything? And I'd read the whole rant if you typed it Smile
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #8
GL_TEXTURE_RECTANGLE_EXT or one of its cousins.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Sage
Posts: 1,066
Joined: 2004.07
Post: #9
Skorche Wrote:GL_TEXTURE_RECTANGLE_EXT or one of its cousins.

Thanks. Now at least I have a yellow box Smile.

So I think I'm having some error in my code for rendering the light. Right now I only have one light, but I don't see any shadows at all (just one big yellow box). Anything seem out of the ordinary?

Code:
glEnable( GL_STENCIL_TEST );
    
    //clear the color buffer
    glClear( GL_COLOR_BUFFER_BIT );
    
    vector< shared_ptr< Light > >::iterator litr = lights.begin(), lend = lights.end();
    vector< Shape >::iterator itr, end;
    while( litr != lend ) {
        shared_ptr< Light > light = ( *litr );
        
        //clear the stencil buffer
        glClear( GL_STENCIL_BUFFER_BIT );
        
        //turn off the color and depth buffers
        glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
        glDepthMask( GL_FALSE );
        
        //render the shadows to the stencil buffer
        glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFF );
        glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );
        
        itr = shapes.begin(), end = shapes.end();
        
        glColor4f( 0, 0, 0, 1 );
        while( itr != end ) {
            Shape shape = ( *itr );
            
            if( !shape.isPointInShape( light->_pos ) )
                renderShadow( shape, light->_pos );
            
            ++itr;
        }
        
        //turn on the color and depth buffers
        glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );
        glDepthMask( GL_TRUE );
        
        //render our light color where there is no shadow
        glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );
        glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
        
        light->_color.set();
        glBegin( GL_QUADS );
        glVertex2f( 0, 0 );
        glVertex2f( 0, size.y );
        glVertex2f( size.x, size.y );
        glVertex2f( size.x, 0 );
        glEnd();
        
        ++litr;
    }
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #10
In the beginning, there was TEXTURE_2D. It has texture coordinates that run 0..1,0..1 ("normalized"), allows a variety of wrapping modes (most usefully REPEAT and CLAMP_TO_EDGE_SGIS), and allows mipmapping.

Around the time of the GeForce 2, NVidia's hardware started supporting non-power-of-two textures, in a limited kind of way. They made an extension, NV_texture_rectangle, to expose this functionality. NV_texture_rectangle introduced a new texture target (TEXTURE_RECTANGLE_NV), disallowed certain wrap modes (notably REPEAT), disallowed mipmapping, and used pixel coordinates rather than normalized coordinates, because that's what their hardware does internally.

Roll forward a few years, and in the 10.1 timeframe Apple's looking for a good way to upload things like movies to the video card. NV_texture_rectangle looks like a good fit for the job, but they need something for ATI hardware as well. It turns out that ATI Radeon cards actually do support the same functionality, so Apple rebrands the extension EXT_texture_rectangle, and implements it for both NVidia and ATI hardware. So far so good.

Roll forward a few years, and NVidia releases the GeForce 6 series of GPUs. These support unrestricted non-power-of-two texturing, and so a new extension is called for. ARB_texture_non_power_of_two is very simple conceptually, it says "it's not an error for your TEXTURE_2D texture to have non-power-of-two dimensions any more". Note that texture coordinates are still normalized, all wrap modes are still supported, and most importantly, mipmapping is still allowed.

Shortly after that, the OpenGL ARB were specifying OpenGL 2.0, and ARB_texture_non_power_of_two caught their eye as a useful kind of thing to add, seeing as how it removes a lot of "unnecessary" restrictions. Add it they did, very much against ATI's wishes -- whose hardware didn't support it at all.

Now, here's where things get interesting -- ATI can't very well sit back and say "well, um, we don't support OpenGL 2.0" whilst NVidia's out there trumpeting their OpenGL 2.0 support from the rooftops... but they can't get around the fact that their hardware doesn't support mipmapping (or REPEAT, I think) on non-power-of-two textures.

So they do something a little bit sneaky... they implement a software fallback for the parts they don't support, and claim "OpenGL 2.0 support". However, they don't export the ARB_texture_non_power_of_two extension, to indicate that one shouldn't expect this to run in hardware. In my mind, this is all fair enough -- there are plenty of parts of OpenGL 1.2 even that *no* hardware supports, and this isn't really any different.

Shortly after, somebody realizes that it'd be useful to use the old EXT_texture_rectangle textures in GLSL, and the new ARB_texture_rectangle extension is born, which is just EXT_texture_rectangle + some new bits in GLSL to allow access to these textures.

As if all this isn't bad enough, in 10.4.8, Apple comes to implementing OpenGL 2.0 (at last!). For whatever peculiar and unhelpful reasons of their own, they choose to export the ARB_texture_non_power_of_two extension on all hardware that exports OpenGL 2.0, including all the ATI hardware that doesn't really support the extension. That means that the extension check on Mac OS X is *not* a useful way to determine hardware support for the feature, and we must fall back to mac-specific ways to detect whether rendering is hardware-accelerated.

All-in-all, a right mess:

NVidia on Windows/Linux:
* supports NV_texture_rectangle
* supports ARB_texture_rectangle
* supports ARB_texture_non_power_of_two
* supports OpenGL 2.0

ATI on Windows/Linux:
* supports EXT_texture_rectangle
* supports OpenGL 2.0 (in software)

NVidia on Mac OS X:
* supports EXT_texture_rectangle
* supports ARB_texture_rectangle
* supports ARB_texture_non_power_of_two
* supports OpenGL 2.0

ATI on Mac OS X:
* supports EXT_texture_rectangle
* supports ARB_texture_rectangle
* supports ARB_texture_non_power_of_two (in software)
* supports OpenGL 2.0 (in software)

Notice that there's no single cross-platform way to use rectangle textures (best bet is to try ARB_texture_rectangle [everywhere except ATI/PC] and fall back to EXT_texture_rectangle).

Notice that there's no single cross-platform way to use non-power-of-two textures with GLSL (best bet is to try ARB_texture_rectangle [everywhere except ATI/PC] and fall back to OpenGL 2.0)

Notice that there's no single cross-platform way to check if ARB_texture_non_power_of_two textures are hardware-accelerated (best bet is to test for ARB_texture_non_power_of_two [everywhere except Mac OS X], and then on Mac OS X, test if it's actually accelerated using the Apple-specific APIs)

If that's not enough to make you cry, you're stronger than I am.
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #11
Your stencil function looks the same as mine. Are you sure you aren't missing some other important state that you set?

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Sage
Posts: 1,066
Joined: 2004.07
Post: #12
I never would've guessed it was that complicated.

I wound up just using a power of two texture larger than my screen and using just the portion necessary. My screen is 800x600 and the texture is 1024x1024 so I'm not wasting too much memory. At least it works now...
Quote this message in a reply
Sage
Posts: 1,066
Joined: 2004.07
Post: #13
Skorche Wrote:Your stencil function looks the same as mine. Are you sure you aren't missing some other important state that you set?

I'm not sure, but I do have it working now. However, I'm having trouble with using light color now. I can't figure out a good way to get the two light's colors to blend over each other and each other's shadows without blending with their own shadows. Here's my total rendering code as is. Any ideas?

Code:
void render() {
    glLoadIdentity();
    
    vec3i size = Application::application()->size();
    
    glEnable( GL_STENCIL_TEST );
    glDisable( GL_TEXTURE_2D );
    
    //clear the color buffer
    glClearColor( 0, 0, 0, 0 );
    glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );
    
    vector< shared_ptr< Light > >::iterator litr = lights.begin(), lend = lights.end();
    vector< Shape >::iterator itr, end = shapes.end();
    int layer = 0;
    while( litr != lend ) {
        shared_ptr< Light > light = ( *litr );
        
        //clear the stencil buffer
        glClear( GL_STENCIL_BUFFER_BIT );
        
        //turn off the color and depth buffers
        glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
        glDepthMask( GL_FALSE );
        
        //render the shadows to the stencil buffer
        glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFF );
        glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );
        
        itr = shapes.begin();
        
        glColor4f( 0, 0, 0, 1 );
        while( itr != end ) {
            Shape shape = ( *itr );
            
            if( !shape.isPointInShape( light->_pos ) )
                renderShadow( shape, light->_pos );
            
            ++itr;
        }
        
        //turn on the color and depth buffers
        glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );
        glDepthMask( GL_TRUE );
        
        //render the shadows themselves
        glStencilFunc( GL_EQUAL, 1, 0xFFFFFFFF );
        glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
        
        glColor4f( 0, 0, 0, .9f );
        glBegin( GL_QUADS );
        glVertex3f( 0, 0, layer );
        glVertex3f( 0, size.y, layer );
        glVertex3f( size.x, size.y, layer );
        glVertex3f( size.x, 0, layer );
        glEnd();
        
        //render our light color where there is no shadow
        glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );
        light->_color.set();
        glBegin( GL_QUADS );
        glVertex3f( 0, 0, layer );
        glVertex3f( 0, size.y, layer );
        glVertex3f( size.x, size.y, layer );
        glVertex3f( size.x, 0, layer );
        glEnd();
        
        ++litr;
        ++layer;
    }
    
    //copy out our light texture
    glEnable( GL_TEXTURE_2D );
    glBindTexture( GL_TEXTURE_2D, lightTexture );
    glCopyTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, 0, 0, 1024, 1024, 0);
    glDisable( GL_TEXTURE_2D );
    
    //clear the color and depth buffers
    glClearColor( 1, 1, 1, 1 );
    glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );
    
//    //render our "dynamic shapes" which only
//    //appear outside of shadows
    glDisable( GL_STENCIL_TEST );
//    glStencilFunc( GL_EQUAL, 0, 0xFFFFFFFF );
    glColor4f( 1, 1, 1, 1 );
    glEnable( GL_TEXTURE_2D );
    glBindTexture( GL_TEXTURE_2D, box->id );
    glBegin( GL_QUADS );
    glTexCoord2f( 0, 0 );    glVertex2f( 400, 300 );
    glTexCoord2f( 0, 1 );    glVertex2f( 400, 350 );
    glTexCoord2f( 1, 1 );    glVertex2f( 450, 350 );
    glTexCoord2f( 1, 0 );    glVertex2f( 450, 300 );
    glEnd();
    glDisable( GL_TEXTURE_2D );
    
    
//    //draw our background image
    glPushMatrix();
    glEnable( GL_TEXTURE_2D );
    glBindTexture( GL_TEXTURE_2D, ground->id );
    glTranslatef( 0, 0, -2 );
    glColor4f( 1, 1, 1, 1 );
    glBegin( GL_QUADS );
    glTexCoord2f( 0, 0 );    glVertex2f( 0, 0 );
    glTexCoord2f( 0, 1 );    glVertex2f( 0, size.y );
    glTexCoord2f( 1, 1 );    glVertex2f( size.x, size.y );
    glTexCoord2f( 1, 0 );    glVertex2f( size.x, 0 );
    glEnd();
    glDisable( GL_TEXTURE_2D );
    glPopMatrix();
    
    //render static geometry that is drawn whether in-
    //or outside of a shadow
    itr = shapes.begin();
    glColor4f( .5f, .3f, 1.0f, 1.0f );
    while( itr != end ) {
        Shape shape = ( ( *itr ) );
        
        shape.draw();
        
        ++itr;
    }
    
    glPushMatrix();
    glTranslatef( 0, 0, 10 );
    glColor4f( 1, 1, 1, 1 );
    litr = lights.begin();
    glPointSize( 5 );
    glBegin( GL_POINTS );
    while( litr != lend ) {
        glVertex2f( ( *litr )->_pos.x, ( *litr )->_pos.y );
        ++litr;
    }
    glEnd();
    glPopMatrix();
    
    glDisable( GL_STENCIL_TEST );    
    glPushMatrix();
    glEnable( GL_TEXTURE_2D );
    glBindTexture( GL_TEXTURE_2D, lightTexture );
    glTranslatef( 0, 0, 10 );
    glColor4f( 1, 1, 1, 1 );
    glBegin( GL_QUADS );
    
    glTexCoord2f( 0, ( float )size.y / 1024.0f );
    glVertex2f( 0, 0 );
    
    glTexCoord2f( 0, 0 );
    glVertex2f( 0, size.y );
    
    glTexCoord2f( ( float )size.x / 1024.0f, 0 );
    glVertex2f( size.x, size.y );
    
    glTexCoord2f( ( float )size.x / 1024.0f, ( float )size.y / 1024.0f );
    glVertex2f( size.x, 0 );
    
    glEnd();
    glDisable( GL_TEXTURE_2D );
    glPopMatrix();
    
    SDL_GL_SwapBuffers();
}

Edit: Here's a picture of the problem. Though this is only using white lights, it shows the problem. The one light gets rendered first and the shadows are cast. Then the next light lightens those shadows a bit and casts shadows of it's own. Unfortunately, these shadows then are not lit by the first light. Any ideas?

[Image: shadow_screen3.png]
Quote this message in a reply
Sage
Posts: 1,482
Joined: 2002.09
Post: #14
Are you alpha blending the light values? Just add them.

Scott Lembcke - Howling Moon Software
Author of Chipmunk Physics - A fast and simple rigid body physics library in C.
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #15
Code:
render scene to depth, and ambient light to color buffer
for each light
    clear the stencil buffer
    render the shadow volumes to the stencil buffer
    render the scene to the color buffer, lit by the light, with
        * depth test set to equal; and
        * depth writes turned off; and
        * additive blending enabled
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  Vector shapes Miglu 16 8,471 Sep 22, 2010 07:52 AM
Last Post: Miglu
  Window drop shadows NelsonMandella 9 5,404 Mar 21, 2010 02:34 PM
Last Post: NelsonMandella
  Making 2D Stencil Shadows Soft metacollin 16 14,947 Jul 22, 2009 01:59 PM
Last Post: NelsonMandella
  softening 2d shadows NelsonMandella 5 4,533 May 19, 2009 04:19 AM
Last Post: Najdorf
  Breaking down a concave mesh into convex pieces Willem 5 4,665 Aug 10, 2008 05:49 AM
Last Post: Willem