iDevGames Forums
Making 2D Stencil Shadows Soft - Printable Version

+- iDevGames Forums (
+-- Forum: Development Zone (/forum-3.html)
+--- Forum: Graphics & Audio Programming (/forum-9.html)
+--- Thread: Making 2D Stencil Shadows Soft (/thread-997.html)

Pages: 1 2

Making 2D Stencil Shadows Soft - metacollin - Jul 16, 2009 11:24 PM

First a screenshot, so you get an idea of what I'm doing:


To do this, I am using the lighting algorithm as detailed out towards the bottom of my other thread 2D Dynamic Lighting in OpenGL! Sort of... but instead of just drawing each light, I stencil out a shadow area, draw the light, clear the stencil buffer, and repeat.

The problem is I have absolutely no idea how to make these shadows soft. I mean, let's face it, those shadows look pretty ugly in that screenshot.

I successfully implemented the texture penumbra algorithm as detailed in this article:, however it is more or less useless to me. There isn't any way to blend a textured triangle over or under the light, using color or alpha, to make the light fade out without messing up the blending with other lights. I also can't think of anyway to do this without using the stencil buffer to block out parts of the light without effecting the other lights.

I'm wondering if something can be done around the edges of the stencil volume using a GLSL shader program, which I am willing to try.

Any ideas? I don't necessarily need super ultra realistic pimpumbras or anything, just some softness to the shadows so they look nicer. If only the stencil buffer had an alpha (% of color that gets drawn or something)....

Making 2D Stencil Shadows Soft - metacollin - Jul 17, 2009 04:16 AM

Nevermind, I think I've got I re-read my post, I had a pretty hard facepalm moment after reading the last line. I realized I could just use a combination of the alpha and stencil buffers.

Will post results and algorithm soon.

Making 2D Stencil Shadows Soft - Skorche - Jul 17, 2009 06:16 AM

Also, you should try putting a texture on your background. IMO hard shadows look like crap when drawn over a flat color, but look fine when drawn over a textured surface.

[Image: screenshot_071709_081447.png]
[Image: screenshot_071709_081326.png]

That's even with bland boring linear falloff and additive blending. Nothing fancy. Putting a texture underneath it gives it a lot of depth and makes it look much more natural for whatever reason.

Though if you've already got soft shadows working for you go with it I guess. Though I don't see why you need both the alpha and stencil buffer. The article I linked only uses the alpha. I personally was put off by some of the weird artifacts of how soft shadows were implemented in that paper. The fin clipping seemed like it used up a fair amount of CPU and required performing some sort of culling unless you had tiny scenes. You can't have shadow casting geometry smaller than one of your lights, and you can't let lights get too close to the shadow casting geometry unless you project the shadows to infinity. (something you can do actually unless you use the triangular texture for the shadow fin like in the article.)

Making 2D Stencil Shadows Soft - Skorche - Jul 17, 2009 07:46 AM

Oh, one last note. Gish uses hard shadows and they look dandy. That was originally why I made my own 2D shadow code. At the time, the physics and shadows were the coolest things I had ever seen in a 2D game.

Gish must use an implementation similar to what is described in that link that I sent you. That was how I originally figured out how they did the shadows in Gish, when a shadow casting segment was very close to a light, it wouldn't cast the shadow very far. I figured they must simply be drawing quads for each segment by projecting it away from the light by a certain amount. I fixed that by using homogenous coordinates to project the shadows to infinity. That can also be done without needing to calculate vertexes on the CPU or using immediate mode. I guess I'm sort of partial to my own non-soft shadow implementation for that reason.

If you get soft shadows working nicely I'd love to see that too. Smile

Making 2D Stencil Shadows Soft - metacollin - Jul 17, 2009 02:57 PM

So after much messing around, I finally settled on a solution I like. I agree, the penumbras from that article seem like a huge waste, have some pretty serious limitations (at least for a platformer like mine) and honestly don't look particularly better than hard shadows in my opinion.

I went for a cheap, easy, but effective 'compromise'. I saw 'compromise' because it's really unrealistic, but who cares, it looks nice. The everything in my game world is made up of infinitely thin colored squares, I don't think they'll mind. Wink

At the edges of the shadow, I simply tack on a .15 rad wedge with the penumbra texture applied. I do need a better texture, there is artifacts and such on occasion, but it looks pretty good. I'm hoping getting a better texture will help out.

Soft Shadows

As for drawing the shadow rays, I just clamp the value to the size of the light...the ones I've been using are sometimes 512x512 quads, sometimes 256x256. This may or may not be a stupid way of doing it, I have no idea.

And as for why I'm using stencil buffers, it's to avoid self-shadowing. Since I'm using a bunch of tiles, I initially treat each one like a convex hull. Then I go through and merge all the horizontally adjacent tiles into a single hull, then repeat for the vertically adjacent tiles (except for one's already gobbled up by a horizontal strip). After that, try to merge any adjacent strips of the same length into rectangles. This cuts down a lot of edges, but there will always be some orphaned tiles and other hulls with boundary lines within other shadows.

Though I think I can still get it to work only using alphas, now that I think about it.

Well, if anyone has any more optimization suggestions/make stuff look better ideas, feel free to share, but I'm about ready to be finished with the lighting part of my game. (Finally!).

Oh, and good point about backgrounds, Skorche. I'm eventually going to have scrolling tiled backgrounds etc, but content generation has always been my problem. Shadows will never look good on a pasty white background.

Making 2D Stencil Shadows Soft - Skorche - Jul 17, 2009 03:32 PM

As for the self shadowing thing, I personally find it looks a bit odd to have shadow casting objects also receive them. It looks sort of funny when you have a pile of boxes, but only the outer piles of boxes are lit. It looks even funnier when you have cracks of light that penetrate into the pile.

What I did in my prototype game a while back was to have 4 layers. A background (the back wall), a layer of shadowed sprites that don't cast shadows (players enemies etc), a layer of non-shadowed, shadow casting sprites (boxes and such), and finally the foreground layer (like the tiles you are drawing in your game). For the second sprite layer and the foreground layer I used a second ambient light value that was generally brighter than the ambient value used in the lightmap. This looked nice because things like players and enemies could duck in and out of the shadows, but larger objects like the level itself and boxes always had consistent lighting. You could still do fun/interesting effects with the foreground ambient value for things like lightning or other effects.

Also, if you want to avoid using the triangle wedge texture fin like that article used, you can simply use a 1D gradient texture (or 1xX 2D texture) and supply homogenous coordinates for your texture coordinates.

(0, 0, 0, 1) for the left projected away tex coord
(1, 0, 0, 1) for the right projected tex coord
(0, 0, 0, 0) for the near tex coord

Because the scale coordinate is zero on the last tex coord, it will do some fancy filtering to make it look right. I got that to work at one point, though I'm not 100% certain that was how. Also, you will need to use mipmapped texture for the gradient.

Making 2D Stencil Shadows Soft - metacollin - Jul 17, 2009 04:14 PM

Skorche Wrote:As for the self shadowing thing, I personally find it looks a bit odd to have shadow casting objects also receive them. It looks sort of funny when you have a pile of boxes, but only the outer piles of boxes are lit. It looks even funnier when you have cracks of light that penetrate into the pile.

That's EXACTLY the problem I was having! Based on your posts and shots (looks great, btw), it sounds like you've pretty much already written what I'm working on, hehe. Yeah, my solution (it's sort of hard to tell from the screenshot) was to reincorporate the alphas of the tiles when mixing the light with it, so the shadows don't effect them, but they still glow/mix with the lighting around them.

I'm going to try this 1D texture thing...I've got to learn about homogenous coordinates first, I've never heard of them. It sounds like it will look much nicer and be a lot easier and cheaper as well.

Making 2D Stencil Shadows Soft - metacollin - Jul 17, 2009 06:43 PM

I also noticed another bug that I'm not sure how to fix, but going to mull it over for a while before using a 1D texture. When I change from one back-facing edge (one casting a shadow) to another, the penumbra cone 'shifts' suddenly, creating a jumpiness to the shadows sometimes. I believe this was mentioned on the 2D shadow tutorial, but his solution doesn't really apply to my situation.

Making 2D Stencil Shadows Soft - Skorche - Jul 19, 2009 11:28 PM

I would expect that your problem is the same really. You need to split your shadow fin across all edges that lie within the wedge.

Making 2D Stencil Shadows Soft - metacollin - Jul 20, 2009 11:55 PM

Ok, I've given up on appending wedges to the edge of the shadows, I simply can't get it to work properly (avoid popping).

I'm still pretty fuzzy on the whole homogeneous coordinate thing. All I was really able to find was some basic information, and that it appears to be the same system OpenGL represents all its matrices with. I still have no idea how to project a shadow into infinity, but I like moving it to the GPU. Even if it's on the GPU, I can probably get soft shadows to work using shaders.

As for using a 1D gradient texture, I'm also not sure how to use this to avoid popping. By using homogeneous texture coordinates, do you mean s, t, r, and q? I thought that was 3D textures only. And would I texture the shadow wedge itself? Anyway, as near as I can tell, any solution that involves appending/removing part of the shadow wedge will cause the 'popping' effect.

Yeah, I don't really know much about this higher-level geometry stuff, and you've basically been walking me through this the whole time, but any more help you can give would be appreciated =).

Making 2D Stencil Shadows Soft - NelsonMandella - Jul 21, 2009 12:42 AM

Metacollin, your screens and videos look fantastic, you've made incredible progress it would appear. I myself am working on a 2d game and struggled for some time over the problem of how to implement soft shadows with penumbras, finally settling with ray tracing the shadows, or really just casting a ray from every subdivided tile(that is within the sphere of a light) to every light source, checking at regular intervals along the ray for collisions with shadow casters. You could do something like this in your game.

Here's a video showing off the shadows in my game(shadows aren't totally soft in this build)

Making 2D Stencil Shadows Soft - Skorche - Jul 21, 2009 07:23 AM

That looks amazingly sweet in isometric space. I've been thinking about trying my own shadowing technique and applying it to some semi-2D stuff like that. Soft shadows definitely look a lot better in that case though. Keep up the good work.

Metalcoin: PM me with your contact information and I'll email you the source for that GPUShadow program. I don't want to just post it because I think it does some clever stuff that I haven't seen done before on the net. For the moment I want to be able to use that as an advantage until we've released a game or two that uses it.

Making 2D Stencil Shadows Soft - metacollin - Jul 21, 2009 01:21 PM

Awesome vid, Nelson! Those lighting and shadow effects look amazing, regardless of the blockiness of the edges. Is it going to be some sort of dungeon crawler? Any ETA? =)

Anyway, I spent last night rewriting my entire shadow and lighting code. Since I wrote it in a really piecemeal way, it was pretty awful. I managed to put in a fair number of optimizations. For one, I no longer use a wedge of triangles based on the shapes geometry (slow) and just use 1 quad for the whole shadow, using the first and last vertices of the rear-facing part , speeding up pretty much everything, especially the edge-finding algorithm. Reduced my entire render time from 4ms to under 1ms or less (can't be more accurate than that, using SDL). There are still a few dirty shadows that are cast into an already shadowed area, but I now use alphas only for the shadows. If I alpha blend the (at this point, probably generated by a shader or something) penumbra first, then the umbra, this won't matter.

At this point, I feel fairly comfortable implementing something as crazy/needlessly complex as a shader for soft shadows, and make it an option the user can turn on and off. We'll see what happens, edumecating myself on shaders right now.

Making 2D Stencil Shadows Soft - Skorche - Jul 21, 2009 05:57 PM

Wee. Finally made a video of the iPhone engine we have been working on for the past week. Got it up to 55+ fps on a 1st gen iPod touch with 5 lights on the screen. I'm pretty happy with that.

Making 2D Stencil Shadows Soft - monteboyd - Jul 22, 2009 01:31 AM

I gotta say (and I hope this isn't too OT) but the work being presented in this thread is just amazing. I love 2D games and the stuff you guys are doing is showing just how amazing it can look when you use some modern techniques but still in the 2D world.

Skorche - 55+ fps on a first gen iPod Touch is a real achievement - can't wait to see the game that comes from it.

I think all of these would be great in the screenshots gallery Carlos was suggesting. In fact a gallery with video as well would really rock. I'll pop over to that thread now and suggest it.