Stencil buffer emulation

Sage
Posts: 1,232
Joined: 2002.10
Post: #1
I'm trying to emulate the stencil buffer of some console hardware with GL. The desired behavior is very simple:
A 1-bit flag controls the stencil value written to the framebuffer (masked or not masked.)
A 1-bit flag controls the stencil test (do not write incoming fragment if framebuffer mask bit is set.)

This behavior is a bit different than GL's stenciling:
Disabling the stencil test does not disable writing the stencil bit (it is always written, unless the stencil test fails.)
GL shares one reference for both the stencil value to write and the value to test against.

Those two things are making it hard for me to map this behavior to GL:
Code:
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_REPLACE, GL_REPLACE);
if (bCheckMask) glStencilFunc(GL_LESS, bTheMask, 1);
else glStencilFunc(GL_ALWAYS, bTheMask, 1);
Here, STENCIL_TEST is always enabled so the bit is always written along with the color value, unless the test fails, per StencilOp.

The problem is that the bit to write is shared with the reference value to test-- if bCheckMask is false, I correctly write out the mask bit, and always pass the test.
But if bCheckMask is true, I want to write out the mask bit, but only pass if the framebuffer bit is 0. There's no GL enumeration to do that!

Or, can anyone see a tricky way to accomplish this?
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #2
so there are four cases,

dont-write-or-test
glDisable(GL_STENCIL_TEST);

dont-write-but-do-test
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

write-but-don't-test
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

write-and-test
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_REPLACE, GL_REPLACE);
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #3
Err... like I said, disabling the test does not disable the writes. So there are four cases:

write zero
write zero if test passed
write one
write one if test passed
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #4
oh...

icky Smile
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #5
so, have I understood yet?

glEnable(GL_STENCIL_TEST);

write zero
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_DECR, GL_DECR, GL_DECR);

write zero if test passed
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_DECR, GL_DECR);

write one
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

write one if test passed <-- isn't this a no-op?
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_REPLACE, GL_REPLACE);
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #6
A mask bit of 1 signifies 'do not write', but otherwise yes, breaking it apart like that allows the functionality I wanted. Thanks!

Unfortuntely, it turns out that for textured primitives, there is an additional interaction with the high bit of the (paletted) texel color-- if it is set (opaque) the mask bit is also set. So, the stencil buffer isn't going to work out here after all... Sad I'll have to come up with something else.
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #7
Well, I sort of have it working now, with a combination of the stencil buffer and destination alpha.
Take a look at Silent Hill... no more white boxes around the character/trees...

Unfortunately this is currently using EXT_blend_func_separate... needs more head scratching to get around that...
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  SLOW performance drawing multiple polygons per frame with stencil buffer clam61 7 3,916 Apr 27, 2013 11:53 AM
Last Post: clam61
  Masking without a stencil buffer? Bachus 12 13,427 Sep 2, 2010 02:42 PM
Last Post: Skorche
  Making 2D Stencil Shadows Soft metacollin 16 15,014 Jul 22, 2009 01:59 PM
Last Post: NelsonMandella
  Stencil shadows meant to look like this? ia3n_g 2 3,697 Nov 23, 2006 06:57 PM
Last Post: akb825
  Stencil buffers in an FBO... trouble TomorrowPlusX 17 7,617 Mar 23, 2006 08:19 PM
Last Post: kberg