I'm trying to emulate the stencil buffer of some console hardware with GL. The desired behavior is very simple:
A 1-bit flag controls the stencil value written to the framebuffer (masked or not masked.)
A 1-bit flag controls the stencil test (do not write incoming fragment if framebuffer mask bit is set.)

This behavior is a bit different than GL's stenciling:
Disabling the stencil test does not disable writing the stencil bit (it is always written, unless the stencil test fails.)
GL shares one reference for both the stencil value to write and the value to test against.

Those two things are making it hard for me to map this behavior to GL:
Here, STENCIL_TEST is always enabled so the bit is always written along with the color value, unless the test fails, per StencilOp.

The problem is that the bit to write is shared with the reference value to test-- if bCheckMask is false, I correctly write out the mask bit, and always pass the test.
But if bCheckMask is true, I want to write out the mask bit, but only pass if the framebuffer bit is 0. There's no GL enumeration to do that!

Or, can anyone see a tricky way to accomplish this?

so there are four cases,

dont-write-or-test
glDisable(GL_STENCIL_TEST);

dont-write-but-do-test
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

write-but-don't-test
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

write-and-test
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_REPLACE, GL_REPLACE);

Err... like I said, disabling the test does not disable the writes. So there are four cases:

write zero
write zero if test passed
write one
write one if test passed

oh...

icky

so, have I understood yet?

glEnable(GL_STENCIL_TEST);

write zero
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_DECR, GL_DECR, GL_DECR);

write zero if test passed
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_DECR, GL_DECR);

write one
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

write one if test passed <-- isn't this a no-op?
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_REPLACE, GL_REPLACE);

A mask bit of 1 signifies 'do not write', but otherwise yes, breaking it apart like that allows the functionality I wanted. Thanks!

Unfortuntely, it turns out that for textured primitives, there is an additional interaction with the high bit of the (paletted) texel color-- if it is set (opaque) the mask bit is also set. So, the stencil buffer isn't going to work out here after all... I'll have to come up with something else.

Well, I sort of have it working now, with a combination of the stencil buffer and destination alpha.
Take a look at Silent Hill... no more white boxes around the character/trees...

Unfortunately this is currently using EXT_blend_func_separate... needs more head scratching to get around that...