iDevGames Forums
Cubemap lookup with normalmap "bumps" - Printable Version

+- iDevGames Forums (http://www.idevgames.com/forums)
+-- Forum: Development Zone (/forum-3.html)
+--- Forum: Graphics & Audio Programming (/forum-9.html)
+--- Thread: Cubemap lookup with normalmap "bumps" (/thread-1932.html)

Pages: 1 2


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 10, 2009 06:04 PM

I've written a simple image-based-lighting shader using cubemaps ( two, one gaussian blurred for "diffuse" and a sharp one for specular ) and it looks great. So for my next step, I'm adding normalmaps to perturb the cubemap lookup.

The approach I'm taking is simple: The vertex shader computes the tangent-space matrix and passes it as three vec3 to the fragment shader. The fragment shader then computes the normal in world space by multiplying the normalmap normal by the inverse of the tangent space matrix to bring it to the model's coordinate space, and then multiplies that normal by the model's model matrix to bring it to world coordinates. I can then use that to lookup into the cubemap. Simple enough, but it doesn't actually work Rasp

I've tested it by rendering the world normal as the output color, and as you'll see in the attached screenshots, it's junk.

Here are correct normals, what I should be seeing:
[Image: ibltest-2009-01-10-12.png]

And here I am using a flat normalmap. It should look the same but doesn't:
[Image: ibltest-2009-01-10-13.png]

Now, I know the tangents I'm using to compute tangent space are good, because I've tested my bump lighting with a flat normalmap ( vec3( 128,128,255 )) as well as a bumpy normalmap and the lighting is visibly correct. If my tangents were wrong, my bump lighting would be wrong. Ergo, my tangents are good.

So I have to assume my math is incorrect, or I've written some subtle bug in an otherwise valid algorithm. First, the relevant bits of my vertex shader:

Code:
uniform mat4 ModelMatrix; // the model transform

varying vec3 IBLWorldPosition;
varying vec3 IBLTangentSpace_T;
varying vec3 IBLTangentSpace_B;
varying vec3 IBLTangentSpace_N;

// shader
IBLWorldPosition = (ModelMatrix * gl_Vertex).xyz;

// create a matrix that will convert from model space to tangent space
IBLTangentSpace_N = normalize(gl_Normal);
IBLTangentSpace_T = normalize(gl_SecondaryColor.xyz); // Tangent stored in gl_SecondaryColor
IBLTangentSpace_B = cross(IBLTangentSpace_N, IBLTangentSpace_T);

Now, the relevant bits of my fragment shader:
Code:
uniform sampler2D NormalMap;

uniform mat4 ModelMatrix;
uniform vec3 CameraPosition;

varying vec3 IBLWorldPosition;
varying vec3 IBLTangentSpace_N;
varying vec3 IBLTangentSpace_T;
varying vec3 IBLTangentSpace_B;

// shader
vec3 normalMap = texture2D(NormalMap, gl_TexCoord[0].st).xyz;
vec3 bump = normalize( normalMap * 2.0 - 1.0);
    
// assemble the model->tangentspace matrix
mat3 TM = mat3( normalize( IBLTangentSpace_T ),
                normalize( IBLTangentSpace_B ),
                normalize( IBLTangentSpace_N ));
        
// create a rotation matrix from the model matrix
mat3 RM = mat3( ModelMatrix[0].xyz,
                ModelMatrix[1].xyz,
                ModelMatrix[2].xyz );
                    
// I read that vec3*mat3 is equivalent to transpose( mat3 ) * vec3, so we're
// bringing the bumpmap normal into modelspace by multiplying it by the inverse
// of the tangent space matrix, since transpose of an orthonormal matrix is the inverse
// and then bringing that normal into world space by multiplying it by the model rotation
vec3 worldNormal = normalize( RM * ( bump * TM ) );
vec3 reflectDir = normalize( reflect( normalize( IBLWorldPosition - CameraPosition ), worldNormal));

Can anybody see anything boneheaded here? Perhaps there's a bug, perhaps my algorithm is all wrong. This is driving me crazy!

Two notes:

1) I'm certain my tangents are good. My lambertian lighting with normalmaps looks great.

2) The vec3*mat3 = transpose( mat3 ) * vec3 code concerned me, so I did write a transpose( mat3 ) function to verify that the assertion is actually true, and it is. It produced the same ( albeit junk ) results.


Cubemap lookup with normalmap "bumps" - DoG - Jan 11, 2009 04:08 AM

Well, vec3*mat3 doesn't work mathematically, unless the vec3 is transposed as well, but, that said, I don't know what GLSL thinks it should do. I'd stick with transpose( mat3 ) * vec3 and let the shader compiler sort it out. You also seem to haven an awful lot of normalizing going on. It's for example not necessary for the first arg of the reflect() call, if you normalize the result.

The second image looks as if the values are stuck in tangent space, aren't you supposed to multiply the normal by the tangent-space matrix itself, not the inverse?


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 11, 2009 05:37 PM

The matrix I assemble from IBLTangentSpace_[N|T|B] in the vertex shader would bring normals from object space to tangent space, so I need to invert it to bring a bumpmap normal out of tangent space into the object's local space. Then the ModelMatrix will bring that normal to world space.

That's the idea, at least.


Cubemap lookup with normalmap "bumps" - DoG - Jan 12, 2009 02:50 AM

Seems weird, so gl_Normal doesn't actually contain the usual normal? Also, its not multiplied by gl_NormalMatrix?


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 12, 2009 03:56 AM

Well, you were right in the first place. I'm not exactly certain of the math, but the key is to not invert the tangent matrix, at all.

Code:
vec3 worldNormal = RM * TM * bump;

And that did it!

EDIT: Regarding the spurious normalizations. I'd noticed during original development that reflect acted wonky when the inputs weren't normalized... that being said, I took out the normalization in the construction of the tangent matrix, and everything continued to work fine.

Anyway, long story short, IBL looks awesome. I got all worked up when I saw the wolfire blog post about IBL... I had to give it a stab for myself. http://blog.wolfire.com/2008/12/object-lighting-part-1/


Cubemap lookup with normalmap "bumps" - aarku - Jan 12, 2009 12:40 PM

Cool! (Any final donut screenshot to share? Grin ) We used cubemap lighting in some of our games too because it performed a ton better on integrated cards without hardware T&L. A texture lookup performed a lot faster than one directional light and looked better -- you can bake in a zillion lights into one texture lookup. Hooray!

Cheers,
-Jon


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 12, 2009 03:17 PM

Okeydoke. It looks pretty decent -- but I need to tune it. RIght now the diffuse lighting is pretty garish. But I'm just happy the math's working...

Here I've got a 50/50 mix between diffuse and specular ( reflection ):
[Image: ibltest-2009-01-12-02.png]

And a quick movie:
http://shamyl.zakariya.net/screenshots/IBL.mov

I'm treating IBL diffuse/specular as my ambient render pass, with subsequent lit passes additively drawn on top. So it should be rather subtle...


Cubemap lookup with normalmap "bumps" - DoG - Jan 12, 2009 03:44 PM

Looks damn good Smile


Cubemap lookup with normalmap "bumps" - AnotherJake - Jan 12, 2009 06:00 PM

Wow Blink


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 12, 2009 06:29 PM

I just finished writing a correct cubemap gaussian blur algorithm. If anybody's interested, I'd be happy to post it. It blurs cubemaps using 2 1D convolutions ( horizontal & vertical ), and it's able to sample across cubemap faces. Looks great! Fast too -- though I doubt it's fast enough for a realtime blur, like my 2D gaussian is.


Cubemap lookup with normalmap "bumps" - AnotherJake - Jan 12, 2009 08:32 PM

TomorrowPlusX Wrote:If anybody's interested, I'd be happy to post it.

Heck yeah! All your stuff is awesome, and we're always happy when you break these doors open. Grin

Admittedly, I'm busy with iPhone stuff right now (and have been for months), but all your threads are bookmarked for when I get a chance to hit the Mac end again.


Cubemap lookup with normalmap "bumps" - DoG - Jan 13, 2009 02:44 AM

TomorrowPlusX Wrote:I just finished writing a correct cubemap gaussian blur algorithm. If anybody's interested, I'd be happy to post it. It blurs cubemaps using 2 1D convolutions ( horizontal & vertical ), and it's able to sample across cubemap faces. Looks great! Fast too -- though I doubt it's fast enough for a realtime blur, like my 2D gaussian is.

The 1D version should be significantly faster. If k is your kernel size and n the number of pixels, it needs to churn only through O(2 k n) data, instead of O(k^2 n) for the 2D kernel. So for kernel sizes > 2, it should be faster.


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 13, 2009 04:56 AM

DoG Wrote:The 1D version should be significantly faster. If k is your kernel size and n the number of pixels, it needs to churn only through O(2 k n) data, instead of O(k^2 n) for the 2D kernel. So for kernel sizes > 2, it should be faster.

Yup, that's why I implemented it this way. But I realize the issue -- I said:

Quote:though I doubt it's fast enough for a realtime blur, like my 2D gaussian is

I was referring to my gaussian blur implementation for GL_TEXTURE_2D and GL_TEXTURE_RECTANGLE_EXT; that code also uses two 1D passes.


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 13, 2009 04:57 AM

AnotherJake Wrote:Heck yeah! All your stuff is awesome, and we're always happy when you break these doors open. Grin

Admittedly, I'm busy with iPhone stuff right now (and have been for months), but all your threads are bookmarked for when I get a chance to hit the Mac end again.

I was cleaning up my very hacky code and have broken something.LOL

When it's back in order, I'll be certain to post the code. See, I generally write something ugly first, to verify that the algorithm is good. Then I clean it up to keep it maintainable.


Cubemap lookup with normalmap "bumps" - TomorrowPlusX - Jan 13, 2009 08:02 AM

I was able to fix it by rolling back code cleanups I made. Right now, I don't know why the particular cleanups I made fix the problem, but the important part is that it's fixed.

So, I'll post the code below. Note, the code requires my SGF framework to compile. But I assume anybody who would make good use of this already has a texture loading system, already has FBO code, shader loading code, etc etc. So this is more of a helpful hint for others to use to adopt, should they find it useful.

First, the C++ header:
Code:
/*
*  CubeMapBlur.h
*  SRETests
*
*  Created by Shamyl Zakariya on 1/12/09.
*  NO COPYRIGHT: ENJOY
*
*/

#ifndef __SGF_IMAGE_CUBE_MAP_BLUR_H__
#define __SGF_IMAGE_CUBE_MAP_BLUR_H__

#include <SGF/SGF.h>

namespace sgf {

/**
    @brief Blur a cubemap
    @param cubemap The cubemap to blur
    @param radius The gaussian kernel radius
*/

TextureInterfaceRef BlurCubemap( TextureInterfaceRef cubemap, int blurRadius, int blurPasses, int cubemapSize = -1 );

}

#endif

Now, the C++ implementation:
Code:
/*
*  CubeMapBlur.cpp
*  SRETests
*
*  Created by Shamyl Zakariya on 1/12/09.
*  NO COPYRIGHT: ENJOY
*
*/

#include "BlurCubemap.h"

#define PROFILE_CUBEMAP_BLUR 0

namespace sgf {

namespace {

    const vec2 quad_vertices[4] = {
        vec2( 0,0 ),
        vec2( 1,0 ),
        vec2( 1,1 ),
        vec2( 0,1 )
    };
    
    const vec3 unit_cube[8] = {
        normalize( vec3( -1,-1,-1 )),
        normalize( vec3( -1,+1,-1 )),
        normalize( vec3( +1,+1,-1 )),
        normalize( vec3( +1,-1,-1 )),
        normalize( vec3( -1,-1,+1 )),
        normalize( vec3( -1,+1,+1 )),
        normalize( vec3( +1,+1,+1 )),
        normalize( vec3( +1,-1,+1 )),
    };

    void enter_ortho( GLint viewport[4], int cubeSize )
    {
        glGetIntegerv( GL_VIEWPORT, viewport );
            
        glDisable( GL_LIGHTING );
        glDisable( GL_DEPTH_TEST );
        glDisable( GL_CULL_FACE );
        glDisable( GL_BLEND );

        glMatrixMode( GL_PROJECTION );
        glPushMatrix();

        glLoadIdentity();
        glViewport( 0, 0, cubeSize, cubeSize );
        gluOrtho2D( 0, cubeSize, 0, cubeSize );

        glMatrixMode( GL_MODELVIEW );
        glPushMatrix();

        glLoadIdentity();
    }

    void leave_ortho( GLint viewport[4] )
    {
        glPopMatrix();
            
        glMatrixMode( GL_PROJECTION );
        glViewport( viewport[0], viewport[1], viewport[2], viewport[3] );

        glPopMatrix();      
    }
    
    std::vector< vec2 > create_kernel( int size )
    {
        std::vector< vec2 > kernel;
        
        for ( int i = -size; i <= size; i++ )
        {
            int dist = std::abs( i );
            float mag = 1.0f - ( dist / size );
            
            kernel.push_back( vec2( i, mag ));
        }

        //
        // normalize
        //
        
        float sum = 0;
        for ( int i = 0, N = kernel.size(); i < N; i++ )
        {
            sum += kernel[i].y;
        }

        for ( int i = 0, N = kernel.size(); i < N; i++ )
        {
            kernel[i].y /= sum;
        }
                        
        return kernel;
    }
    

    void render_pos_x( int cubemapSize )
    {
        vec3 directions[4] = {
            unit_cube[3],
            unit_cube[2],
            unit_cube[6],
            unit_cube[7]
        };
        
        int rot = 3;        
        for ( int i = 0; i < 4; i++ )
        {
            int index = ( i + rot ) % 4;
            glColor3fv( directions[index] );
            glSecondaryColor3fv( cross( vec3( 0,0,1 ), directions[index] ) );
            glVertex2fv( quad_vertices[i] * cubemapSize );
        }                  
    }

    void render_neg_x( int cubemapSize )
    {
        vec3 directions[4] = {
            unit_cube[1],
            unit_cube[0],
            unit_cube[4],
            unit_cube[5]
        };
        
        int rot = 1;
        for ( int i = 0; i < 4; i++ )
        {
            int index = ( i + rot ) % 4;
            glColor3fv( directions[index] );
            glSecondaryColor3fv( cross( vec3( 0,0,1 ), directions[index] ) );
            glVertex2fv( quad_vertices[i] * cubemapSize );
        }                  
    }

    void render_pos_y( int cubemapSize )
    {
        vec3 directions[4] = {
            unit_cube[0],
            unit_cube[3],
            unit_cube[7],
            unit_cube[4]
        };
        
        for ( int i = 0; i < 4; i++ )
        {
            glColor3fv( directions[i] );
            glSecondaryColor3fv( cross( vec3( 0,0,1 ), directions[i] ) );
            glVertex2fv( quad_vertices[i] * cubemapSize );
        }                  
    }

    void render_neg_y( int cubemapSize )
    {
        vec3 directions[4] = {
            unit_cube[5],
            unit_cube[6],
            unit_cube[2],
            unit_cube[1]
        };
        
        for ( int i = 0; i < 4; i++ )
        {
            glColor3fv( directions[i] );
            glSecondaryColor3fv( cross( vec3( 0,0,1 ), directions[i] ) );
            glVertex2fv( quad_vertices[i] * cubemapSize );
        }                  
    }

    void render_pos_z( int cubemapSize )
    {
        vec3 directions[4] = {
            unit_cube[4],
            unit_cube[7],
            unit_cube[6],
            unit_cube[5]
        };
        
        for ( int i = 0; i < 4; i++ )
        {
            glColor3fv( directions[i] );
            glSecondaryColor3fv( cross( vec3( 0,1,0 ), directions[i] ) );
            glVertex2fv( quad_vertices[i] * cubemapSize );
        }                  
    }

    void render_neg_z( int cubemapSize )
    {
        vec3 directions[4] = {
            unit_cube[3],
            unit_cube[0],
            unit_cube[1],
            unit_cube[2]
        };
        
        for ( int i = 0; i < 4; i++ )
        {
            glColor3fv( directions[i] );
            glSecondaryColor3fv( cross( vec3( 0,1,0 ), directions[i] ) );
            glVertex2fv( quad_vertices[i] * cubemapSize );
        }                  
    }

}

TextureInterfaceRef BlurCubemap( TextureInterfaceRef source, int blurRadius, int blurPasses, int cubemapSize )
{
    #if PROFILE_CUBEMAP_BLUR
        realtime::Stamp stamp;
    #endif

    //
    // Input cleanup;
    //      cubemapSize must be POT
    //      radius must be odd
    //

    if ( blurRadius < 1 ) return source;
    if ( !(blurRadius % 2 )) blurRadius++;

    blurPasses = std::max( blurPasses, 1 );
    
    if ( source->target() != GL_TEXTURE_CUBE_MAP )
    {
        throw LogicException( "BlurCubemap - input texture is not a cubemap!" );
    }
    
    if ( cubemapSize <= 0 )
    {
        cubemapSize = source->width();
    }

    if ( !isPow2( cubemapSize ))
    {
        cubemapSize = nextPowerOf2(cubemapSize);
    }
    
    //
    // Load shaders
    //

    std::string vertexShader = PATH( "Filters/cubemap_blur.vs" ),
                fragmentShader = PATH( "Filters/cubemap_blur.fs" );
                
    glsl::Shader::Definitions hDefs, vDefs;
    
    hDefs[ "HORIZONTAL" ] = str(1);
    hDefs[ "SIZE" ] = str( blurRadius * 2 );
    hDefs[ "RADIUS" ] = str( blurRadius );

    vDefs[ "VERTICAL" ] = str(1);
    vDefs[ "SIZE" ] = str( blurRadius * 2 );
    vDefs[ "RADIUS" ] = str( blurRadius );
    
    glsl::ShaderProgramRef shaders[2] = {
        glsl::ShaderProgramRef( new glsl::ShaderProgram( vertexShader, fragmentShader, hDefs )),
        glsl::ShaderProgramRef( new glsl::ShaderProgram( vertexShader, fragmentShader, vDefs ))
    };

    //
    // create the two FBOs
    //

    FrameBufferObjectRef targets[2] = {
        FrameBufferObjectRef( new FrameBufferObject( cubemapSize, cubemapSize, GL_TEXTURE_CUBE_MAP, false, 0 )),
        FrameBufferObjectRef( new FrameBufferObject( cubemapSize, cubemapSize, GL_TEXTURE_CUBE_MAP, false, 0 ))
    };
    
    TextureInterfaceRef sources[2] = {
        source,
        targets[0]
    };
    
    glError();
    
    GLint viewport[4] = { 0 };
    enter_ortho( viewport, cubemapSize );

    glError();

    //
    // Make our kernel
    //

    std::vector< vec2 > kernel = create_kernel( blurRadius );

    for ( int blurPass = 0; blurPass < blurPasses; blurPass++ )
    {
        for ( int pass = 0; pass < 2; pass++ )
        {

            //
            // Iterate through cubemap faces; note: they're consecutive
            //
            
            targets[pass]->begin();

            glError();

            for ( int face = GL_TEXTURE_CUBE_MAP_POSITIVE_X; face <= GL_TEXTURE_CUBE_MAP_NEGATIVE_Z; face++ )
            {
                //
                // Bind target cubemap and shader
                //
                targets[pass]->beginCubemapFace( face );
                
                glError();

                //
                // bind source cubemap
                //
                
                glActiveTexture( GL_TEXTURE0 );
                glBindTexture( sources[pass]->target(), sources[pass]->textureID() );

                glError();

                //
                // Set uniforms
                //

                shaders[pass]->setUniform( "Cubemap", 0 );
                shaders[pass]->setUniform( "CubemapSizeReciprocal", 1.0f / float( cubemapSize ));
                shaders[pass]->setUniform( "Kernel", kernel );

                //
                // And render!
                //
                
                shaders[pass]->bind();

                glError();
                glBegin( GL_QUADS );
                switch( face )
                {
                    case GL_TEXTURE_CUBE_MAP_POSITIVE_X:
                    {
                        render_pos_x( cubemapSize );
                        break;
                    }

                    case GL_TEXTURE_CUBE_MAP_NEGATIVE_X:
                    {
                        render_neg_x( cubemapSize );
                        break;
                    }

                    case GL_TEXTURE_CUBE_MAP_POSITIVE_Y:
                    {
                        render_pos_y( cubemapSize );
                        break;
                    }

                    case GL_TEXTURE_CUBE_MAP_NEGATIVE_Y:
                    {
                        render_neg_y( cubemapSize );
                        break;
                    }

                    case GL_TEXTURE_CUBE_MAP_POSITIVE_Z:
                    {
                        render_pos_z( cubemapSize );                
                        break;
                    }

                    case GL_TEXTURE_CUBE_MAP_NEGATIVE_Z:
                    {
                        render_neg_z( cubemapSize );                
                        break;
                    }
                    
                    default: break;
                }
                
                glEnd();
                glError();
                
                //
                // Cleanup for this face
                //

                glActiveTexture( GL_TEXTURE0 );
                glBindTexture( sources[pass]->target(), 0 );

                glError();

                shaders[pass]->unbind();

                glError();
            }

            //
            // Release target FBO
            //

            glError();
            targets[pass]->end();
            glError();
        }
        
        //
        // Update sources so next pass will read from the results of this pass
        //
        
        sources[0] = targets[1];
        
    }
    
    glError();
    leave_ortho( viewport );
    glError();
    
    #if PROFILE_CUBEMAP_BLUR
        std::cout << "BlurCubemap - Performing " << blurPasses << " blur passes of radius: " << blurRadius << " on a cubemap of size: "
            << cubemapSize << " took " << stamp.elapsed() << " seconds" << std::endl;
    #endif
    
    
    return targets[1];
}

}

Whew!

Now, the vertex shader:
Code:
varying vec3 Direction, Right, Up;

void main()
{
    Direction = gl_Color.xyz;
    Right = gl_SecondaryColor.xyz;
    Up = cross( Direction, Right );
    
    gl_Position = ftransform();
}

And finally, the fragment shader:
Code:
uniform samplerCube Cubemap;
uniform float CubemapSizeReciprocal;
uniform vec2 Kernel[ SIZE ];

///////////////////////////////////////////////////////////////////////

varying vec3 Direction, Right, Up;

///////////////////////////////////////////////////////////////////////

vec4 sample( in vec3 dir, in vec3 right, in vec3 up, in vec2 offset )
{
    vec3 lookupDir = normalize( dir +
                               ( offset.x * right * 4.0 * CubemapSizeReciprocal ) +
                               ( offset.y * up * 4.0 * CubemapSizeReciprocal ));

    return textureCube( Cubemap, lookupDir );
}

void main()
{
    vec3 dir = normalize( Direction ),
         right = normalize( Right ),
         up = normalize( Up );
    
    vec4 color = vec4(0.0);
    
    for ( int i = 0; i <= SIZE; i++ )
    {
        #ifdef HORIZONTAL
        color += sample( dir, right, up, vec2( Kernel[i].s, 0 )) * Kernel[i].t;
        #else
        color += sample( dir, right, up, vec2( 0, Kernel[i].s )) * Kernel[i].t;
        #endif
    }
    
    gl_FragColor = vec4( color.xyz, 1.0 );
}

In short, what this is doing is is rendering each face of the source cubemap into the corresponding face of the target cubemap, passing 'direction' & 'right' vectors to the fragment shader, which then creates an 'up' vector as their cross. It can use those three vectors to sample neighbor fragments from the source cubemap when performing the blur.

The blur is implemented in two passes, one horizontal and one vertical.