I'm trying to implement lighting using the OpenGL lighting model but I'm having some trouble, random triangles are showing up as fully lit when they should be dark, it just looks like a mess. I've got a couple images to show what I mean, but first let me describe the scene.

The sphere is supposed to be a planet in space. The scene has ambient lighting disabled, so the light is supposed to be pretty harsh. There's only a single OpenGL light source, a specular-only point source (representing the sun) that's a substantial distance away with attenuation disabled. Obviously it should have a diffuse component as well, but for testing purposes it's specular as that is the component that seems to be causing all my troubles.

I've tried two different approaches to the normals. First I tried doing what my book says is the correct way to calculate normals, which is adding the face normals of all the attached faces together and normalizing the resulting vector. This looked vaguely okay on the planet itself, with a few weird artifacts, but it turned the spaceship into an awful mess, even though the normals looked correct (to my eye)

After that I've tried setting each vertex to use the normal of the face it's attached to, resulting in each vertex location having multiple normals, one for each face. This seemed to work better, but it seems to be basically equivalent to "flat" shading with no smoothing, and there are still a few strange artifacts even when doing it this way.

With the averaged normals
Using the face normals

Note that the top of the spaceship in the first image is completely flat geometry-wise, so the bizzare lighting effects don't make any sense.

Anyone have any idea what's screwed up here? Do I have the normals wrong? Do I have the lighting wrong? It's entirely possible my models just suck. Or maybe OpenGL lighting just isn't the right choice for this scene... Would learning shaders help? I have no idea what they are really or what they do, I just know everyone seems to use them...

For rendering a sphere, it makes sense to use smooth shading. Each vertex would have one normal, emanating directly out of the sphere.

Make sure you are building the faces properly.

For testing, you can render glutSolidSphere() with certain materials and lights.

Shaders might be worth looking into.

They let you replace the fixed-function pipeline with your own. They will definitely improve your understanding.

There are two types of shaders - the vertex shader and the fragment shader.

As the vertices are processed, certain attributes can be calculated position, color, etc... Then to rasterize a triangle, each fragment is calculated by interpolating the related vertex varying attributes.

You can implement any lighting model you want.

I recommend the Orange Book - it really helps with glsl (and even understanding the fixed function pipeline)

I will definitely check those out, thanks.

For the record, I've done some looking into this problem and the "random fully lit triangles" seems like it might only be a problem on my MacBook's GMA950 video card (yay)... the shading is better on nvidia/radeon cards.

Edit: Just took a look at the images on this page and it looks to me (given the sample pic they have at the bottom) that vertex/fragment shaders might be exactly what I'm looking for. So looks like it's time for me to start studying.

Not sure if bumping this is OK, but if you want your planet smooth and your spaceship sharp, you will need to add some system to determine for each vertex if it should use the smooth or the flat (face) normal.
I did this using the vector dot product between different normal vectors - if it is below some defined value (I use about 0.8), it means the normals don't differ a lot, thus it would be better to have a smooth transition between these faces. I hope what I said makes some sense.