I think I get the idea, but it wont help me too much at this stage, since I don't even know what to test for, since I am not really sure what OpenGL expects that I am not doing.

Otherwise, I actually did use this technique when writing my matrix code. I wrote a cpp tool which I quickly ran when I made some changes to the code and verified that everything was Ok. Though I used console output to give meaningful error messages instead of simple assert.

If you know what a function's supposed to be doing, then you know what to test for. As you write tests for functions, you'll gradually gain confidence that the error is not in those functions, so you'll narrow down where it could be. Once you've written enough tests, you'll either spot the error (and write a new test to make sure it doesn't happen again!), or a test will fail and you'll find the error that way.

Quote:meaningful error messages instead of simple assert.

Here is some sample output from cppunit.


[SOURCECODE]Running
.F....F.....F..F..F.......F..F..........................F....F.............

!!!FAILURES!!!
Test Results:
Run: 66 Failures: 9 Errors: 0


1) test: 10FTFontTest.testOpenFont (F) line: 71 ../test/FTFont-Test.cpp
expected: 1
but was: 6

2) test: 10FTFontTest.testSetFontSize (F) line: 113 ../test/FTFont-Test.cpp
expected: -14
but was: -15

3) test: 10FTBBoxTest.testGlyphConstructor (F) line: 49 ../test/FTBBox-Test.cpp
expected: 3
but was: 2

4) test: 10FTBBoxTest.testPlusEquals (F) line: 93 ../test/FTBBox-Test.cpp
expected: 3
but was: 2

5) test: 10FTSizeTest.testSetCharSize (F) line: 55 ../test/FTSize-Test.cpp
expected: -14
but was: -15

6) test: 16FTVectoriserTest.testGetOutline (F) line: 350 ../test/FTVectoriser-Test.cpp
expected: 29
but was: 28

7) test: 16FTVectoriserTest.testMakeMesh (F) line: 400 ../test/FTVectoriser-Test.cpp
expected: 29
but was: 28

8) test: 10FTFaceTest.testOpenFace (F) line: 34 ../test/FTFace-Test.cpp
"face1.Error() == 1"

9) test: 10FTFaceTest.testGetFontTable (F) line: 80 ../test/FTFace-Test.cpp
"testFace->FontTable( 'hmtx')"


UnitTests has exited with status 0.[/SOURCECODE]

Quote:Originally posted by henryj
Here is some sample output from cppunit

That's cool. I think it is hard to argue with your facts.

I like design, and recommend it for how to think about a problem, but I guess it is impossible to use design to isolate and fix bugs. I don't know if designs themselves can have bugs, but if so, the only way to find them would be to try to implement the design (in a computer program or some manual way). Design is a way to help think about the problem solution more clearly, to help improve algorithms and keep things organized so they are easier to understand.

I think that you can design a solution to a problem, but you cannot really design a program. You can implement a design by writing a program that adheres to it, but you have to make sure all the parts of the design are actually programmable, and then test those parts like you show. Design is inherent; I guess some people like to make it more explicit, while others don't want to waste the time.

Wow Feanor...are you feeling alright? You're the last person I would have thought would come out on the side of no design.

Quote:Originally posted by Feanor
That's cool. I think it is hard to argue with your facts.

I like design, and recommend it for how to think about a problem, but I guess it is impossible to use design to isolate and fix bugs. I don't know if designs themselves can have bugs, but if so, the only way to find them would be to try to implement the design (in a computer program or some manual way). Design is a way to help think about the problem solution more clearly, to help improve algorithms and keep things organized so they are easier to understand.

I think that you can design a solution to a problem, but you cannot really design a program. You can implement a design by writing a program that adheres to it, but you have to make sure all the parts of the design are actually programmable, and then test those parts like you show. Design is inherent; I guess some people like to make it more explicit, while others don't want to waste the time.

To my shame, I have to admit that I have not used cppunit before, but I will give it a try.

I think the problem with unit testing is that it cannot test for really obscure bugs, which only appear at rare and/or extreme conditions. Furthermore, you need the rest of the bunch, such as functional tests.

The design should be a positive proof of the algorithm to be implemented, something which is quite tough to do by itself, but there is no way to test this with a limited number of test cases, typically. You have to make some assumptions about how something is implemented to be able to design tests which exploit potential weaknesses.

Quote:To my shame, I have to admit that I have not used cppunit before, but I will give it a try.

You don't have to use cppunit. I use it because I find it useful and it has become second nature. There are alternatives or you can roll your own.

Quote: I think the problem with unit testing is that it cannot test for really obscure bugs, which only appear at rare and/or extreme conditions. Furthermore, you need the rest of the bunch, such as functional tests.

This is a common misconception. One of the best things about unit tests is it allows you to apply extreme conditions in a controlled environment. And it doesn't really matter if you miss a special case. When it does turn up it is a lot easier to simulate, debug and test in the 'clean room' of the unit test than in the production system with it's added complexity and noise as you have seen with your texture bug.

Quote: The design should be a positive proof of the algorithm to be implemented, something which is quite tough to do by itself, but there is no way to test this with a limited number of test cases, typically. You have to make some assumptions about how something is implemented to be able to design tests which exploit potential weaknesses

This is another misconception and an argument for test driven development. Most programming tasks involve calling a function with some arguments and getting a result back. This is true even of very complex algorithms. If you know the algorithm you should be able to predict the results from a given set of parameters. (If you don't know the algorithm well enough to predict the results you shouldn't be using it.) This being the case, you write the test first eg.

void testCompositeCurve()
{
FTContour contour( compositePoints, compositeTags, 18);
CPPUNIT_ASSERT( contour.PointCount() == 50);
}

This takes a known set of 18 bezier control points and produces a series of points on the curve. Independent (of the code) analysis says that there should be 50 points produced so that's what I test for. Simple. Then you write the code to produce the desired result.

Then you can write some boundary cases.

void testNullCurve()
{
FTContour contour( NULL, NULL, 0);
CPPUNIT_ASSERT( contour.PointCount() == 0);
}

As you get used to writing unit tests you realise that it is unnecessary to test every possible combination. You can apply the 'zero, one, infinity' rule and it doesn't matter anyway. Some tests are better than no tests.

Now there will be cases where it seems almost impossible to write a test. This can be because of...
a. A lazy programmer. Happens a lot. Sometimes the work to write the test is far more than writing the actual code. Do it anyway.
b. The code is a mess. In this case, rewrite your code. This is double win. Well tested, more understandable code.
c. Some technical hurdle. This almost never happens. So if you think this might be the case it's more likely a or b.

I did roll my own unit tests in the end. cppunit seemed a little bloated for any quick testing. I have reduced the overhead to 2 macros per test, and 1 per assertion, plus another one to give summarized results. No schmuck classes for testing.

Anyway, if you can tell me how I can test interaction with OpenGL rendering with a UNIT TEST, I will tell you how to travel faster than light . This would be a functional test, IMHO, because it tests the interaction of different entities. Of course, I could check OpenGL state before and after the process to be tested, but if something goes wrong in between, I won't know, and maybe because I don't have enough design to know what I am supposed to be doing (assuming the other unit tests are all ok).

Quote:This is a common misconception. One of the best things about unit tests is it allows you to apply extreme conditions in a controlled environment. And it doesn't really matter if you miss a special case. When it does turn up it is a lot easier to simulate, debug and test in the 'clean room' of the unit test than in the production system with it's added complexity and noise as you have seen with your texture bug.

If your unit tests pass, but the code is still bad? You need to do some functional tests, to narrow done your weak spot, then re-do the unit tests (and the design) to test for it. You cannot solely rely on unit tests without looking outside your cleanroom. The unit tests are the most atomic tests you can possibly do, but you have to figure out what to test for. This should be in your design, but if the design is evolving with the code, the unit tests will change when you find bugs, if you can figure out the cause.

Quote:As you get used to writing unit tests you realise that it is unnecessary to test every possible combination. You can apply the 'zero, one, infinity' rule and it doesn't matter anyway. Some tests are better than no tests.

You have to find that zero and infinity first, only then you can test it. My problem is exactly that I don't know what's wrong, because the "informal" unit tests I have performed on seperate pieces of code seem correct.

Anyhow, it seems like a good idea to write unit tests for most things, so I will do that now instead of trying to find this GL bug. Maybe it is just a stupid typo somewhere, after all. With some tens of classes, and over 30k lines of code, it is quite likely.

Quote:Originally posted by henryj
Wow Feanor...are you feeling alright? You're the last person I would have thought would come out on the side of no design.

Well who said "no design"? Sheesh. I still write requirements and overviews and build object graphs.

I do admit there's a definite limit. I guess I grew some more sense after some kind of experience accrual. Maybe I just levelled up? And there's that whole thing about code verifiability, or the theoretical lack thereof.

Quote:Originally posted by DoooG
I think the problem with unit testing is that it cannot test for really obscure bugs, which only appear at rare and/or extreme conditions. Furthermore, you need the rest of the bunch, such as functional tests.

This is not what I have been taught. Anyway, what is an "obscure" bug? Testing is meant to focus on edge conditions. If you know the input data as well as you should, you can formulate tests that check all the borderline cases, where bugs are more likely to occur, as well as typical cases. Or so I gather.

Quote:The design should be a positive proof of the algorithm to be implemented, something which is quite tough to do by itself, but there is no way to test this with a limited number of test cases, typically. You have to make some assumptions about how something is implemented to be able to design tests which exploit potential weaknesses.

Design can't prove anything, only account for things as a result of spending time thinking about the task. I guess if the problem you are working on is susceptible to mathematical proof (like by induction), you might be able to demonstrate that the design matches the algorithm, and hence is also correct, but that doesn't mean that the code will match the design.

Quote:Anyway, if you can tell me how I can test interaction with OpenGL rendering with a UNIT TEST, I will tell you how to travel faster than light.

.

Write a test harness that stands in for OpenGL and link to that instead. Then you can catch all of the data you are sending in and make sure it matches the specified parameters of the particular API call.

DoooG...

Pretty much all of your concerns are unfounded and I know where you are coming from. I was a sceptic at first but once you get a taste for unit tests you will understand.

As for testing OpenGL, you answered that one yourself.

Quote: The unit tests are the most atomic tests you can possibly do

You test the specific bit you're interested not the whole system.

If there is something you don't know how to test, send it to me and I'll see if I can find some time to help.

Quote:Originally posted by henryj
DoooG...

Pretty much all of your concerns are unfounded and I know where you are coming from. I was a sceptic at first but once you get a taste for unit tests you will understand.

As for testing OpenGL, you answered that one yourself.



You test the specific bit you're interested not the whole system.

If there is something you don't know how to test, send it to me and I'll see if I can find some time to help.

Ummm, ok, I think I understand my misunderstanding. In this case, I just realized, I have made an acceptance/functional test by making the actual progam, and the test failed, since it does not display correctly. So the question is:
what if my unit tests are all correct, yet the overall result is bad?

I think the answer is that my unit tests are bad, and the design bug has to be found. Correct?

To get an object drawn on the screen, the interaction of 8 classes is required at a minimum, and as far as I can tell, the way for testing this is to make a little app that tries just that (just what I did). The computer won't be able to tell if the rendering is correct, so I have to do it.

Quote:To get an object drawn on the screen, the interaction of 8 classes is required at a minimum, and as far as I can tell, the way for testing this is to make a little app that tries just that (just what I did).

Go back to your correct 'atomic' comment again.

Don't test the results of the interactions of those eight classes, test the individual classes in isolation. This is WAY easier from a coding point of view and easier to work out what to test.

Make sure your tests at least mirror what is going on in your system. Copy and paste the code into your tests

If all eight classes are well tested and correct then it's highly likely that the system will behave as expected.

If it doesn't then your logic (design) is wrong.

Quote:The computer won't be able to tell if the rendering is correct, so I have to do it.

You can grab a 'correct' frame and then write a test to compare this against. This assumes at some point you got your program to work.