Advanced Engine Design

Member
Posts: 46
Joined: 2008.10
Post: #1
Hello again! It's been a while, but it's a delight to return. (I have a real job now--yay!--and a child--very cute!--which are, between the two of them, taking up nearly all of my time.) This post has a significant amount of background material, so scroll down to the bold text to skip straight to the question.

Background
I've developed a number of games and applications, some of which use variations on a mid-level engine I spent some time developing (with significant help from iDG!) a year or two ago. Now it's time for a big step, and I could use your advice taking my game (double-meaning here) to the next level.

Let's talk about advanced engine design. Most low-level games just sort of throw bits together, while mid-level (and mobile) games will have a more well-organized framework in mind--typically OO, though this could mean anything from simple encapsulation of common calls to GL/SDL/etc to something more abstract, like multi-thread subengine nodes streaming information across sockets (ok, that's a bit much, and partly nonsense, but you get the idea).

I've found these two levels of approaches work just fine for their own domains, but are both horrific for serious, high-level, hard-usage, big-scale games/apps. There reaches a point where you need to completely change your mindset and approach, lest you become too bogged down in hacking events, state management, and other parts together to get something that works. These problems increase exponentially with scale, meaning--among other things--any 'early' or lower-level engine hacking will have very bad-tasting consequences and time-sucks later on down the road.

I've tried a number of different approaches, but so far none of my ideas for engine design have really stuck, or come off as smoothly as I had hoped. (For more details concerning some of my approaches, see the end of this post.) It's clear that I need to learn how 'the pros' do it, particularly under the hood. Unfortunately, none of the standard gaming resources I've read even come close to thinking or explaning solid concepts or theory on this level.

The Question
What approaches are taken for designing advanced, professional-level game engines? What does such an engine look like 'under the hood'--how is it laid out, what kind of objects and relationships are used, how are events/mechanics implemented, etc? Most importantly, what resources can I use to learn the answers to these questions (preferably books)?

Thanks! Now sit back and enjoy an adventure through amateur game engine design, with your host, Tythos.

Legend
Until the final engine design, the following legend is used: red dots are entry points, dashes lines are scopes (namespaces or classes), blue circles are functions or events, purple squares are states or variables, and black arrows indicate dependencies in a particular direction.

Engine #1: "Alphonse" (Procedural)
http://tythos.net/creatives/engineAlphonse.jpg
No real organization here--a significant amount of key states are stored in the global scope, and nearly all logic takes place on a function-to-function basis. This is fine for your basic games, and is actually great for implementing or testing new features and / or technologies that you just want to mess around with on their own. Not surprisingly, it's horrific for anything beyond the basics.

Engine #2: "Bartelby" (Basic Encapsulation)
http://tythos.net/creatives/engineBartelby.jpg
Much more organized, but scalability-wise or ability-wise, this isn't a huge improvement over Alphonse. The main advantage here is that you're hiding a lot of the gritty, necessary-for-any-application dirt work, like managing graphics contexts, etc. Your program is still completely out in the open, though, and trying to create anything resembling decent game mechanics is a nightmare--your events and logic are still defined by functions in the global scope that aren't easily organized or encapsulated.

Engine #3: "Christina" (Advanced Encapsulation)
http://tythos.net/creatives/engineChristina.jpg
Ok, now we're getting somewhere. No more icky global-scope nonsense; everything's nice and neat in it's own class family. 3d and 2d objects and panels (as I call 2d objects) are handled much better here, and events (including game mechanics and logic) is stored as a list of if-then pair of function pointers in the main app object. I actually had an excellent series of successful applications with this design--both games and scientific modeling/simulations. It's still a bit messy with respect to events, though, and if you're building an engine for reuse (by yourself and others, where you don't want to allow access to the app or source code) this becomes not-quite-ideal. External data communication, like networking, file, and database access, is non-existent. Probably the worst part is how limited your events are in scope--you have to pass or store all the states they need explicitly, as they have to be defined in the global scope before their pointers are stored in the app.

Engine #4: "Dorothy" (Patterning)
http://tythos.net/creatives/engineDorothy.jpg
Ok, Dorothy was a thing of beauty--in theory. If you're familiar with the MVC pattern, this is built from that--only you still have a master app managing the program as a whole, as well as data transfer protocols (networking, files, etc) at the root of it all. The idea is to store all of your states in relevant model classes, do all of the rendering in view objects associated with models, and have all of your events defined as controls that change some states based on other states. Again, this is grand in theory, but in practice there are far too many scope and memory management issues that fly in the face of all those pretty little dependency-limitation arrows. For example, what happens when a gun is fired? That's an event, but the control class object that detects a 'gun fired' state (whatever that is) has to still create its own model (and corresponding view, and events/controls...) as part of the 'gun fired' state for the bullets and effects that result. On top of that, these new objects have to be stored somewhere, which is difficult since the app has to manage the list of all objects, and the control event only has itself and its children within scope. Events and scope are really what killed this idea, which never came to fruition long enough to be used for a working project.

Engine #5: Ebeneezer (Separation and Scripting)
http://tythos.net/creatives/engineEbeneezer.jpg
Ok, I think I've got it this time. The diagram may look a bit complicated, but in implementation, this actually stands to be simpler than Dorothy and maybe even Christina. The big, big saving grace of this approach is that humble little purple box on the left ('interpreter'). It became obvious with Dorothy that scripting game mechanics and events externally would have major advantages over trying to store and check lists of function pointers, or trying to encapsulate events to a limited scope, internally within the engine itself. The trick is to find a way to create the right bindings and resolve access to states in an effective manner, but I think I've got it. Everything else just fell in to place after that, as it's become more and more clear what works and what doesn't. Next best part: because everything that's customizable for different games is completely outside the core engine (the core engine is above the purple stuff, which represent file resources), you can use a single build of the engine for multiple games, or publicly share it with your friends, or use for bragging rights / job applications, etc. This was 70% the case with Christina; now it's 100%.

Lessons Learned
Data access and scope is key. Expect a proper event management system to be the biggest pain. Implementing a scripting engine of some kind (link an interpreter to Lua or Python, for example) is by far the best way around this--if you can create all the necessary bindings and provide a way to resolve scope of states and objects from the scripts. It's also so much better to plan an engine that, even if it isn't ideal, can be built from smaller, workable parts, instead of trying to implement one grand ideal collection of abilities all simultaneously. These may seem obvious, especially on smaller projects, but when working on a large-scale, long-term program, missing these points or letting them slip will kill you.

Edit: Fixed spelling in title. Whoops.

"Who's John Galt?"
Quote this message in a reply
Moderator
Posts: 1,560
Joined: 2003.10
Post: #2
One of the books that helped me most in thinking about organizing my code, though not directly game related, was The Art of Unix Programming (HTML version isn't great to read; also on Amazon). Of particular interest is Basics of the Unix Philosophy, which sets out 17 rules that I've found to be excellent guidelines when designing any system.
Quote this message in a reply
Moderator
Posts: 916
Joined: 2002.10
Post: #3
This would make a great article
Quote this message in a reply
Member
Posts: 46
Joined: 2008.10
Post: #4
Excellent advice, and for anyone who hasn't read that article and those rules, I second them with a strong recommendation.

That having been said, one of the main issues I've been having is how to properly allow and resolve access to states from other objects and events. This has primarily been an issue because, like a good little FOSS/Unix developer, I've incorporated strict dependencies and minimal-access interfaces, building up the engines from smaller blocks. You may see the Unix design particularly strong in Dorothy, though nothing of course is perfect.

To be more specific, the solution I've settled with is to categorize objects and resources by sorted string-pointer tables which can be used to parse an interpreter request to a specific state or function. This is not ideal, smacks of very non-Unix interdependency and dynamic/reflective programming, and incorporates all of the potential problems thereof, but seems to be something that will work and scale reasonably well. (I should probably mention at this point that I'm developing in C/C++/SDL/OpenGL/Python, for those who are curious about the technicalities and context in which such an effort is taking place.)

"Who's John Galt?"
Quote this message in a reply
Post Reply