Win32/DirectX to OSX/OpenGL: Initial request for help for spare time project

Toby at home
Unregistered
 
Post: #1
Hi folks,

I've got a lot of questions and a lot of background relating to whether I should embark on a spare-time project to port our company's client-part of our general purpose MMOG engine to the Mac, thus enabling all our current and future projects to work under MacOSX.

Ultimately, I wish to get some pointers to get me going and have a very large message drafted that provides appropriate context; but I don't wish to abuse people's good nature here with a vast social fopar as my first post.

The forum pointers don't say too much about length of post - so I figure I'd ask permission first before posting a vast essay with my context and questions to date with something a bit briefer... the dreaded "Executive Summary", if you'll forgive the Dilbert speak:

90% of the code is standard C++ using the C run-time library and STL. The display and audio interfaces are hidden behind our own API. It is a real time 3D app that under Windows uses low-level DirectX9 to render itself. Win32 handles main loop, threads, windows messages, menus.

I'm currently lost in Carbon, the Apple documentation and the Web. I have lots of questions! My spare time work on this so far (a week) has led to XCode, a Carbon book and a whole pile of silly windows, menus, programmer-art icons and applications that generally demonstrate my ability to stick "Hello World" on the screen on any computer, anywhere, even when drunk.

Is it OK to post something even more verbose than this? I solumnly promise that it contains real content.

Toby
(C++ and Win32 developer, previously an Amiga 68K programmer, er... then BBC-B and Oric with their delicious 6502... the 8085... my own history makes me feel old. Oh, and some ARM programming.)

Edit: I should have said the whole reason I'm doing this is that I ended up with a Mac at home last yar, fell in love with it, and use it for all my home computing, music and TV. As a user, I adore the Mac and OSX: I *want* to do this!
Quote this message in a reply
Moderator
Posts: 1,560
Joined: 2003.10
Post: #2
Toby at home Wrote:Is it OK to post something even more verbose than this?

Absolutely!
Quote this message in a reply
Toby at home
Unregistered
 
Post: #3
ThemsAllTook Wrote:Absolutely!

Ok, you asked for it :-)

Sorry it has taken me a few days to post this follow-up, I've been lost in a deadline and only just surfaced.

Most of my issues lie around not knowing which level of abstraction to hit OSX at.

The majority of the code, as I mentioned in the opening post, is standard C++, STL and Microsoft's C run-time library. Thus I use _beginthreadex() to start a thread rather than Win32's CreateThread.

Thread management is done using Win32's critical sections, interlocked operations (increment, decrement) and their events. I understand that these can easily be #ifdef'd to their equivalent (pthreads?) on the Mac so that the main code base can "swing both ways", if you'll pardon the expression. Is this the case? Given my use of the CRT rather than Win32 where possible and using STL and the standard C library and STL, what books and/or documentation should I be looking at?

Win32 also manages the main window, its menus and the various message pumps that let me receive keyboard and mouse input. The main Win32 message pump uses idle time to ensure that the timed systems are updated on a guaranteed average-over-time rate of 50 ticks per second and uses remaining time to burn on the display. If the frame rate hits the monitor's refresh rate, the remainder of the time the main thread generously donates back to the system. There's also a "play well with other apps" option that deliberately sacrifices a small percentage of display performance so that you can use the client in a window and still do heaps of other stuff without grief.

Should I be using Carbon for this? I read that using Carbon's idle event in this way is "super-no-no" (mind you, the same is written about MFC's equivalent). How would I guarantee that my update loop ticks at 50 times a second this way? Is Carbon's idle loop and its timers accurate in a "WM_TIMER" way (i.e., not accurate in the slightest) or accurate in a high-resolution timer way? I use Win32's QueryPerformanceCounter() thingie (the high-res one), work out how much time has passed and call the update loop appropriately. Remainder of the idle loop goes to the display (so there may be *many updates for one frame* on slow display systems or *many frames for one update* on faster systems).

It is very important to me that I "play well" with MacOSX. I'm only doing this because I love my Mac and want to make a true Mac application, but at the same time:

* It would be nice to prepare the main code base for multiple platforms such as PS3, Linux, etc., rather than adapt it to simply be a case of "#ifdef mac #elseif do windows stuff" job (as a C++ programmer at heart, I hate #ifdefs and defines anyway as they don't help the pre-processor one iota when it comes to spotting my silly mistakes).

* The easier it is to adapt the existing Win32 code-base to "just compile" on multiple platforms, the more likely this is to fly. If doing a Mac port involves vast amounts of code, it'll never get maintained either by me or anyone else (spare time can only stretch so far :-)). The idea here is to strip away system specific stuff from the existing code base and then create the minimum amount of wrapper code to make it compile lock-stock and five smoking barrels on the Mac. I want at least 50% of this job to be generalising and de-windows-ing the existing code (as that's for keeps forever) and the rest with the Mac.

This deserves some sort of executive summary:

- How do I handle my real-time application? Ticking the timed systems 50 ticks per second and burning the remainder of the time up to the desired frame rate whizzing the display?

- What APIs should I use for thread management, dialogs, menus, windows and user input that would best match to the style of the existing Win32 code? Are there any advantages or disadvantages of specific approaches?

- Is Carbon the thing for me, or not?

- Our display engine has a platform independent API. I would only have to port the low-level rendering systems and the data load stuff to swizzle everything around from DirectX vertex-o-arrangement to OpenGL's. Not much code, pretty easy to do. What APIs, documentation, linker libraries and what-nots do I need in order to be able to hit OpenGL (without GLUT) and what APIs would I need to research to figure out how to attach an OpenGL display to a nice tidy Mac window? I last used OpenGL in '99, but I'm sure it'll all eventually come flooding back to me... :-)

- What sources of reference, documentation or books do the assembled crowd recommend that I obtain that will help me with this?

- Are there any big gotchas that will affect my ability to make OSX development utterly incompatible with other UNIX based operating systems?

- Is there any example code I could look at that might help me out?

I'm really sorry for all the questions. I know I can eventually figure out this stuff by myself, but doing that with Windows took years. With a toolbox that's a little like Doctor Who's TARDIS, sometimes it's hard to know which screwdriver, chisel and hammer to take first. I'm hoping the collective wisdom can save me a little grief and at least set me off down the right road.

Many thanks in advance for any and all responses, I appreciate it greatly. Think of it as a charitable donation to "help an elderly programmer learn a new platform in a hurry" (taking donations since 1981).

Cheers :-)

Toby
Quote this message in a reply
Moderator
Posts: 1,560
Joined: 2003.10
Post: #4
Whew, that is a lot. I'll just answer the stuff I actually know something about, and hope other forum members can fill in the blanks:

Toby at home Wrote:- How do I handle my real-time application? Ticking the timed systems 50 ticks per second and burning the remainder of the time up to the desired frame rate whizzing the display?

In Carbon and Cocoa, the usual way to do this is with timers (see InstallEventLoopTimer and NSTimer, respectively). Carbon and Cocoa timers are a lot more accurate than WM_TIMER, but still in no way guarantee that you'll be called as frequently as you request; it'll just do its best to call your timer function approximately that often.

What you can do to manage this is to decouple the speed of your game mechanics from your graphics frame rate. There are several popular ways of doing this, each with their own pros and cons; here's a tutorial I've written which describes the way that's worked best for me: http://www.sacredsoftware.net/tutorials/...tion.xhtml

Toby at home Wrote:- What APIs should I use for thread management, dialogs, menus, windows and user input that would best match to the style of the existing Win32 code? Are there any advantages or disadvantages of specific approaches?
Toby at home Wrote:I want at least 50% of this job to be generalising and de-windows-ing the existing code

As long as you do that first, you should be able to use whatever API you want for those things, as they'll be exclusively in platform-specific code.

Toby at home Wrote:- Is Carbon the thing for me, or not?

It could work, but I'd suggest at least taking a serious look at Cocoa before you decide one way or another.

Toby at home Wrote:- Our display engine has a platform independent API. I would only have to port the low-level rendering systems and the data load stuff to swizzle everything around from DirectX vertex-o-arrangement to OpenGL's. Not much code, pretty easy to do. What APIs, documentation, linker libraries and what-nots do I need in order to be able to hit OpenGL (without GLUT) and what APIs would I need to research to figure out how to attach an OpenGL display to a nice tidy Mac window?

AGL or CGL if you're using Carbon; NSOpenGL if you're using Cocoa. Note that last I knew, CGL could only create full-screen contexts.
Quote this message in a reply
Toby at home
Unregistered
 
Post: #5
ThemsAllTook Wrote:Whew, that is a lot. I'll just answer the stuff I actually know something about, and hope other forum members can fill in the blanks:

Thanks :-) Any and all help is greatly appreciated. I'm just feeling around in the dark at the moment as a bit of a newbie (or 'noob' as the youth of today seem to prefer ;-))

ThemsAllTook Wrote:In Carbon and Cocoa, the usual way to do this is with timers (see InstallEventLoopTimer and NSTimer, respectively). Carbon and Cocoa timers are a lot more accurate than WM_TIMER, but still in no way guarantee that you'll be called as frequently as you request; it'll just do its best to call your timer function approximately that often.

I shall look up the documentation of this and have a look. I guess the key to my accurate timing is to know what "now" is really accurately. If I know what "now" is, and I know what "then" was, I'm sorted - I can catch up and ensure the timing works out in the long-term.

ThemsAllTook Wrote:What you can do to manage this is to decouple the speed of your game mechanics from your graphics frame rate. There are several popular ways of doing this, each with their own pros and cons; here's a tutorial I've written which describes the way that's worked best for me: http://www.sacredsoftware.net/tutorials/...tion.xhtml

The game mechanics and the frame rate are already decoupled... It is, however, critical that the update and display run synchronously. A display cannot happen whilst an update is happening. This would require thread synchronisation that would be prohibitive in the display engine and harm performance and stall our access to the GPU. Thus, it's OK for the GPU to be still rendering stuff whilst the next update kicks off, but it's not OK for the render loop to be running at the same time as the update loop.

The next thing that is critical is that the update loop's accuracy over time is bang-on. Any significant deviation leads to mis-synchronisation of the client and server and a choppy experience. At T+10, T+20 and T+1000, I've got to be able to know that T/50 updates have occurred, give or take a really, really small percentage for the client/server synchronisation to be good (for bandwidth reasons: we have a deterministic newtonian dynamics engine - and this allows us to put real-time newtonian dynamics into MMOGs and let you connect on a 28K modem, if anyone's still got one of those :-)).

The reason the engine runs its stuff (display and update) in a real-time loop (I PeekMessage() to see if anything is in the queue before giving WIn32 a bean, CPU wise, it has to pre-empt my main thread's quantum if it wants a look-in if I'm busy) is to ensure that we work the GPU and the CPU real hard whilst giving the operating system uncluttered access to our thread's time if it wants it (although we'll claim it back in spades with multiple update loops later on to catch up). I.e., we play nice with the operating system, but we're a full-screen app: your machine is temporarily ours (cue evil laugh). However, if we're running in windowed mode, we're faaaaar more generous with surrendering our main thread's time.

Everything on the client that can happen in parallel with the minimum of stalls to the main thread happens in other threads (there are many tens of threads involved in running the client application overall, so we gain significantly from multi-core CPUs)

ThemsAllTook Wrote:It could work, but I'd suggest at least taking a serious look at Cocoa before you decide one way or another.

AGL or CGL if you're using Carbon; NSOpenGL if you're using Cocoa. Note that last I knew, CGL could only create full-screen contexts.

Thank-you for your help and patience! I'll check all that out and see what I can learn. Sorry for so many words :-)

Toby
Quote this message in a reply
Luminary
Posts: 5,143
Joined: 2002.04
Post: #6
Win32 threads and pthreads have somewhat different capabilities and quite different APIs. There are good pthread-compatibility wrappers for Win32 threads which might be worth looking into to get the same code-base working on both platforms.

I'd recommend you use Cocoa rather than Carbon -- it's better-documented, less of it is deprecated, and many more people know how it works and are therefore able to help with problems you might have.

Your update/display architecture sounds massively and unnecessarily complicated Wacko
Quote this message in a reply
Member
Posts: 110
Joined: 2002.04
Post: #7
Hi if you are using C++ STL etc

You really should take a look at BOOST.

The future direction of STL

Among otherthings it includes a LOT of good xplatform wrappers
example, threads, files, paths,

Most of these are portable or have been ported to most game consoles.

So that is one rather standardzied way you could help make your whoel code base more crossplatform.

http://www.boost.org/

Im sure you know but you also have options like using libSDL
to provide a lot of your crossplateform warppers for threads and events etc.

http://www.libsdl.org/

That also runs on most consoles... PSP, DS you name it.

Any how I have NOT read all the posts above yet.

- Mac Lead ZeniMax Online Studios
- Owner Plaid World Studios
- Resume: http://www.chrisdillman.com
Quote this message in a reply
Toby at home
Unregistered
 
Post: #8
OneSadCookie Wrote:Your update/display architecture sounds massively and unnecessarily complicated Wacko

:-)

It's simpler than I make it sound - 20 lines of code for the main loop. It has been evolved over many years to provide the service it needs to: I'm connecting up to 2,500 players to a server and those players are interacting with potentially 50,000 autonomous objects, all with physics.

Without us having all the bandwidth in the universe, we have to be smart about how we keep the two sync'd, which requires a huge degree of things like deterministic physics. The way it is updated has proved to be exceptionally effective: we have a MMOG RTS which can be played with hundreds of units over a 9600 baud modem (indeed, if you chose to, you could play the first day of a game on a 300 baud accoustic coupler) -- this means we don't need the pipe of the gods, less traffic is excellent for everyone, and we all come out smiling :-)

Toby
Quote this message in a reply
Toby at home
Unregistered
 
Post: #9
ChrisD Wrote:Hi if you are using C++ STL etc
You really should take a look at BOOST.

I've spent a lot of time researching it; but it does not provide anything significant to the application that we do not already have wrapped up in our own systems: I'll wait until either it solves something or we know where we stand with TR1.

We've already dealt with platform specific issues in the vast volume of code (hundreds of thousands of lines), it really is just thread management, the tiny amount of windows and OS management, the display engine and the graphics engine. I just need to know how to get the standard stuff compiling and what the best route is to implement the main loop and surrounding gubbins.

Thanks very much for your post, most appreciated.

Toby
Quote this message in a reply
Member
Posts: 110
Joined: 2002.04
Post: #10
Toby at home Wrote:I've spent a lot of time researching it; but it does not provide anything significant to the application that we do not already have wrapped up in our own systems: I'll wait until either it solves something or we know where we stand with TR1.

We've already dealt with platform specific issues in the vast volume of code (hundreds of thousands of lines), it really is just thread management, the tiny amount of windows and OS management, the display engine and the graphics engine. I just need to know how to get the standard stuff compiling and what the best route is to implement the main loop and surrounding gubbins.

Thanks very much for your post, most appreciated.

Toby

Oh from what you posted before it sounded like you had not wrapped these things up and were looking for crossplatform solution you could use to possibly modernize or update the code base to be more crossplatform at a core level vs
just having a mac port with IFDEFs all over.

Did I get that totally wrong?

BTW Toby What is this RTS game?

I only remember 2 MMO style RTS games over the last 10 years.
and I only thoughrt 1 was still in business so Im interested....
Im also a big time RTS fan in general.

- Mac Lead ZeniMax Online Studios
- Owner Plaid World Studios
- Resume: http://www.chrisdillman.com
Quote this message in a reply
Toby at home
Unregistered
 
Post: #11
ChrisD Wrote:Oh from what you posted before it sounded like you had not wrapped these things up and were looking for crossplatform solution you could use to possibly modernize or update the code base to be more crossplatform at a core level vs just having a mac port with IFDEFs all over.

Did I get that totally wrong?

Sort-of, I guess I did a poor job of explaining myself. The vast, vast majority of the code is already crossplatform: we have invested significant time and effort into doing this. Some still remains, but that'll get hit the first time we actually try and make it work on another platform.

Each platform needs to deal with its own stuff behind the Display API, Audio API and each platform needs a "main.cpp" which does all the OS stuff, runs the main loop, deals with the menus, windows, full-screens, screenblankers, power-down requests, user switching, etc.

Thus, I need to port "main.cpp": The main loop and the surrounding "stuff" that links everything together and the rest of the C++ application will probably 'just compile' give or take a few errors. I have already done a straight compile on both our command line run-time content compilers and they went through with 0 errors and 0 warnings (which is nice).

Now, given the rest of the app is C standard library, C++ and STL my problem remains: how do I do the "master wrapper" that drives the rest of the (already platform independent) code?

I did a test compile on the VM and a few other client bits last night and they compiled under an XCode carbon project first time (again, no errors or warnings): so it really is a case of porting less than a thousand lines of code, slapping a temporary OpenGL wrapper behind the display API, disabling the sound temporarily and I *should* be able to run all of our products immediately (even if every texture is blank and every mesh is a cube or teapot for test purposes).

ChrisD Wrote:BTW Toby What is this RTS game?

It's called "Time of Defiance" - we did it as a technology test originally, released it in 2002 and then our publisher generously went into Chapter 11 (these things happen). We still run it for the existing fans of the game and it helps us to ensure our server software operates flawlessly 24/7 and the client runs on a wide variety of machines. We also periodically make additions, tweak it, etc. Indeed, a new release will fall out over our auto-update system in a week or so because we've sped up the display, added a user-requested feature or two and generally fiddled with this and that.

Think of it as "1000 player Risk®" and you won't go far wrong.

If you want to give it a spin, sign up for the free trial and PM me your customer ID and I'll upgrade your account so you can fiddle around on all the servers for a couple of months free.

However, it's the massive 30,000 concurrent MMOG for a childrens' program for a major UK broadcasting company that is fueling my interest in a Mac port of the client, quite apart from the fact I could play Time of Defiance at home (which would be nice). It's a pretty massive project for us and having a Mac version would be cooler than a frozen penguin on Europa.

Maybe I should have picked decorating the house as a spare time project instead ;-)

Toby
Quote this message in a reply
Member
Posts: 110
Joined: 2002.04
Post: #12
Hmm
Well this is my simple answer...

What I do in BANG3D is...

I have a main loop or application class per platform.

This handles fetching OS events and posting then to the even broadcasters.

Out side of that it simply calls a registered call back function ptr or game object.

That object calls all the game specific code ... move things, render etc.

At that point everything has been properly abstracted and the OS events all abstracted away etc.

I use if defs around my entire application class definitions based on what platform Im building for.

All of which is set in a master config file or prefix file.

- Mac Lead ZeniMax Online Studios
- Owner Plaid World Studios
- Resume: http://www.chrisdillman.com
Quote this message in a reply
Toby at home
Unregistered
 
Post: #13
Hi folks,

I actually got the main loop going using Carbon to manage my window. I used NewEventLoopTimerUPP() to set up a timer on a 0.001/sec frequency and use Core Service's Microseconds() for my own time measurements. This is a raw loop: render and engine update are empty void(void) loops:

Code:
[Session started at 2007-05-12 13:33:16 +0100.]
Run took 96.19973 seconds
Time given back to operating system = 95.01 seconds
Emergency time given back to operating system (running hot) = 0.00 seconds
Main engine 50FPS updated at 49.99 ticks/sec
Accuracy: 99.98% (off by +/- 0.0205%)
Idle engine updated at 198.65 ticks/sec

Incidentally, the time we're accounting for isn't us - it's ALL of the OS, so iTunes and what-nots that I've got going.

Anyway, figured someone might be curious as to my progress :-)

Toby
Quote this message in a reply
Member
Posts: 110
Joined: 2002.04
Post: #14
Toby at home Wrote:Hi folks,

I actually got the main loop going using Carbon to manage my window. I used NewEventLoopTimerUPP() to set up a timer on a 0.001/sec frequency and use Core Service's Microseconds() for my own time measurements. This is a raw loop: render and engine update are empty void(void) loops:


Set the NewEventLoopTimerUPP() call back for the main loop

to kEventDurationMicrosecond

I was able to get more CPU time then the default kEventDurationMillisecond
I see recommended in examples an I think you are using now?

Oh and email me Wink
I PM you did you get it?

- Mac Lead ZeniMax Online Studios
- Owner Plaid World Studios
- Resume: http://www.chrisdillman.com
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  game development using directx/cryengine codetoeternity 2 1,478 Oct 10, 2013 06:34 AM
Last Post: codetoeternity
  win32 / osx equivalents pikmini 3 3,572 Oct 12, 2008 11:37 PM
Last Post: Bachus
  Initial Svn Import? bronxbomber92 5 3,931 Mar 2, 2008 12:21 AM
Last Post: Skorche
  Cross Compiler for Win32 bronxbomber92 2 2,702 Dec 17, 2007 05:18 PM
Last Post: Terrydil
  DirectX anthony 3 3,064 Jun 18, 2007 02:36 PM
Last Post: anthony