I really don't see why you would do the actual rasterizing on the server side, especially for a real-time app, as transmitting a movie stream is much more bandwidth intensive than the data you need to draw the same thing on the client side.

Apart from that, the response time would be abysmal, as you not only need to send input to the server, and send the rendered image back, but you need to do the rasterizing and compression, too. Lag lag lag, I don't think this is a good idea.