What is the fastest way to blit a framebuffer to the iPhone screen?

Apprentice
Posts: 13
Joined: 2011.02
Post: #16
I am just starting and this thread caught my attention. I did some search and came up with this:

http://www.opengl.org/discussion_boards/...ber=262751

It says if you use GL_BGRA instead of GL_RGBA you can avoid swizzling! It also says the iPhone supports GL_BGRA. Is this true? Are there any samples that shows simple plotting of glTexSubImage2D that does not use the other higher level frameworks?
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #17
Look at the extension string.

OpenGL ES does not support BGRA. But there is an extension exposed on all iOS devices which adds this capability. It used to be GL_IMG_texture_format_BGRA8888 (poorly documented), but is now GL_APPLE_texture_format_BGRA8888. The only difference between the two is a subtle semantic; APPLE_texture_format_BGRA8888 says BGRA is a valid <format>, but not a valid <internalformat>.



More to the original topic: using BGRA can be slightly faster than RGBA if it means swizzling can be skipped.

But this doesn't imply anything about other transformations the driver may need to do on texture data (i.e. twiddling to cache-friendly layout.) Generally speaking, texture upload is not a fast operation. At least, repeatedly modifying and moving data from the CPU to the GPU is not as fast as specifying it once and sampling from it repeatedly.
Quote this message in a reply
Apprentice
Posts: 13
Joined: 2011.02
Post: #18
You just wrote down a lot of important things that I think needs to be clarified.

You are stating the following:

1) Althought OpenGL ES does not support BGRA, Apple made a non-standard extension only on iOS devices to support it. Apple makes the distinction between them by labeling <format> as iOS compatible, and <internalformat> as OpenGL ES compatible. By supporting <internalformat>, your program may run on other devices supporting OpenGL ES, and by supporting <format>, it can only run on iOS devices (apple essentially).

2) It is extremely slow to move the background texture from main memory to the GPU, so you should keep a clean copy in the GPU, and use another same size buffer to do the work on it. Without the clean copy, you would need to keep loading a clean copy from the main memory. In fact, this points to three buffers. One clean background copy. One buffer to do work on that gets a copy of the background. The last one for displaying. (double buffer). You switch the display pointer alternately between second and third buffer. When one is displayed, you use the other one to work on. But can the iPhone store 3 copies in the GPU buffer? Also, there is no GPU buffer, so I am assuming this is just first few megabytes in memory that the iOS reserves and you can't touch, so essentially it is acting like one.
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #19
(Feb 28, 2011 01:17 AM)edepot Wrote:  1) Althought OpenGL ES does not support BGRA, Apple made a non-standard extension only on iOS devices to support it.

Really, the hardware vendor made an extension, GL_IMG_texture_format_BGRA8888, because their (MBX) hardware prefers data in that order. Exposing native formats via extensions is a common practice in OpenGL.



edepot Wrote:Apple makes the distinction between them by labeling <format> as iOS compatible, and <internalformat> as OpenGL ES compatible.

This deserves some clarification.


First, OpenGL. Every API that transfers pixels ([Get][Copy]Tex[Sub]Image, ReadPixels) is a potential conversion operation. Consider glTexImage2D: you are copying data from your application (a CPU malloc) to OpenGL (CPU or GPU, this detail is opaque.) The API has an <internalformat> parameter which hints how OpenGL will store the data. The <format> and <type> parameters exactly define how your application is storing the data.

A kabillion permutations are possible here, for example you could have RGBA, UNSIGNED_BYTE data in your app. But ask OpenGL to convert it to 4-bit greyscale with LUMINANCE4 <internalformat>. See the "Pixel Rectangle" and pack/unpack language in the specification for the details.


Second, OpenGL ES. The goal of ES is to simplify the API, both for the app programmer and the driver implementation. So ES tries very hard (it doesn't quite succeed...) to eliminate all format conversion. It assumes that an application will provide textures in pre-optimized formats for the device, and the driver can simply copy the data. In reality, this isn't always possible; there are still hardware requirements that a driver will have to satisfy. Twiddling texture data to a hardware-friendly format is the relevant example for Imagination's GPUs.

Anyway, the ES specification says this:
Quote:internalformat ... must match format; no conversions between formats are supported during texture image processing.

The subtlety here is interpreting the English word "match". The GL_IMG_texture_format_BGRA8888 extension interprets "match" as "==", so you would write

Code:
glTexImage2D(GL_TEXTURE_2D, 0, GL_BGRA, w, h, 0, GL_BGRA, GL_UNSIGNED_BYTE, data);

However this is counter to how <internalformat> has always worked in OpenGL (specifically, the glTexEnv interaction.) So the GL_APPLE_texture_format_BGRA8888 extension interprets "match" as "be compatible with", and you would write

Code:
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, w, h, 0, GL_BGRA, GL_UNSIGNED_BYTE, data);

which is how desktop OpenGL has always worked.


This is all just a tiny detail over spec language. The goal of both extensions is to provide a (slightly) faster texture upload path.



edepot Wrote:2) It is extremely slow to move the background texture from main memory to the GPU, so you should keep a clean copy in the GPU, and use another same size buffer to do the work on it.

I think you're mixing two concepts here, because you're trying to use glTexImage as a way to write pixels directly to the framebuffer. It really can't do that.


When you call glTexImage, OpenGL must immediately make a copy of the data from your pointer, because you can modify (or delete) the storage right after that. So there are already two copies, at least instantaneously. Typical applications will immediately delete their copy, so there is only one copy after the app is done loading.

To get those pixels onto the screen, you then need to draw the texture to the framebuffer. So there's another allocation for the framebuffer itself. Typical applications draw complicated scenes (3D, etc) but if your goal is to blast pixels to the screen in 2D, your texture and framebuffer will end up with exactly the same content, which is redundant.

On iOS, you allocate "the framebuffer" with -renderbufferStorage:fromDrawable: which is really creating a queue. The details of this are opaque, but the system is smart and will double/triple buffer on your behalf to improve performance. This is why, by default, when you call -presentRenderbuffer: the content becomes undefined. Really the system is rotating through a queue of images.

There is no OpenGL API, however, to modify pixels directly in a renderbuffer. The point of OpenGL is to abstract away the hardware details (like twiddling) so that you don't have to re-write your application for every GPU ever made. You can't see the video memory directly, all you can do is render primitives, like colored points, or textured quads.


The summary here is that the current ES implementation is designed and optimized for applications that create a few static textures, and then draw with them a lot. This model is sufficient for most of the several hundred thousand shipping apps. There isn't a great solution at the moment for fast streaming texture modification in iOS.

Desktop OpenGL has several solutions to this problem, including a buffering technique to stream texture data.
Quote this message in a reply
Apprentice
Posts: 13
Joined: 2011.02
Post: #20
Hmm... Your sample code is muddling up the link reference you gave on the Apple extension.

Here are the two codes:

glTexImage2D(GL_TEXTURE_2D, 0, GL_BGRA, w, h, 0, GL_BGRA, GL_UNSIGNED_BYTE, data);

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, w, h, 0, GL_BGRA, GL_UNSIGNED_BYTE, data);


The only difference is the <internalformat>, which is using I suppose standard OpenGL ES convention in the second instance. Also, I couldn't find GL_BRGA in the list of acceptable <internalformat> parameter options from: http://www.opengl.org/sdk/docs/man/xhtml...mage2D.xml



Also, from the spec at http://www.khronos.org/registry/gles/ext...RA8888.txt shouldn't the code look like this (note the <format> parameter)?

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, w, h, 0, GL_BGRA_EXT, GL_UNSIGNED_BYTE, data);

Or this (note the <internalformat> AND <format> parameters)?

glTexImage2D(GL_TEXTURE_2D, 0, GL_APPLE_texture_format_BGRA8888, w, h, 0, GL_BGRA_EXT, GL_UNSIGNED_BYTE, data);

Actually, what is confusing me is the term GL_APPLE_texture_format_BGRA8888.

Is GL_APPLE_texture_format_BGRA8888 the name of the <internalformat> parameter option?
Or is it just a name to describe the spec, and you still use GL_RGBA as the <internalformat>.

Also, would this also apply to OpenGL 2.0? I would like to stick to 2.0 because fragment shaders are supported. How would what you have stated above change in OpenGL 2.0? Same, but with the added parameter of course in 2.0 version.

Also, the GL_APPLE_texture_format_BGRA8888 implies 24bits plus alpha =32bits. Does this mean if you use this extension, you are stuck with 24bits? (ok by me though).

I am also confused by the 2n+2 border for width and height. Do they mean 2n + 2 * <border>, where n is a power of 2. Or do they mean literally 2n+2 only. Why is "border" in the definition.
In reference to: http://www.opengl.org/sdk/docs/man/xhtml...mage2D.xml

Another thing is: http://www.opengl.org/sdk/docs/man/xhtml...mage2D.xml

It doesn't have a parameter for <internalformat> in the function parameter definition. I am assuming you need glTexImage2D(), THEN use the glTexSubImage2D?

I am sorry, but to me this learning process has been overly complicated. Imagine a new guy coming to OpenGL, and must deal with a function definition requiring two similar inputs <internalformat> and <format> and deal with subtle and not subtle differences between OpenGL ES versions. I think in your reply it would be easier if you stick to one version: OpenGL ES 2.0. Especially code samples and links to specs. Otherwise I am afraid everything you have said won't work in 2.0.
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #21
(Mar 1, 2011 10:48 AM)edepot Wrote:  I couldn't find GL_BRGA in the list of acceptable <internalformat> parameter options
That's exactly the point. The IMG extension made BGRA a valid <internalformat>. In Apple's opinion this is wrong, because BGRA is not a base format and TexEnv is undefined. So the APPLE extension fixes this spec language to work like desktop OpenGL.

edepot Wrote:shouldn't the code look like this (note the <format> parameter)?
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, w, h, 0, GL_BGRA_EXT, GL_UNSIGNED_BYTE, data);
Yes, in ES it should be GL_BGRA_EXT. Good catch.

edepot Wrote:glTexImage2D(GL_TEXTURE_2D, 0, GL_APPLE_texture_format_BGRA8888, w, h, 0, GL_BGRA_EXT, GL_UNSIGNED_BYTE, data);
No, this won't compile. Extension strings aren't enums.

edepot Wrote:Also, would this also apply to OpenGL 2.0?
If the extension string is exported in an ES2.0 context (it is, on all iOS devices) then you can use it exactly the same way.

edepot Wrote:Also, the GL_APPLE_texture_format_BGRA8888 implies 24bits plus alpha =32bits. Does this mean if you use this extension, you are stuck with 24bits?
It doesn't imply it. It explicitly says it:
Code:
Section 3.6.2, add the following row to Table 3.3:
    
        Format Name    Element Meaning and Order    Target Buffer
        -----------    -------------------------    -------------
        BGRA_EXT       B, G, R, A                   Color

    Section 3.6.2, add the following row to Table 3.4:
    
        Format      Type             Bytes per Pixel
        --------    -------------    ---------------
        BGRA_EXT    UNSIGNED_BYTE    4




edepot Wrote:I am also confused by the 2n+2 border for width and height.
Desktop OpenGL supports the notion of a "border" texel. This is exactly what it sounds like-- a border running around the texture image. This is useful when you want to break a large image into a bunch of smaller textures (i.e. Google Maps) and then seamlessly filter across the tiles. See this explanation.

On old hardware that only supports power-of-two textures, the valid texture sizes are 2^n + 2 * <border>, where border is either zero or one. So a 16x32 texture without a border is valid. And an 18x34 texture with a border is valid.

OpenGL ES does not support border texels. So <border> must always be zero. (And if you look carefully, you can see in Google Maps on the iPhone that there are filtering discontinuities between tiles when you zoom in.)

edepot Wrote:Another thing is: glTexSubImage2D doesn't have a parameter for <internalformat>
Correct. TexImage allocates a texture image, defining the dimensions and format.
TexSubImage modifies an existing texture image.




edepot Wrote:I am sorry, but to me this learning process has been overly complicated.
What would make it easier? Was it any easier to lean C / Java / Cocoa / HTML ?

edepot Wrote:I think in your reply it would be easier if you stick to one version: OpenGL ES 2.0.
The reality is that there is a bunch of different hardware in customer's hands. The hardware has different capabilities. There are different versions of GL to expose those capabilities.

You should decide what hardware you are targeting. Then decide what API features you need. Then decide what version to use.
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  Need help with pixel oriented framebuffer edepot 24 19,015 Mar 3, 2011 07:15 AM
Last Post: mariocaprino
  Fastest sprite method markhula 9 6,348 Sep 24, 2010 08:04 AM
Last Post: markhula
  Fastest Possible Blending Function, OpenGL ES 2.0 Macmenace 12 15,076 Apr 16, 2010 11:44 AM
Last Post: Macmenace
  rendering framebuffer texture not working lookitsash 2 3,870 Oct 30, 2009 09:49 AM
Last Post: arekkusu
  Screenshot of iPhone screen Carlos Camacho 1 2,575 Jun 6, 2009 06:56 AM
Last Post: Najdorf