art with code

2009-03-05

The anatomy of the Canvas 3D extension

Been fiddling around with Firefox's Canvas 3D extension for the last week. Canvas 3D adds an OpenGL ES 2.0 context to the HTML5 Canvas element, giving you access to a few hundred GFLOPS of graphics computing power.

I've been working on adding framebuffer objects, glReadPixels, getImageData, toDataURL and a test suite to the extension. And it's a bit hostile to one's sanity - as OpenGL isn't very good at reporting errors - but what can you do?

It's been educational though. Here's a small overview of the way the extension works:

Organization of the extension code


The code for the extension is split into five major bits, outlined below.

C++ wrapper around OpenGL


  • src/glwrap.h
  • src/glwrap.cpp

Implement the GLES20Wrap-class, which wraps the OpenGL shared library by loading the OpenGL ES 2.0 symbols from the shared object (e.g. /usr/lib/libGL.so) in much the same way as GLEW.

Platform-specific GLPbuffer implementations


  • src/nsGLPbuffer.h
  • src/nsGLPbufferGLX.cpp
  • src/nsGLPbufferAGL.cpp
  • src/nsGLPbufferWGL.cpp
  • src/nsGLPbufferOSMesa.cpp

These set up the rendering context for the canvas, deal with resizing it, and implement a SwapBuffers() that uses glReadPixels() to read the current framebuffer contents into the Thebes surface for the nsGLPbuffer.

The Thebes surface is then used for drawing the canvas element on the page, and also provides image data for getImageData and toDataURL (Thebes is the Firefox rendering engine, essentially a Cairo backend wrapper with heavily extended text capabilities.)

Platform-independent plumbing for dealing with the nsGLPbuffer


  • src/nsCanvasRenderingContextGL.h
  • src/nsCanvasRenderingContextGL.cpp

The class nsCanvasRenderingContextGLPrivate (I'll call it "ContextGL" from here on) stands between the browser and the OpenGL wrappers described above. ContextGL implements the <canvas> element side of the GL canvas.

When you create a new GL canvas context, ContextGL creates a nsGLPbuffer and binds it to the canvas context in the SetCanvasElement-method.

When you resize the canvas, ContextGL calls the nsGLPbuffer's Resize-method.

When the browser redraws the document, it calls ContextGL's Render-method to draw the GL framebuffer (the Thebes surface mentioned above) onto the browser window.

The DoSwapBuffers-method is called by gl.swapBuffers() and prompts a redraw of the document (by invalidating the canvas element.)

And the GetInputStream-method is used by canvas.toDataURL() to encode the canvas contents into e.g. a PNG image.

C++ implementation of the JavaScript OpenGL context interface


  • src/nsCanvasRenderingContextGLWeb20.cpp

If ContextGL above was the implementation of the canvas element, ContextGLWeb20 is the implementation of the moz-glweb20 drawing context. It wraps the C++ OpenGL wrapper into a JavaScript library, defined in ContextGLWeb20.idl below.

Most of ContextGLWeb20 is pretty straightforward translation (in fact, a large part is defined by one-liner macros such as GL_SAME_METHOD_1(UseProgram, UseProgram, PRUint32)), but anything that deals with arrays, pointers and indices (genTextures etc. gen*, buffers, textures, vertexAttribPointer, uniform*, readPixels, getImageData) needs to cast values between JS and C++, and do bounds-checking (or should, at least.)

There are also a few methods that implement a higher-level interface over the basic OpenGL functions, e.g. gl.uniformf(some_uniform, [1.0, 2.0, 3.0, 4.0]) is turned internally into glUniform4fv(some_uniform, 1, arr).

In terms of API additions, the only truly new method is gl.texImage2DHTML(tex_id, image_or_canvas_element) for using HTML images and canvases as textures.

JavaScript interface definitions


  • src/nsCanvas3DModule.cpp - the extension module setup
  • public/nsICanvasRenderingContextGL.idl - GL constants
  • public/nsICanvasRenderingContextGLWeb20.idl - GL functions

The IDL files work sort of like header files shared between JavaScript and C++, basically saying "Hey, these are the JavaScript methods of the GL context, you better have an implementation for them in your C++ class!"

For example, if you have void useProgram (in PRUint32 program); in the IDL, you need NS_IMETHODIMP nsCanvasRenderingContextGLWeb20::UseProgram(PRUint32 program) {...} in the cpp.

Some performance numbers


The Canvas 3D is a bit of an odd beast performance-wise, as it's hobbled by Cairo on one side and JavaScript on the other.

E.g. on my computer, doing a 30 fps animation of a 400x400 canvas uses something like half of a single core. The CPU time breakdown is ~10% for JS matrix math, another 10% for premultiplying the pixels in SwapBuffers, 30% for GL calls, and 50% for Cairo drawing the GL framebuffer on the HTML document.

In case you're interested, the animation draws a spinning per-pixel lit cube with a depth blur done using 6 gaussian blur passes. And a premultiply-unpremultiply-pass to make alpha work ok with blur. (OGG video)

And JavaScript. Well. I did a small benchmark, with a 7x7 gaussian blur kernel over a 256x256 Firefox logo (decomposed into a horizontal blur and a vertical blur.) JavaScript took 0.8 seconds to do a single blur. With GLSL, it took 0.4 seconds to do a thousand blurs.

Yes, that's two thousand times faster. And this on a 3-year-old Geforce 7600 GS that I bought because it was cheap, had two DVI outs and passive cooling.

So, if you want good performance, push as much of your number crunching to the shaders as you can, and rewrite Firefox's graphics engine to use OpenGL for compositing.

7 comments:

Robert said...

A GPU-based cairo backend is the holy grail, yes.

I think the big issue blocking Canvas3D from reaching the masses is security against malicious content. Interested in doing some fuzz testing of GLSL programs?

Ilmari Heikkinen said...

Yes, fuzz tests for GLSL would be nice. I'll add them to my to-do.

How does one exploit GLSL then?

A success would be drawing data on the screen not owned by the current GL context, out-of-bounds data read from CPU RAM, crashing the video driver or hardware, DoS against the video card. Maybe browser DoS, but that can be achieved just by spamming a lot of divs or 1Mpx canvases. Enough content and composition will take minutes or the computer run out of memory.

So, driver crashes and out-of-bounds reads?

The attribute arrays and textures are interpolated and given one at a time, and their sizes are known at compile-time (ditto for uniforms and arrays defined in the shader.) And accessing beyond them errors in the shader compilation phase. I guess the compiler does its own fuzz testing.

One could do while (1) {}, but at least Nvidia's driver refuses to compile that. And trying to do a long loop failed to compile at 2.2 million iterations.

I'll try to write a bunch of nasty shaders and put them in canvas3d-tests.

Ilmari Heikkinen said...

given one at a time -> given to the shader one at a time

Ilmari Heikkinen said...

Re: compiler fuzz testing, I tried to read from a float array using a computed index that goes beyond the array bounds. Error at compile phase.

buzzilo said...

3D content is obviously cool thing, but what about 2D content? Will HTML canvas work on top on OpenGL power sometime?

Ilmari Heikkinen said...

The Opera, Safari and Chrome canvases do use OpenGL. At least Chrome's does, as they have a blending bug because they use GL_ONE or something for the 'lighter' blend mode.

Ilmari Heikkinen said...

(And I don't personally hold much hope for Cairo getting magically fast. It is slow, it has been slow, and - by extrapolation - it will be slow.)

Blog Archive