art with code


Easy 3D on the web

Problem: can't do 3D graphics on the web. Solution: WebGL. New problem: no, I mean, I want to put this 3D model onto a web page. Solution: Sketchfab / Three.js / Babylon.js / ShaderToy. New new problem: I need to download libraries and code stuff or host the files on a SaaS or develop a third brain to model using modulo groups of signed distance fields moving along Lissajous curves.

Could easy 3D be part of the web platform? And how? With images, the solution was simple. The usual image file is a 2D rectangle of static pixels and all you really needed to figure out was how to layout, size and composite it with regards to the rest of the web page. When you step outside of simple static 2D images, all hell breaks loose.

Animated GIFs have playback timing and that's not so easy to figure out, and so animated GIFs are pretty broken. Videos need playback controls with volume and scrubbing, potential subtitle tracks, and a bunch of codecs both on the video and audio sides, so they were pretty broken as well. (Then YouTube started using HTML5 videos because mobiles don't do Flash. And magically the video issues were fixed~)

SVGs also need to handle input events and have morphed from "umm, put this in an embed and it'll maybe work" to their current (pretty awesome!) state where an SVG can be used as a static image, an animated image, embedded document and an inline document. Something for everyone!

Displaying a 3D model on a web page is a bit like mixing SVG with video elements. You'd like to have controls to turn the model around - it is 3D after all. And you'd like to have the model animate - again, it's 3D and 3D's good for animations. It'd be also nice to have the model be interactive. I mean, we were promised a 3D data matrix by a bunch of fiction writers in the 80s, and that's going to require some serious animated 3D model clicking action (to be fair, they also promised us a thermonuclear war, but let's not go there right now.)

So. 3D model. Web page. <3d src="castle_in_the_sky.3d" onclick="event.target.querySelector('#gate').classList.add('open')"> Right?

What file format do you standardize on? How do you load in textures and geometry? How do you control the camera? How do you do shaders for the materials? How do you handle user interaction? What are the limits of this 3D model format? How do you make popular 3D packages output this file format (plus animations, plus semantic model for interactivity)? How do you compress the thing and progressively stream it?



I wrote a small painting program on ShaderToy, using the new multipass rendering feature. It's called ShaderPaint and it was a lot of fun to write.

The fun part about writing programs in ShaderToy is the programming model it imposes on you. Think of it as small pixel-size chunks of memory wired together in a directed graph. Each chunk of memory runs a small program that updates its contents. The chunks are wired together so that a chunk can read in the contents of chunks that it's interested in. The read gives you the contents on the last time step, so there's no concurrent crosstalk happening.

This is... rather different. Your usual programming model is all about a single big program, executing on a CPU, reading and writing to a massive pool of memory, modifying anything, anywhere, at any time. To go from that to ShaderToy's memory-centric approach is a big shift in perspective. Instead of thinking in terms of "The program is going to do this and then it's going to do that and then it's going to increment that counter over there.", you start to think like "If the program is running on this memory location, do this. If the program is running on that memory location, do that. If the program is running on the counter's memory location, increment the value." You go from having a single megaprogram to an army of small programs, each assigned to a memory location.

In the figure above, I've sketched the data flow of an output pixel. First, the UI inputs coming from the shader uniforms modify the UI state layer, which is read by the stroke layer to draw a brush stroke on it. The stroke layer is then composited with the draw layer when the brush stroke has ended. The final step is to composite the stroke layer and draw layer onto the output canvas, and draw the UI controls on it, based on the values on the UI state layer.


Flat cameras

Well, this is really cool. It's a flat camera. A camera sensor with a special mask in front of it that makes it possible to computationally deconvolve the sensor data into an image. Imagine credit-card-slim mobiles and mobile cameras with a very large image sensor for shooting in low light. Or front-facing cameras embedded into the display.

On a similar vein, this Light L16 thing looks pretty nifty: jam 16 small cameras into a smartphone and do computer vision magic to treat the ensemble as a single big sensor.


Display tech

I was playing with a Kindle the other day. It's got an e-ink display that's quite different from the usual mobile displays. For one, it works as a reflective display, much like the pages of a paper book. That means that you can read it without a backlight, unlike the LCD display you may have in your phone. The second big difference is that it's black & white only. Third, the screen can stay on and display an image without using any power. And finally, the refresh rate is very slow. So what's going on here? Why does it work that way?

E-ink displays have a tiny capsule for each pixel. This capsule is filled with black ink particles and white ink particles. The different colors have different charges. There's a charged plate underneath the capsule that can change the sign of its charge so as to attract one color and repel the other.

Let's say that you have black particles with a positive charge and white particles with a negative charge. When you set the charged plate to a positive charge, the white particles fly towards the charged plate at the bottom of the capsule and the black particles are pushed to the top of the capsule. As you're looking at the display from above, you see that the pixel turns black.

When you flip the charge, the black particles are drawn down to the bottom and the white particles are pushed to the top. The pixel now turns white.

Because the pixels use ink particles, the way they interact with light is much like you'd see with a regular sheet of paper. Light hits the ink at the top of the screen and bounces away towards your eyes. The black ink particles absorb more of the light than the white ones, and you see the resulting contrast. Nice and simple. But kinda slow, as the charging plate needs to flip its charge and the physical ink particles need to swap places for the pixel to change color.

By comparison, LCDs are quite a different beast. An LCD pixel is made out of three layers of material that polarize light. Polarized light is light where all the photons are vibrating in the same direction. In unpolarized light, the photons are vibrating in every which direction. To make polarized light, you either need a light source that produces polarized light or a filter that blocks photons that are vibrating in the wrong direction.

If you put two polarizing filters on top of each other, they do a bit of a trick. If the polarized filters are pointing to the same direction, they let light through. As you rotate them relative to each other, they let less and less light through, and once they're rotated to a 90 degree angle, they don't let any light through. What if you could change this rotation angle with electricity?

That's pretty much what an LCD pixel does. It's got a liquid crystal layer that rotates the polarization of light depending on how much electricity you put through it. To make the trick complete, it has two polarizing filters below and above the liquid crystal layer. The polarizing filters are at a 90 degree angle to each other, so when the liquid crystal layer is in the unrotating state, the pixel doesn't let any light through and appears dark. When the liquid crystal is set to rotate incoming light by 90 degrees, light passes through the pixel, making the pixel look bright.

Usually an LCD screen has a bright light behind it. The LCD in front blocks some of the light, creating the image that you see. Because the filters in the LCD are not perfect, some of the light hitting the dark pixels seeps through, making the darks brighter than ideal. Additionally, the black pixels are lit from behind at all times at the same intensity as the brightest white pixels, which makes LCDs relatively power-hungry. Because the liquid crystals can twist rapidly, LCDs have good refresh rates and work for displaying moving graphics.

OLEDs. OLEDs are tiny tiny lights, printed on the screen surface. The brightness of each of lamp is controllable by the amount of electricity fed to it. Because they're tiny individual lights, they are more power-efficient than LCDs as dark pixels don't use any energy at all. As each pixel is a tiny lamp, and black pixels are lamps that are turned off, the black levels of OLEDs are better than with LCDs. But because each pixel is a tiny lamp surrounded by circuitry, the total brightness achieved by an OLED display is worse than an LCD that can use a large powerful light behind the screen.

Ink, filters and lamps. That's what powers your mobile displays. Could you have something else? Maybe tiny waveguides that absorb a certain color of light - like the ones on butterfly wings - and have a controllable level of absorption? Something DLP-like where each display pixel is a mirror that either reflects incoming light back towards the viewing surface or away from it?


Faster than light!

Spaceships traveling faster than the speed of light! What would that look like? Pretty weird.

Suppose you're on planet B, one light-year away from planet A. One day, a spaceship appears from nowhere.

"What the f?!", you say, as the spaceship splits in two. One of the ships just sits there, doing whatever ships do when they arrive on planet B and encounter incredulous aliens. The other ship flies at 12 times the speed of light back towards planet A. You immediately warn planet A about the crazy spaceship about to crash into them. A few moments later you figure out that none of your messages traveling at the speed of light are going to arrive in time. Oh well, they won't know what hit them.

A month later, after your savage primitive surface-dwelling population has been subjected to enforced planetary volumization by the spaceship, you look up from your grid-cell and gaze through the impenetrable volume-grid that used to be your planet. Somewhere there, a lightyear away, on now-certainly-doomed planet A, your virtual telescopic eyes see the spaceship approaching planet A. It seems to be rapidly decelerating on its approach.

You keep track of the spaceship from your grid-cell. At this rate of deceleration, planet A might actually receive your warning in time. There is a flicker of hope growing in your mind.

Months tick by and the spaceship keeps decelerating. Soon it'll be moving slower than lightspeed and your warning will overtake it. Excellent. Give it a kicking, planet A!

Nearly a year after sending the message, you take a closer look at planet A. They should receive the warning any day now. But too late! There already is an identical spaceship on orbit around planet A. And it seems to be moving towards its incoming twin in greeting.

Perhaps a little too much towards it, in fact. As the spaceships get closer and closer, they suddenly merge together and disappear! Planet A is saved by the incompetent pilots who crashed their ships and disappeared from spacetime for good. You silently cheer for the thus-saved inhabitants of planet A.

The End...?



Plot twist: the spaceship originally came from planet A and there was only one of them.

Plot tangle: what the heck happened inside the spaceship to maintain "c is always c, regardless of the reference frame"?


HTTPS and HTTP2 on Apache2 with Let's Encrypt

This morning I figured I'd set up HTTPS and HTTP2 on my web server. It was pretty easy, too. And man, HTTP2 is fast, especially on silly sites like mine that have a large amount of small images on the page. Good riddance to sprites.

Here's how I set up my Ubuntu Apache2 web server for HTTPS and HTTP2:

For starters, let's get a HTTPS cert. You can get one for free using Let's Encrypt, a non-profit certificate authority from the US. It has an automagical command line tool that creates certs for you and registers them with the CA. It can even automate installation for Apache. Sadly, my Apache config didn't work with the automatic tool, so I had to do it manually. Which wasn't too bad either.

First, I shut down the Apache web server with sudo service apache2 stop. Then, I used the Let's Encrypt client to fetch the cert (this needs to be run on the server pointed to by the domain name):

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
./letsencrypt-auto certonly --standalone -d MY.DOMAIN.NAME

If everything goes well, you should now have the certificate files in /etc/letsencrypt/live/MY.DOMAIN.NAME/. To get HTTPS running, I edited my Apache2 configuration to set up the SSL module and use it for my domain.

<VirtualHost *:443>
  ServerAlias MY.DOMAIN.NAME

  SSLEngine on
  SSLCertificateFile "/etc/letsencrypt/live/MY.DOMAIN.NAME/cert.pem"
  SSLCertificateKeyFile "/etc/letsencrypt/live/MY.DOMAIN.NAME/privkey.pem"
  SSLCertificateChainFile "/etc/letsencrypt/live/MY.DOMAIN.NAME/chain.pem"


Ok, HTTPS working. Let's do HTTP2 now. If you haven't yet, you need to upgrade your Apache to version 2.4.17 to get HTTP2 support. Older versions of Ubuntu don't have Apache 2.4.17, so you may need to add a custom PPA to your software sources with sudo add-apt-repository ppa:ondrej/apache2 or such.

After upgrading Apache, turn on the HTTP2 module with sudo a2enmod http2. Almost there! The last step is to turn on HTTP2 on our HTTPS virtual host by adding h2 to the Protocols directive. I also turned on the H2Direct directive, as the description said that it'll spare the server from upgrading a HTTP/1.1 connection if the client starts talking HTTP2.

<VirtualHost *:443>
  ServerAlias MY.DOMAIN.NAME

  SSLEngine on
  SSLCertificateFile "/etc/letsencrypt/live/MY.DOMAIN.NAME/cert.pem"
  SSLCertificateKeyFile "/etc/letsencrypt/live/MY.DOMAIN.NAME/privkey.pem"
  SSLCertificateChainFile "/etc/letsencrypt/live/MY.DOMAIN.NAME/chain.pem"

  Protocols h2 http/1.1
  H2Direct on


That's it! Now turn on Apache again with sudo service apache2 start and you should have HTTP2 running. You can check for it in Chrome DevTools by going to the Network pane, right-clicking on the columns header and turning on the Protocol column.

Thanks for reading! Hope this helps you getting your site up and running on HTTP2.


MM2 - part 2 - React + Meteor + PostgreSQL

Finally got the Meteor postgres-packages react-todos example working with user accounts and my dataset. But my dataset is 67k rows, which completely obliterates the app. I benchmarked Knex to see if it can handle a query result of that size. The "select * from items;" query takes around 2.3 seconds. Doing a pg_dump of the table takes 0.8 seconds. Running cat * > foo on the table files takes 0.7 seconds. Running grep on the table files takes around 2.7s, so the Knex result is pretty good.

And I also found out that PostgreSQL notify fails if your total notification size is large enough. The Meteor Postgres integration creates triggers that send over the entire updated row, which would be a decent idea if there wasn't a notification size limit. As it is, probably need to hack Meteor to include only the changed row id in the notification and do a select to get the row contents.

Still don't fully understand React. I think you create custom elements and hook them all together and magic happens. And then the app hangs trying to update your 70k item list every 3 seconds, and you go "Hmm. We need to go deeper."

Currently gravitating towards using Meteor + React as the outer shell for the app and for detail views and other non-insane things. For the things that need to deal with the item list, I'll brew my own cauldron of madness.

And Meteor seems to lose my logged in status on server restarts. It seems to be specific to the PostgreSQL version, as the MongoDB one doesn't suffer from that. [edit] Tracked it down to accounts-base/os/accounts_server.js deleting all user sessions on startup. I can't understand why it would do that, and the comment doesn't make it any clearer.

About Me

My Photo

Built art installations, web sites, graphics libraries, web browsers, mobile apps, desktop apps, media player themes, many nutty prototypes, much bad code, much bad art.

Have freelanced for Verizon, Google, Mozilla, Warner Bros, Sony Pictures, Yahoo!, Microsoft, Valve Software, TDK Electronics.

Ex-Chrome Developer Relations.