art with code

2018-02-28

Building a storage pyramid

The office IT infrastructure plan is something like this: build interconnected storage pyramids with compute. The storage pyramids consist of compute hooked up to fast memory, then solid state memory to serve mid-IOPS and mid-bandwidth workloads, then big spinning disks as archive. The different layers of the pyramid are hooked up via interconnects that can be machine-local or over the network.

The Storage Pyramid

Each layer of the storage pyramid has different IOPS and bandwidth characteristics. Starting from the top, you've got GPUs with 500 GB/s memory, connected via a 16 GB/s PCIe bus to the CPU, which has 60 GB/s DRAM. The next layer is also on the PCIe bus: Optane-based NVMe SSDs, which can hit 3 GB/s on streaming workloads and 250 MB/s on random workloads (parallelizable to maybe 3x that). After Optane, you've got flash-based SSDs that push 2-3 GB/s streaming accesses and 60 MB/s random accesses. At the next level, you could have SAS/SATA SSDs which are limited to 1200/600 MB/s streaming performance by the bus. And at the bottom lie the HDDs that can do somewhere between 100 to 240 MB/s streaming accesses and around 0.5-1 MB/s random accesses.

The device speeds guide us in picking the interconnects between them. Each HDD can fill a 120 MB/s GbE port. SAS/SATA SSDs plug into 10GbE ports, with their 1 GB/s performance. For PCIe SSDs and Optane, you'd go with either 40GbE or InfiniBand QDR, and hit 3-4 GB/s. After the SSD layer, the interconnect bottlenecks start rearing their ugly heads.

You could use 200Gb InfiniBand to connect single DRAM channels at 20 GB/s, but even then you're starting to get bottlenecked at high DDR4 frequencies. Plus you have to traverse the PCIe bus, which further knocks you down to 16 GB/s over PCIe 3.0 x16. It's still sort of feasible to hook up a cluster with shared DRAM pool, but you're pushing the limits.

Usually you're stuck inside the local node for performance at the DRAM-level. The other storage layers you can run over the network without much performance lost.

The most unbalanced bottleneck in the system is the CPU-GPU interconnect. The GPU's 500 GB/s memory is hooked to the CPU's 60 GB/s memory via a 16 GB/s PCIe bus. Nvidia's NVLink can hook up two GPUs together at 40 GB/s (up to 150 GB/s for Tesla V100), but there's nothing to get faster GPU-to-DRAM access. This is changing with the advent of PCIe 4.0 and PCIe 5.0, which should be able to push 128 GB/s and create a proper DRAM interconnect between nodes and between the GPU and the CPU. The remaining part of the puzzle would be some sort of 1 TB/s interconnect to link GPU memories together. [Edit] NVSwitch goes at 300 GB/s, which is way fast.

The Plan

Capacity-wise, my plan is to get 8 GB of GPU RAM, 64 GB of CPU RAM, 256 GB of Optane, 1 TB of NVMe flash, and 16 TB of HDDs. For a nicer-cleaner-more-satisfying progression, you could throw in a 4 TB SATA flash layer but SATA flash is kind of DOA as long as you have NVMe and PCI-E slots to use -- the price difference between NVMe flash and SATA flash is too small compared to the performance difference.

If I can score an InfiniBand interconnect or 40GbE, I'll stick everything from Optane on down to a storage server. It should perform at near-local speeds and simplify storage management. Shared pool of data that can be expanded and upgraded without having to touch the workstations. Would be cool to have a shared pool of DRAM too but eh.

Now, our projects are actually small enough (half a gig each, maybe 2-3 of them under development at once) that I don't believe we will ever hit disk in daily use. All the daily reads and writes should be to client DRAM, which gets pushed to server DRAM and written down to flash / HDD at some point later. That said, those guys over there *points*, they're doing some video work now...

The HDDs are mirrored to an off-site location over GbE. The HDDs are likely capable of saturating a single GbE link, so 2-3 GbE links would be better for live mirroring. For off-site backup (maybe one that runs overnight), 1 GbE should be plenty.

In addition to the off-site storage mirror, there's some clouds and stuff for storing compiled projects, project code and documents. These don't need to sync fast or are small enough to do so.

Business Value

Dubious. But it's fun. And opens up possible uses that are either not doable on the cloud or way too expensive to maintain on the cloud. (As in, a single month of AWS is more expensive than what I paid for the server hardware...)

2018-02-27

Figments

Ultraviolet Fairies


"Can you make them dance?", Pierre asked. Innocent question, but this was a cloud of half a million particles. Dance? If I could make the thing run in the first place it would be cause for celebration.

The red grid of the Kinect IR illuminator came on. Everything worked perfectly again. Exactly twelve seconds later, it blinked out, as it had done a dozen times before. The visiting glass artist wasn't impressed with our demo.

Good tidings from France. The app works great on Pierre's newly-bought Acer laptop. A thunderhead was building in the horizon. The three-wall cave projection setup comes out with a wrong aspect ratio. I sipped my matcha latte and looked at the sun setting behind the cargo ships moored off West Kowloon. There's still 20 hours before the gig.

The motion was mesmerizing. Tiny fairies weaving around each other, hands swatting them aside on long trajectories off-screen. I clenched my fist and the fairies formed a glowing ring of power, swirling around my hand like a band of living light. The keyboard was bringing the escaping clouds to life, sending electric pulses through the expanding shells of fairies knocked off-course.

Beat. The music and Isabelle's motion become one, the cloud of fairies behind her blows apart from the force of her hands, like sand thrown in the air. Cut to the musicians, illuminated by the gridlines of the projection. Fingers beating the neon buttons of the keyboard, shout building in the microphone. The tension running through the audience is palpable. Beat. The flowing dancer's dress catches a group of fairies. Isabelle spins and sends them flying.

The AI


A dot. I press space. The dot blinks out and reappears. One, two, three, four, five, six, seven, eight, nine, ten. I press space. The dot blinks out and reappears. Human computation.

Sitting at a desk in Zhuzhou. The visual has morphed into a jumble of sharp lines, rhythmically growing every second. The pulse driving it makes it glow. Key presses simulate the drums we want to hook up to it. Rotate, rotate, zoom, disrupt, freeze. The rapid typing beat pushes the visual to fill-rate destroying levels and my laptop starts chugging.

Sharp lines of energy, piercing the void around them. A network of connections shooting out to other systems. Linear logic, strictly defined, followed with utmost precision. The lines begin to _bend_, almost imperceptibly at first. A chaotic flow pulls the lines along its turbulent path. And, stop. Frozen in time, the lines turn. Slowly, they begin to grow again, but straight and rigid, linear logic revitalized. Beginning of the AI Winter.

The fucking Kinect dropped out again! I start kicking the wire and twisting the connector. That fucker, it did it once in the final practice already, of course it has to drop out at the start of the performance as well. Isabelle's taking it in stride, she's such a pro. If the AI doesn't want to dance with her, she'll dance around it. I push myself down from my seat. How about if I push the wire at the Kinect end, maybe taping it to the floor did ... oh it works again. I freeze, not daring to move, lying on the floor of the theater. Don't move don't move don't move, keep working you bastard! The glowing filter bubbles envelop Isabelle's hands, the computer is responsive again. Hey, it's not all bad. We could use this responsiveness toggle for storytelling, one more tool in the box.

We're the pre-war interexpressionist movement. Beautiful butterflies shining in the superheated flashes of atomic explosions.

2018-02-11

Office build, pt. 2

With pretty much all the furniture built (apart from fixing a shelf and a filing cabinet to a wall), the office is getting close to version 1. That is, more than half empty and running at 10% utilization.

What do I mean? Well, what I've got in there now are two big desks and a table, two office chairs and two random chairs. Add in a filing cabinet, a shelf and a couple of drawers, and .. that's it. There's a room in the back that's going to be the demo build area / VR dev area. And probably where the servers and printers are going to live.

As it is now, the main office has space for 3-4 more big desks. I'll haul one desk from home there. And that's it for now. Room to expand.

If you're looking for a place to work from in Tsuen Wan, Hong Kong, come check it out. Or if you want to work on building awesome interactive generative visuals -- fhtr.org & fhtr.net -style but with more art direction, interactivity and actual clients. Sales genius, UI developer and 3D artist could use right away.


2018-02-09

Office build

Recently I've been setting up an office in Hong Kong, a.k.a. setting money on fire. Generating heat that enables us to set up installations in-house, tackle bigger projects and expand the company from its current staff of 1.25. Good plan? Great plan!

Very IKEA, much birch veneer. Spent a couple enjoyable days building furniture, almost done for now. Plan is to get a big board in front of the office's glass door and mount a big TV on it to run a demo loop of interactive content for the other tenants in the building to enjoy.

Also planning some network infrastructure build. The place has a wired network from the previous tenant, with five Cat5e cables strung along the walls, which I hooked up to an 8-port gigabit switch. There are another five cables running inside the walls that terminate at ethernet sockets, but those seem to be broken somehow.

Getting gigabit fibre internet there next week, which was surprisingly affordable. Like, $100/month affordable. For WiFi, I got a small WiFi router, 802.11ac. It's all very small-time setup at the moment.

So hey, how about some servers and a network upgrade? Get a couple of used old Xeon servers, fill them with SSDs and high-capacity drives for a proper storage tower. Run a bunch of VMs for dev servers and NAS. And, hmm, how about QDR InfiniBand cards and cables for 32Gbps network speed between servers and wired workstations, with the GigE and WiFi for others. Sort of like a 2012 supercomputing node setup. The best part? All of this kit (apart from the new drives) would be around $1500, total.

That said, I'll start with one server and a direct connection to the workstation. It'll keep the budget down to $400ish and let me try it out to see if it actually works.

Next to find the parts and get that thing built and running.

And um, yeah, do some sales too.




Here are some recent visuals done with our friends and partners at cityshake.com and plot.ly. For awesome sites and corporate app development in Austria, contact raade.at, who we have also worked with on various projects. Interested? Of course you are!

Mail me at my brand spanking new email address hei@heichen.hk and buy some awesome visuals or a fancy interactive office/retail/hotel installation or a fabulous fashion show. Or your very own VM hosted by a DevOps noob on obsolete hardware. I'm sure a VM can't catch on fire. Or... can it?

Blog Archive