Live two-camera video stream from Raspberry Pi to Oculus Rift

I’ve been playing around with the Oculus Rift on and off since the DK1 came out two years ago. Now I finally took the time to make my own live-streaming rig. Here’s a quick video:

It might not look like much. But it captures 180°x120° of video in HD, encodes it in h264 and sends it out over IP, live to a PC which displays it in the Oculus Rift. It gives a good teleportation presence. The video quality is surprisingly good (the wide-angle lenses are a huge improvement over the stock pin-hole lenses of the raspberry pi cameras).

The Raspberry Pi came out in February 2012 so it is no surprise that the BCM2835’s CPU is very far from being capable of live-encoding and streaming video.

But the VideoCore GPU has some nice tricks up its sleeve. The GPU has direct connections to cameras (CSI) and offers hardware accelerated video encoding to h264. So when it comes to live streaming video with low latency, the Raspberry Pi is actually not a bad choice. For this project I chose the “Compute Module” version, which is made for embedding into things and offers two camera connectors.

For software on the Pi I hacked together a solution where raspivid pipes video to gstreamer which sends it out over RTP to the PC. This is run in two separate instances, four processes in all. I think there is a lot of room for improving the latency in how streams are handled on the CPU. On the PC side this project would have been impossible without my friend Jonathan who has a neat code base called Oden from his startup Voysys that is built for exactly this - taking live streams and putting them in the Oculus Rift with low latency. They’ve been building on this version of it for 12 months+. We hacked together that part of it Friday night (explains the choice of a wine glass for camera stand, if anybody wondered…).

The next thing I want to do is to make the streaming setup mobile. I have a 3.5A 5V regulator, a few LiPo:s, a usb WiFi stick and a 3D-printer - the plan seems pretty clear on how to proceed. I also have a quadcopter… :)