(Intro)

Hank: In any story that takes place in a nebulous high-tech future, there always seems to be some sort of augmented reality. Maybe it's an engineer inspecting a floating 3D model of a high-tech gadget, or a team of scientists working away at a projection of a computer screen. Or even full-sized holograms of people. So, to propel SciShow into this future, this episode has two hosts... kinda.

Now this digital version of Michael is current hologram technology - basically, recording a 3D video of someone or something and dropping it in a new environment. In this case, in the SciShow studio. This Michael Aranda hologram is a 3D video of him that we can watch using Microsoft's HoloLens device: It's basically a self-contained, powerful computer that projects holographic images onto transparent lenses in front of your eyes, so they seem like they're in the room with you.

It's not quite the same as a stand-alone holographic image - like R2-D2's recording of Princess Leia in Star Wars. But it is a 3D video that you can watch from all different angles. It's a type known as a free-viewpoint video. And it took a lot of advanced technology and creative thinking to make it - so Microsoft gave us a sneak peek at everything that goes into creating and viewing one of their holograms.

To make a 3D video, you need to know a lot about the scene you're trying to capture: Specifically, you need to know what it looks like from every angle, and how everything's arranged in 3-dimensional space. Individual video cameras can capture a lot of visual information, like colors and textures. But they don't tell you much about depth - the 3D shape of a thing, and how it's moving and changing in space.

Traditional motion capture is what's happening in behind-the-scenes footage from movies, where actors are wearing those special suits, or have weird dots on their faces. That equipment helps special cameras track every precise movement that they make. But motion capture alone can't give you the other kinds of visual information - like color and texture.

So, the HoloLens team created a special studio setup with 3D capture technology to record both types of information at once: the appearance of the performer, as well as how things are arranged and moving in 3D space. Using their setup, they can create a 3D video based on the exact performance.

And here's how the magic happens: There's a circular green screen stage with a performance space a couple meters in diameter. It's surrounded by around 100 synchronized video cameras pretty much on all sides. By combining images from all these cameras, computers can figure out where something is in 3D space and what it looks like, and then use that data to generate a 3D holographic image.

Half of cameras are RGB cameras, which form an image based on visible light. Their main job is to capture accurate colors and the general outlines of the performer. The other half are IR cameras, which capture the infrared light reflected by the objects in the scene to map out depth. These cameras help generate most of the 3D figure and capture the texture of things like faces, fabrics, or other props.

The team calibrates all these cameras, and then, it's time to record - things like actors, or animals, or whatever you want to digitize to help tell a story. The cameras then send all their synchronized footage - which is a lot of raw video data - to be processed into one compact 3D video file.

The central computer does a few things: It turns the footage from all the different camera perspectives into a single 3D figure. But that 3D model includes so much detail that something like a mobile device wouldn't be powerful enough to play it back. So, their computer software simplifies the model, keeping a lot of detail in important places like hands and faces, but taking away details where you might pay less attention. Then, it adds colors to the image. Finally, it produces a single, compact 3D video file that's small and simple enough to be played back on a computer, or streamed over Wi-Fi to something like the HoloLens. And there you have it: your very own hologram that can be put into movies - or even your own environment - to create an augmented reality.

For now, there's a limit to how much detail this technology can capture: For one thing, it's hard to generate good holograms of translucent material - like water, or glass - or finely-detailed fabric like lace. In those cases, capturing depth becomes much more complicated. And you do need a fancy studio, lots of computing power, and advanced technology to create these holograms in the first place. But the hope is that eventually, it won't be too hard to make your own holograms. Maybe it'll even be as simple as picking up a camera and making a YouTube video.

Michael: Many thanks to Microsoft and Actiongram for letting us play around in your studio, and for turning me into a hologram.

And thanks to you for watching this episode of SciShow, which was brought to you by our patrons on Patreon. If you want to help support the show, just go to patreon.com/scishow. And don't forget to go to youtube.com/scishow and subscribe!