Last week, we shared Oculus Research’s recent work on focal surface displays, which will be presented at SIGGRAPH 2017. Today, we’ll take you behind the scenes with the team who made it possible.

When Nathan Matsuda first joined forces with Oculus Research Scientists Alexander Fix and Douglas Lanman, little did the team know that the research project they were about to embark upon would span two years. For Lanman, who doubles as the computational imaging lead, that makes it one of the longest-running projects in his career as a professional researcher—not that he was particularly surprised.

“Being a researcher is a bit like a mystery novel—you’re constantly trying to guess the ending of how this or that technology will play out when we put it all together in a prototype,” Lanman explains. “Having worked in industry and academia for over a decade as a research scientist, I’ve grown accustomed to waiting a while for a project to come to fruition. With this project, we started with a different take on how to apply phase-only SLMs [spatial light modulators]. We uncovered a lot of interesting things as a result, but it took a while to pivot back to the current form the system takes.”

Oculus Research Scientists Douglas Lanman and Alexander Fix with the focal surface display prototype in all its DIY glory. The team used readily available lab parts to prove out this new design concept for VR and AR headsets.

“Doug and I sat down to look at a variety of potential projects at the beginning of my 2015 internship,” adds Matsuda, who joined Oculus as a graduate student in the Northwestern McCormick School of Engineering. “Of these, exploring the generation and simulation of freeform lenses appealed to me because I could use rendering and simulation techniques I was familiar with and enjoyed using in the past.”

Nathan Matsuda in his home lab in Chicago.

To get the ball rolling, Matsuda, Fix, and Lanman replicated existing work in goal-based caustics—“clever little lenses that push light away from some areas and into others to make dark and bright spots that form an image,” Fix says. “Nathan worked with the folks in our machine shop to create a seemingly normal looking block of acrylic that, when you hold it up to the sun, creates an image of the Oculus logo.”

“On its own, it’s mostly just a trinket,” Fix continues, “but with some further thinking on the problem, we realized that very similar math could be used to focus an image farther away in some parts of the display and focus closer in others.”

As a grad student, Matsuda says he was given the freedom to explore a number of potential avenues—including those that weren’t immediately recognized as viable. “I had a strong hunch based on early simulations that an approach like this would work somehow,” he says. “Because of the supportive environment and the team expertise I drew on, I was able to stick with it until it did work.”

While the team’s work to-date has established the ability of focal surface displays to sharpen images in VR and improve focal depth for a better, more natural viewing experience, it also opens up a wide variety of directions for future research. On a practical level, some interesting next steps include improving the underlying hardware, algorithms, and form factor, plus supporting standard game engines. The potential applications for AR are also promising.

“I think any halfway-decent research paper is, by nature, mostly incomplete,” comments Fix. “Good research seems to always raise more questions than it answers, and this is no different. We’ve laid a good foundation for this new way of building a display, but there are many more questions to be answered before it’s fully practical. Getting other people excited to solve those problems is probably my favorite part of sharing my work.”

Matsuda has also identified the potential for applications beyond VR, through his university work with cultural heritage imaging projects under the guidance of Professor Oliver Cossairt. “There are many objects of significant cultural or historical value made out of glass and other refractive materials that are difficult to capture using existing 3D scanning techniques,” Matsuda explains. Not only do focal surface displays represent a glimpse into VR’s future, then, they also offer us a unique way to preserve the past.

In the meantime, the team will continue exploring new computational imaging and display methods on the path to ever-more compelling, convenient, and comfortable VR and AR systems.

“My personal motto for my own research is that if I’m not working on something that would count as a Secret Project in Sid Meier’s Alpha Centauri—i.e., something that would’ve been magic sci fi future technology in 1998—then I’m not trying hard enough,” says Fix. “The mission of Oculus Research is to create something that would have seemed magical or impossible even five years ago—and we have the right people and resources to go out and actually do it.”

For Lanman, it was a conversation with Chief Scientist Michael Abrash that convinced him Oculus Research had the vision and resources necessary to make the next generation of VR and AR a reality—and a similar love of gaming and sci fi that’s motivated his career.

“I grew up playing video games—a lot—and I came to Oculus foremost to build the next major storytelling platform,” he says. “Over the next 10 years, I expect the dividing lines between VR, MR, and AR to increasingly blur and hopefully disappear completely. As that occurs, the same pair of glasses will seamlessly move from augmenting your office conversations and productivity suites during the day to enveloping you in the latest noir fiction in the evening. I’ve spent most of my life looking at small monitors in dimly lit rooms—I look forward to displays moving into my glasses and out into the natural world, and finally getting some work done on the move.”

Not a bad vision of the future.

— The Oculus Team