In a recent Mind Matters podcast, “Thomas Furness and the invention of virtual reality,” Walter Bradley Center director Robert J. Marks interviewed Thomas Furness, an engineer who wanted American fighter pilots to be able to see what was really happening around them in the air.

It seemed that they needed virtual reality to do it. And that’s how the technology that powers so many computer games today got started. The transcript can be downloaded at the bottom of the page here.

Here is is a partial transcript of highlights from the interview:

Virtual reality was pioneered by Thomas Furness during his Air Force service in 1966 when he served as a Second Lieutenant at Wright-Patterson Air Force Base. Much later, in 1993, the first technically serious conference on virtual reality was put together by Thomas Furness, Tom Caudell, and Robert J. Marks.

02:58 | How virtual reality came to be

Furness’s job at the air force base involved taking turns at the test plane instruments, so he got to see what it felt like:

Thomas Furness (left): I was sitting in a back seat. I had to suit up. I was sitting in the ejection seat and parachutes and g-suits and all this kind of stuff, just like the pilots. But my job was actually running the test instrumentation. And while the pilot was flying the airplane, but half the time, the pilot said, “Ah, you take it.” And so I was flying the airplane while he was eating bananas or something like that. So it was, yeah, we were boring holes through the sky. We’re going near the Mach and beyond the Mach and testing this equipment, flying at low altitude, flying at high altitude, going straight up, going straight down. It was really fun.

The few hours in the air were fun but the many hours on the ground waiting for planes to be prepared were not. As an engineer, Furness decided to focus on building cockpits that would enable pilots to fly at night without being seen by the enemy and to find the enemy on the ground, using sensors. He discovered that the pilot had to physically fly the machine in order to aim it:

Thomas Furness: And I decided, “Well, why isn’t just looking it a good enough? Because we could probably measure where you’re looking.” And so that’s what I started working on, the idea of why don’t we track the pilot’s helmet, the helmet position?

Furness soon discovered a practical problem: Limited space in the cockpit.

Thomas Furness: And this was very expensive and difficult to do because cockpits, fighter cockpits especially, the most expensive real estate on the earth per square inch. … But so we said, “Well, why don’t we, instead of trying to create a real image display where you’re actually looking at the face of a cathode ray tube or some kind of display device, why don’t we make it a virtual image?” And the virtual image is something that you see in space that doesn’t really exist there.

Can we really see something in space that doesn’t really exist there?

Thomas Furness: For example, the simplest example of this is really the mirror in your bathroom. You’re standing in front of your mirror in the bathroom, you see yourself. But that’s not really you, you just appear to be there. You’re not really there. And that is a virtual image of you. And if we could use some mirrors and some what we call combiners, beam combiners, we could actually take a really small picture, like from a miniature cathode ray tube, it may be the size of a quarter. We could draw a picture on that, then we can magnify and collimate it, meaning making it appear in the distance, and to project it into the eye so you get a huge screen, maybe a 30 inch screen that doesn’t take up any cockpit space.

The big project was to attach it to the pilot’s helmet.

Thomas Furness: So what that meant is that we could take our head position and move those sensors around outside the cockpit, and then display a picture in that same angle. And it was just like cutting a hole in the whole airplane and being able to see at night, sort of like having a picture window, being able to see through, see at night. And so this became really the first virtual reality system where we could basically take the place of the scene through the cockpit.

His team then realized that they could project the entire cockpit as virtual reality well, starting in 1977. They also began to work on tactile display: “So we figure out a way to track the hands, the hand position and finger position.”

Thomas Furness: And so, one of the feedbacks was really a sound. You’d hear that switch clicking with binaural sound. By this time, we’re using three-dimensional sound, binaural sound. We’re using individual ear prints of the pilot so that you’re basically mapping the sounds so as true 3D sound and not just stereophonic sound, it’s 3D sound. So you hear things in 360 degrees around you. And so you’d hear that switch click in that direction. And then you would have, in the gloves, you would have tactors that actually would stimulate the fingers. And so you’d feel a little pressure in your finger when you touched that display, even though it’s not really there, but it appears to be there. So when you get this tactical feedback, you know that you’d activated that switch. You got the sound that it clicked and things like that. You take your hand away, the switch panel disappears because it’s just getting into way otherwise.

It was a virtual cockpit for a pilot who was in a real cockpit.

Thomas Furness: We’d have it, we’d take information from the outside world and project it in there in terms of where you’re located, the navigation information, where the friendly aircraft are, where the enemy aircraft are, where the ground surface, the air missile batteries are. All of this you’d see in this grand gestalt, this picture that surrounds you.

How did the pilots react?

Thomas Furness:And so they came into my lab and over to the simulator, the cockpit simulator. And they saw this huge helmet hanging above the cockpit. I mean, this thing weighed about 10 pounds at the time, but it was just the simulator of these things. It had all kinds of bells and whistles built in.

Robert J. Marks (right): So this was the equivalent of today’s headset, virtual reality headset.

Thomas Furness: Yes, it was. And it gave us the wide field of view. It had these miniature cathode ray tubes built in and all the tracking stuff and things like that. And we had this negator spring assembly that would actually support the weight of the helmet. It wouldn’t change the momentum, the angular momentum of the head set when you’re moving it around. But nevertheless, so the first reaction when these pilots came in and they saw this helmet, they pointed at the helmet and they pointed to me and said, “You got to be kidding me.”

He pointed out that that helmet was just the test version and brought out the proposed one, designed by Lucasfilms.

Thomas Furness: It’s like sitting on the front row of an IMAX theater. And they would say, “Wow.” And I said, “Well, look around a little bit.” And they’d look around. And they said, “Wow, this is really cool.” Then we started explaining how it works. Now, they’d never seen this before in their lives, but they knew how to fly airplanes. And we explained some of the things that they’re going to see. For example, they could actually see the radar painting the sky. They could see the radar signals actually painting the sky. They’d see all where these other guys are located in the world in 3D. And they had a God’s eye display where they could look down in their lap and they see this bubble, this hemisphere of the whole world, so to speak.

Then Furness showed them what VR would look like in real-life combat.

Thomas Furness: They’re flying along in this world. They see the outside world, they’re flying along. They’re seeing all this information. And now they’re in this proper, this simulation, this virtual simulation. And then what would happen is we send in the bad guys. The first thing they would do is they’d hear it in 3D sound behind them. They hear this guy radiating them from the behind, and then rear view mirrors would wind into the cockpit and they would see a representation of the guys coming in from behind. And then there’d be a few swear words that come out of the pilots. That happened. And then they would go into afterburner. They’re go into afterburner and start climbing straight up. And then they were in the fur ball, what we call a fur ball, where they are in, they’re working with this adversary in air to air combat. And they were in and out of afterburner, speed brakes, the whole deal. Just watch Top Gun and you’ll see what they were doing. And they were swearing and all of this kind of stuff going on. And then about that time, what I did was I shut off the simulation and everything went dark.

And I said to them over the microphone in their headset, I said, “Okay, guys, we need another quarter please.”

Quarters? This was all happening during the 1980s and Furness’s team, of course, got the funding for their super cockpit.

But virtual reality was not only about aerial warfare.Stay tuned.

Thomas Furness: And after that, I started getting phone calls… And this mother called, and she said, “I watched this program on television when you were talking about this virtual cockpit stuff and this virtual reality stuff.” And she said, “My child has cerebral palsy. Is there anything you can do with this technology to help my child?”

And then a surgeon called me, says, “I’m a thoracic surgeon. I’m trying to do a graft on aorta. I’m up to my elbows inside this patient, trying to feel my way around, and my map is actually a CT scan, which is on a light box on the wall on the side of the operating room. Is there any way that you can take that information and superimpose it into a patient?”

Next week: How they took virtual reality beyond fighter pilot issues