Thanks to a collaboration between NASA and Microsoft aimed at advancing human-robot interactions, the space agency's OnSight software will allow researchers to explore a virtual Martian landscape created from data sent back by the Curiosity rover.

NASA's Curiosity robotic explorer may be a nuclear-powered technological marvel, but looking at its data through a flat screen is a bit like trying to do surgery by postcard. Even 3D images created using stereographs are still very flat and lacking in perspective, depth, area, or context. What scientists want is a way to look at that data that allows them to recreate the area around the unmanned rover as a 3D virtual environment where researchers can meet and study phenomena in context.

The OnSight software was developed by NASA and Microsoft as part of an ongoing partnership to advance human-robot interaction. It works with Microsoft's Hololens, which is a wearable Windows 10 computer hooked up to a high-definition 3D head-mounted display with holographic lenses. With the headset on, the data sent back by Curiosity is mapped as a 3D-simulated environment where avatars of scientists can meet, walk around and study the area using augmented reality.

In addition to studying data, OnSight can also act as a more intuitive planning tool and as an operating system that allows scientist to manipulate the simulation using gestures and menu commands.