Grants and Contributions:

Grant or Award spanning more than one fiscal year. (2017-2018 to 2022-2023)

Conventional video chat provides audio and video connection between distance-separated collaborators. But conventional video chat tools are inadequate for remote collaboration tasks that focus on objects and environments. Such remote collaboration tasks will increasingly become commonplace, including assistance tasks (e.g. machine repair), guidance tasks (e.g. remote tour), and inspection tasks (e.g. design critique). Conventional video chat does a poor job of giving collaborators: (i) awareness information about what a remote collaborator is doing, such as where they are looking, or the hand gestures they are making as part of speech (e.g. pointing while saying “Look at this one”), and (ii) an understanding of the remote space. These interfaces are critical, because remote objects and locations have become be the main focus of collaboration enabled by mobile devices.

We will explore how to design Mixed Reality (MR) interfaces to support remote collaboration tasks. Such interfaces typically do two things: first, they visually anchor virtual content in the real world using augmented reality (e.g. projected visuals, head-mounted displays, or see-through displays), and second, they use sensors to capture aspects of environment (e.g. a depth camera scan/video capture of an environment) or collaborators’ interactions with the environment (e.g. their gestures or what they are looking at). We will produce theories and prototypes that describe and illustrate how to design MR interfaces that capture awareness information about collaborators (such as gaze, view and gesture information), and allow collaborators to independently explore a digital remote environment.

This program is comprised of three sub-projects:

(1) Software infrastructure. We will design and implement new infrastructure that allows us to rapidly prototype new kinds of MR experiences.

(2) Capturing and visualizing awareness information. Our MR prototypes will focus on capturing collaborators’ interactions with the environment (e.g. where are they looking, what are they pointing at, etc.), representing and visualizing this to remote collaborators.

(3) Enabling independent exploration. We will explore ways of digitally capturing and reconstructing remote environments to allow collaborators to independently view and explore these spaces.

Our research will result in new knowledge about how to design and build MR interfaces that allow distance-separated collaborators to work together effectively. We will identify and address challenges that arise in this design space including user interface design, ease of use and fluidity, and technical infrastructure. Our work will inform the research community on designing effective MR interfaces for remote collaboration. It will inform Canadian companies as they continue to develop technologies for remote and telework scenarios.