IllumiRoom is a research project that Microsoft is looking into productising. The prototype uses a Kinect for Windows and a projector with a wide field of view to extend the screen onto the area around the television and produce illusions. The system scans the appearance and geometry of the room using the Kinect to adapt the projected visuals in real-time without needed to pre-process the graphics.
IllumiRoom can create many illusions that can be combined to enhance a gaming experience. It can change the appearance of the room to make it look black and white or make the room look like a it’s part of a cartoon. It can appear to warp reality to induce motion or make it look like it is snowing.
Since the projected area is in the peripheral vision of the user who is looking at the television screen, the image doesn’t have to be as high quality as the image on the TV. The image can display only the edges of objects or only certain elements such as gunfire and explosions. It’s also possible to project markers around objects that can be picked up to make them easier to find.
The room furniture can also be used to mask out the image and make it look like the image is being projected on the wall behind the furniture. A grid or dots can be used instead of an image to improve peripheral motion. Another illusion makes it appear that the light source in room matches the light source on the screen. It’s also possible to make an object appear to fall out of the screen and into the room.
Some of these illusions, such as extending the screen image, require information from the game itself. Other illusions, such as peripheral motion, can be automatically generated from the screen and controller data.
It’s also possible to project a video that was filmed using a dual camera that captures both a narrow and wide field of view. The narrow view is displayed on the screen and the wide view is projected. The wide view can also be blurred to create a peripheral effect.
IllumiRoom will likely be sold as a separate accessory for the next-generation Xbox like how the Kinect is sold separately to the current Xbox.
Microsoft has released the IllumiRoom research paper here.