Prompting a Dialogue in 3D!!!

The most believable responses are reactions from real people to unscripted scenarios, not actors. That is why I am trying to get authentic reactions from people, and edit the goose into the footage. As an extension, I have been working on a teleprompter for unwitting actors.

Goose Vision

I want the audience/viewers reaction to be surprised and exclamatory; something like "Is that a goose with a tuba?!", and not "what is this for" or "How did you make it 3D?". So I am trying to absorb the character in the viewers space, rather than trying to adapt the viewer to the characters world.
The display is 3D (without glasses) with parallax, and the background is from a camera behind the monitor.

The interlacing takes up a huge amount of the processing power, so the live version will have only a few responses, triggered manually (AI reactive responses are slower, and they are unreliable as the input cycles overlap the processing).

The Kinect 3D Sensor

The Kinect generates a lot of information about the environment, it might also be useful in defining the camera's position in the virtual space and grabbing a quick 3D model. 

The two limiting factors for outdoor are sunlight, which has a lot of infrared (the Kinect tracks depth with an array of IR laser beams), and the Kinect is only designed to range objects within 10 to 20 feet of the sensor. I am looking into lenses and filters for both issues.

The Microsoft SDK (seen above on a running on a laptop) is very stable, but it does not communicate with Isadora, so I could not use it to do head tracking in the field, but it might be useful for making a live 3D map of the camera position.