Depth map from the Lightfield Camera

The time I spent trying to match the model to the live footage last semester could have been saved if I had a portable 3D scanner. The Kinect generates a depth map, but it is too noiny, and you cann't catch more than one perspective. When used with Reconstruct Me, you can generate a model in real time that is usable as a shadow mask object. But the processing power of my tablet causes the frame rate to drop and the model generation to abort.

 I thought I would try a lightfield camera because the depth map is generated from the color images, and I thought it would give me a better registration. But the Lytro camera's depth map was much worse than the Kinect.

Lytro Depth Map

Image

 Depth Map

Overlay

The depth is created by a difference comparator function between the images captured across the array. This eliminates the IR beam and captured the depth map and image through the same lens, but the reliance on different pixel color and brightness to generate depth fails to register a depth if the surface is uniform. The software does some adjustments, and with a bit of editing the depth map can be used to create a virtual focus and aperture, but the depth resolution is low and fidelity is low.

Conclusion:

The lightfield camera's depth map is unusable for the creation of 3D models.

Updated Research Statement, Topic, and Questions

Research Statement

The research topic I have been working on is centered on the composition of a story that blends cinematic animation, interactive media, and literature in a complementary way. I am exploring the topic through the creation of a user driven animation that contains some of the poignant elements that have been filtered form children's stories over the last few generations.  The intended form would adapt contemporary devices from cinematography and interactive media, mixing text, animated characters, and live action through camera movement, turning pages, and a touch screen in a way that I am trying to compile seamlessly and coherently. I hope that the final product is an emotionally complex artifact that children and parents will find meaningful and my research will yield revelations into the creation of mixed platform narration.

Research Topic: Construction of a blended media children’s story with complicated emotional impact

Narrowed research topic:

·         Relationship between phrases, implication, and interpretation across cinematography, literature, and interactive media

·         Adaptation of a book to an interactive animation

·         The generation of communicative elements unique to hybrid media

More specific topic: How can I invoke the emotional impact of a somber bedtime story in a contemporary form without corrupting the intimate quality of a book?

Research Questions:

1.      How can analysis of overarching communicative structure aid the fusion of blended media forms?

2.      What are the constituent phrases of cinematography, literature, and interactive media?

3.      How do the dialects of cinematography, literature, and interactive media function (constructively and destructively) when combined?

Mobile Scanning

I am working on putting together a mobile 3D scanner to make a quick 3D model of an environment I am shooting for shadows and reflections. I have the 3D construction working on my PC, and the 3D scanner working on my tablet, but my tablet drops frames which causes the 3D construction software to abort.

I thought I would try to use the depth map and photo to make a 3D model.

I spend several hours trying to do this tutorial in Maya 2016, and the changes are so drastic, I couldn't figure it out. The input output controlls

I found a tutorial to do it with Photoshop, and it worked, but this was the result. There is a descrepency between the focal length of the camera lens and the depth map. I assume the Kenect SDK software compensates.

This is what the live capture looks like.

Long shot point cloud across the lab (about 25 feet)

 Long shot point cloud across the lab from above (about 25 feet)

 Long shot with texture across the lab from above (about 25 feet)

 Long shot with texture across the lab from above (about 25 feet)

 Close up of the chairs

Long shot point cloud across the lab from above (about 25 feet)

  Close Point cloud

Close up color

 The results are not great. The software that creates a  model from the Kinect clearly does a lot of processing to eliminate the noise. My tablet is not up to the task, and classroom services no longer loans laptops. I have a lightfield camera that I am going to have to create the model from photographs and the rough depth map models as a reference.

Butcher Shop Interior

This is my first test texture for the Butcher shop interior. It is a Boolean of the word See mapped on to a scan of my face (with no eyes).

"See what people do"

Unity

I got unity running at home, and I have started the ball game. I got a bit of a problem getting the cubes to disappear, so I am going to head in early to try to get it worked out before class.

The Big Idea (as of 2 am Tuesday 01/19/16)

The Big Idea (subject to change without notice)

This semester I am focused on creating a form that blends cinematic animation, interactive media, and literature in a complimentary way. The shot that I am working on now is the interior of the building Tuba-Goose entered last semester. I am changing the location to a butcher shop because it adds to the dramatic effect, and horror films have a lot of paradigmatic camera and lighting techniques that would be interesting to apply to another medium. I thought it would be complimentary mix of media to make the interior all narrative text (people , shelves, everything except the goose would be extruded polygonal text), use motion capture to animate the people, and use a dolly zoom for a shot where the goose sees the butcher doing what butchers do.

I am not sure how I will texture everything or how I will organize the words so they function as characters and are readable as text, but I think motion capture will add an inherently human quality to the movement that should give me more freedom to deviate from anthropomorphic shapes while relaying the intended actions. I am hoping that when I have the movement for the scene, I will be able to create a model based on the text and the key frames that becomes reveals itself to the reader in a way that alludes to the internal dialogue of the goose when he realizes what is going on in the butcher shop (I want the audience to be thinking "Is that a person chopping off the head of... YES IT IS!" in parallel to the character's realization). This shot does not do much to exploit interactive media, but I am worried compounding too many distracting elements (e.g. putting a tilt interaction, prompting an expand gesture on the touch screen, or something of that nature) would drown the humble, quiet nature of a book.

Unity Game Design Breakdown

My digital popup book is going to run a tablet through the Unity game engine. 

Current Animatic

Execution

Progression of the animation is user driven through a swiping motion. Indication of when and how to swipe is given through "goose tracks"

Most of the animation is CGI mixed with live action, and I do not expect most tablets to be able to handle the level of detail and texture needed to render my model. So I am pre-rendering the footage and the swiping motion sets the speed and scrubs through the footage rather than moving characters in a 3D environment.