What is an upright image and real image?

Object Recognition: What kind of algorithms would an Eden of the East-style image search and registration engine use if implemented in the real world?

• Eden of the East is an anime show directed by Kenji Kamiyama that aired in 2009. The main characters of the show run a start-up company that maintains a real-time image search engine that allows you to register objects and tag them with metadata. For example, I could point my phone's camera at a car and have the Eden app automatically pull up its make, model, and price. I could also turn the camera on someone sitting next to me on the metro, and have their public bio float onto the screen AR-style. The crazy thing about Eden of the East is that it recognizes objects in any lighting, angle, facial expression (if applicable), and contortion. Theoretically, how would such a futuristic application be developed using current state-of-the-art computer vision and machine learning techniques?

• Answer:

  Well if we knew I wouldn't be doing a PhD in the same, would I? :) Jokes apart, this is currently one of the most widely studied problems in computer vision, and we are still not there yet. To proclaim any one approach as the way to go would be rather presumptuous; the field might change drastically anytime. 's answer does a good job of describing the general architecture of the solution, and I am just going to add to it. The key problem is to be invariant to the various transformations an object can go through (rotation, translation, small deformations, articulation, change in color/texture etc). One way to do this is to define features on the image that are invariant to such transformations, and then use machine learning techniques to learn where each object or object category lies in this feature space. Features such as HOG(http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients) or SIFT(http://en.wikipedia.org/wiki/Scale-invariant_feature_transform) are designed to be invariant to small deformations. Often, the kind of deformations that objects such as people undergo is way too severe even for these feature descriptors, and so one typically tries to detect parts of the object (that are somewhat more rigid than the object itself and so amenable to such descriptors), and then one tries to find the parts that occur in some rough relative configuration relative to each other (eg.http://people.cs.uchicago.edu/~rbg/latent/ ). Training such models typically requires you to provide large numbers of example images in all possible poses and all kinds of clutter. One class of methods that has started to gain traction in recent times is to learn invariant descriptors using neural networks or other such architectures (http://en.wikipedia.org/wiki/Deep_learning). Also, techniques tend to vary depending on whether you want to recognize individual instances (eg. my friend's beat-up Mustang), subcategories (eg. Ford Mustang) or object categories(eg. car).