Shoot & Copy
Phonecam-Based Information Transfer from Public Displays onto Mobile Phones
Large electronic displays become increasingly affordable and can thus be found in many public places. They display high-resolution images, commercials, late-breaking news, weather forecasts, or stock exchange prices. In spite of this rich information, people are not intended to actively interact with public displays but to passively view their displayed content. Usually public displays are also too big for personal interaction or even out of reach in order to protect them from damage. Hence, if data displayed on a public screen is of interest for an individual, s/he needs to remember the content in order to retrieve additional information at home. In most cases, the information has been forgotten by the time users reach their home/office. This decreases the intended communication efficiency.
As camera-equipped phones become more and more available and accompany their users most of their time, people use them as their external visual memory when taking pictures of interesting things like advertisements. In addition, today's cameras are capable of taking high-quality images, which allows the retrieval of highly detailed information. But while people can conserve this information, they can not directly interact with it. They have to extract the displayed information themselves and transfer it to other applications (e.g. a browser) – in most cases even to a personal computer – in order to observe the corresponding data and take the desired action (e.g. open a URL or browse the web for certain words contained in the photograph).
So far, direct interaction with pictures taken with mobile phone cameras is only possible through the augmentation with visual markers and the recognition of their visual patterns. Semapedia, for example, uses visual markers for linking objects from the real world to corresponding articles on Wikipedia. The existing approaches in this area use little two-dimensional barcodes that can be attached to different objects. These visual markers usually only encrypt a small amount of information (e.g. a number identifier or a URL) which can nevertheless be used as input for the interaction with different services and applications.
In this light, the combination of powerful mobile devices with high-resolution cameras and large displays bears interesting new opportunities. Besides taking a picture of the information of interest, additional data can be transferred to the mobile phone. For that purpose, the photograph needs to be analyzed using computer vision. In the process, whole screens or displays can be used as rich input into the image recognition process. The technology for communicating information via visual patterns blends in with the media that display them – the screen becomes the marker.
We present Shoot & Copy as a new interaction technique for retrieving information from displays by simply taking a picture of the desired content. We describe its prototypical implementation and the algorithms we used for image comparison, and finally discuss the results of an initial user study.