Thumb On Hand Interaction

Thumb On Hand Interaction is a system that allows single hand gesture input for mobile devices. A chest mounted depth camera detects the gestures, performed infront of the body, and forwards them to the device.

The Distraction caused by mobile touchscreen devices is unneeded and sometimes even dangerous as users are on the go. Thumb on Hand Interaction minimizes the amount of attention mobile devices demand, since it allows the user to interact eyes free and feel the performed input operation.

Thumb on Hand interaction allows the usage of the thumb as a cursor operating on top of the fingers. This concept allows different types of input such as sliders, buttons, 2D navigation and gestures.

Acknowledgements

This System is developed at the Human Computer Interaction Lab (Hasso Plattner Institute) in Potsdam. Feel free to learn more about this Lab and interesting projects here. The corresponding supervisors were Christian Holz and Patrick Baudisch.

Introduction

Interaction technologies for mobile devices became an important research sector, over the past years. A great number of inspiring work concerning different concepts and investigations about the humans abilities to interact has been done.

However commercial mobile interfaces are still based upon input technologies, which have been established for desktop scenarios. Pointing on touchscreens and performing input on keyboards demand the user’s attention, yet only represent secondary tasks in mobile scenarios. People being on the go desire to fulfil a primary task, such as getting to the office or relaxing in the park. Hence the Distraction caused by mobile devices is unneeded and sometimes even dangerous.

This system proposes thumb on hand gestures for interacting with mobile devices. Our concept address the following contradictory need. The user’s hand should be exempted from a physical device. At the same time we want to enable the user to have his device "at hand" at all times.

Our main focus was to implement rich input options generated from the thumb interplaying with the corresponding hand. We suggest gestures like pinching fingers or moving the thumb across the palm as appropriate for quick interactions in mobile scenarios and present the corresponding realization techniques. Our design involves a chest mounted camera system, enabling the detection and analyse of the users hand as it is held in front of the body. The tracking of the thumb’s position and motion in reference to the users empty hand enables the possibility to control imaginary widgets like sliders,dials or buttons placed along our Hand.

Read The Hole Thesis?

Rich Interactions

Thumb On Hand Interaction enables different single hand gestures to be used for different types of applications. Our implemented solution involves two different Algorithms detecting two different interaction types. The thumb on hand interaction and the pinching gesture interaction.

We implemented the detection of the thumb’s relative position in reference to the hand. The finger placed closely together span an input surface the thumb tip operates upon. Hence our solution interprets the hand and its dimensions as a single touchpad. We detect the thumb’s relative position and distance to this pad. Our imaginary widgets addressing thumb on hand interaction, enable the possibility to use the following types of gestures in order to control different application. • (a) Touchpad analogy, horizontal and vertical Sliders across our palm.

Dial across the finger.

Buttons on top of the finger.

Buttons on top of each fingertip to be reached by the thumb.

We expanded common techniques to detect pinching gestures by the use of optical systems. In order to enrich the possibilities of interactions, our solution offers a detailed analyse of the user’s thumb and index finger as they interact with each other. We designed and implemented the following interaction technologies based upon the pinching gesture

Continous slider along our thumb to be reached by the index finger.

Continous dial between thumb and finger as they rub each other.

Discrete itembar along our thumb to be reached by the index finger.

Playing with Apps

Beside controlling the Iphone by the use of our interaction system, we had great fun exploring even more applications and widgets. The following gallery illustrates a couple of our implementations.

Slider Thumb on Hand

This Pic illustrates the famous video game Pong, which is controlled by the user’s thumb tip sliding across the finger. Hence a small paddle, which is supposed to hit a white ball, corresponds to the y-coordinate of the thumb tip in reference to the user’s hand. The images illustrate different positions of the thumb and the paddle, which is associated to it.

Drawing Thumb on Hand

We designed a motion flow visualization in order to demonstrate the imaginary touchpad. The position of our thumb in reference to the hand is related to the controlled pointer within the drawing frame.

Dial Thumb on Hand

The imaginary discrete dial illustrated in this Figure , enable the possibility to select from different items, which are rotationally organized. The user iterates through those items by performing a circular motion with his thumb tip across the finger’s surface. A circular sector which is coloured in red, illustrate a touched item. A turquoise arc represents an item the user’s thumb tip hovers over. This imaginary Widget is also implemented as a continuous widget.

Buttons Thumb on Hand

We implemented imaginary buttons the user imagines to be placed across his fingers as illustrated in this Figure. Hence we fragmented the available inputspace in four squares of equal size. Because of low resolution and the corresponding inaccuracy we decided to avoid the implementation of more buttons for our prototype. Each region adresses a different haircut for the hairy ball. The graphical processing sketch is an application created by Edward Porten.

Slider Pinch

Our solution in order to detect the continuous values of the pinching widget is illustrated by the debug output collage of this Figure. The volume is adjustable by an imaginary dial which turns as the user’s index finger slides along the thumb.

Buttons Pinch

This Figure presents a collage of debug images of our imaginary item bar implementation. This widget is called by double tapping and holding the tip of the index finger on top of the thumb. The user now continues to slide his index finger along the thumb and imagines different items to be placed on top of it. In order to select one, the user simply releases the contact between thumb and index finger close to the position he imagines it to be placed. The active item that was the closest to the release point of the two fingers, will call the corresponding functions within the application.