Notes On Natural Interaction - Alessandro Valli
Notes On Natural Interaction - Alessandro Valli
Notes on
Natural Interaction
http://alessandrovalli.com
http://naturalinteraction.org
Alessandro Valli | Notes on Natural Interaction
Contents
Introduction 4
Goal 6
Framework 7
Natural 8
Interaction 9
Leonardo 10
Cognition 11
Context 12
Attention 13
Periphery 14
Social 15
Competition and collaboration 16
Aesthetics and emotion 17
Movement 18
Freedom 19
Embodiment 20
Architecture 21
Physical space 22
Integration 23
Perception 25
Gestures 26
Gesture recognition 27
Computer vision 28
Visible and infrared 29
Early processing 30
Image analysis 31
Interpretation and accuracy 32
Stereo vision 33
Audio analysis 34
Feedback 35
Stimuli 36
Intuition 37
Imitation 38
2
Alessandro Valli | Notes on Natural Interaction
Style 40
Constraints and augmentation 41
Physics 42
Spatial organization 43
Seamless 44
Less is more 45
Media 46
Tangible 47
Tools and characters 48
Magic 49
Game 50
Human and language 51
Narration 52
Dialogue 53
Acknowledgements 76
Conclusion 77
References 78
3
Alessandro Valli | Notes on Natural Interaction
Introduction
4
Alessandro Valli | Notes on Natural Interaction
5
Alessandro Valli | Notes on Natural Interaction
Goal
6
Alessandro Valli | Notes on Natural Interaction
Framework
7
Alessandro Valli | Notes on Natural Interaction
Natural
8
Alessandro Valli | Notes on Natural Interaction
Interaction
9
Alessandro Valli | Notes on Natural Interaction
Leonardo
10
Alessandro Valli | Notes on Natural Interaction
Cognition
11
Alessandro Valli | Notes on Natural Interaction
Context
This text refers to the term ‘context’ in two ways. The first
one is related to the context in which interaction takes place.
Humans are incredible perceptors: every detail in the area covered
by senses influences the way people discover and interact with the
focus of their attention. For this reason it must be clear to
designers that it is not the system to communicate, it is the full
space. Systems must be able to perceive human context in order to
modulate communication depending on what is happening. Context
awareness can help in interpreting people actions and intentions.
Architectural and interface design can help limiting the context the
system must be able to deal with;
12
Alessandro Valli | Notes on Natural Interaction
Attention
13
Alessandro Valli | Notes on Natural Interaction
Periphery
The role of periphery of attention requires an additional
insight, since it is a dimension completely unknown to common
interfaces. William Mitchell wrote: “Peripheral information is by no
means unimportant; in fact, it plays a crucial role in establishing
the character of a place and sustaining your relationship to it.
When a room has a window, for example, it provides a continuous
flow of information about the external environment - the cycles of
day and night, the movement of sunlight and shadows, the
succession of bright and cloudy moments, and the alternation of
dry and rainy patches. You rarely pay explicit attention to all this,
but you are peripherally aware of it, and you feel uncomfortably
isolated if you are cut off from it”.
14
Alessandro Valli | Notes on Natural Interaction
Social
15
Alessandro Valli | Notes on Natural Interaction
16
Alessandro Valli | Notes on Natural Interaction
17
Alessandro Valli | Notes on Natural Interaction
Movement
18
Alessandro Valli | Notes on Natural Interaction
Freedom
19
Alessandro Valli | Notes on Natural Interaction
Embodiment
20
Alessandro Valli | Notes on Natural Interaction
Architecture
21
Alessandro Valli | Notes on Natural Interaction
Physical space
22
Alessandro Valli | Notes on Natural Interaction
Integration
23
Alessandro Valli | Notes on Natural Interaction
24
Alessandro Valli | Notes on Natural Interaction
Perception
25
Alessandro Valli | Notes on Natural Interaction
Gestures
26
Alessandro Valli | Notes on Natural Interaction
Gesture recognition
27
Alessandro Valli | Notes on Natural Interaction
Computer vision
28
Alessandro Valli | Notes on Natural Interaction
29
Alessandro Valli | Notes on Natural Interaction
Early processing
30
Alessandro Valli | Notes on Natural Interaction
Image analysis
31
Alessandro Valli | Notes on Natural Interaction
32
Alessandro Valli | Notes on Natural Interaction
Stereo vision
33
Alessandro Valli | Notes on Natural Interaction
Audio analysis
34
Alessandro Valli | Notes on Natural Interaction
Feedback
35
Alessandro Valli | Notes on Natural Interaction
Stimuli
36
Alessandro Valli | Notes on Natural Interaction
Intuition
37
Alessandro Valli | Notes on Natural Interaction
Imitation
38
Alessandro Valli | Notes on Natural Interaction
39
Alessandro Valli | Notes on Natural Interaction
Style
40
Alessandro Valli | Notes on Natural Interaction
41
Alessandro Valli | Notes on Natural Interaction
Physics
42
Alessandro Valli | Notes on Natural Interaction
Spatial organization
43
Alessandro Valli | Notes on Natural Interaction
Seamless
44
Alessandro Valli | Notes on Natural Interaction
Less is more
45
Alessandro Valli | Notes on Natural Interaction
Media
46
Alessandro Valli | Notes on Natural Interaction
Tangible
47
Alessandro Valli | Notes on Natural Interaction
How will the interface look like? The role the interface plays
defines the level of interaction it induces. Tool-like interfaces
(those acting like a common tool and those acting like a multimedia
application) will induce operational behaviors, with an emphasis on
manipulation and command (the purpose of a tool can be various:
access to information, media object manipulation, measurement
and computation, etcetera); character based interfaces will induce
social behaviors, emphasizing non verbal (like gesticulation) and
verbal communication.
48
Alessandro Valli | Notes on Natural Interaction
Magic
49
Alessandro Valli | Notes on Natural Interaction
Game
50
Alessandro Valli | Notes on Natural Interaction
51
Alessandro Valli | Notes on Natural Interaction
Narration
52
Alessandro Valli | Notes on Natural Interaction
Dialogue
53
Alessandro Valli | Notes on Natural Interaction
54
Alessandro Valli | Notes on Natural Interaction
55
Alessandro Valli | Notes on Natural Interaction
Research
56
Alessandro Valli | Notes on Natural Interaction
Observing people
57
Alessandro Valli | Notes on Natural Interaction
58
Alessandro Valli | Notes on Natural Interaction
59
Alessandro Valli | Notes on Natural Interaction
60
Alessandro Valli | Notes on Natural Interaction
61
Alessandro Valli | Notes on Natural Interaction
62
Alessandro Valli | Notes on Natural Interaction
63
Alessandro Valli | Notes on Natural Interaction
Remotely
64
Alessandro Valli | Notes on Natural Interaction
65
Alessandro Valli | Notes on Natural Interaction
66
Alessandro Valli | Notes on Natural Interaction
Artifacts
67
Alessandro Valli | Notes on Natural Interaction
Projects
PointAt
An adaptive and robust hand pointing system. Two of such systems
have been installed in Palazzo Medici Riccardi, the third museum in
Florence, in Lorenzo De Medici’s bedroom. Users interact with
natural deictic gestures with digital reproductions of the beautiful
frescos by Benozzo Gozzoli, hosted in the museum.
68
Alessandro Valli | Notes on Natural Interaction
tangiTable
A tangible interface for children. Players can interact with maps
projected on the table moving wooden physical objects, such as
cylindrical markers and round lenses. It is a very robust
collaborative tangible playground. More tables can be connected
remotely through the internet.
minorityMap
Interaction with scenes and objects visualized on a large rear
projection display or a multi-viewpoint autostereoscopic plasma
display. Interaction is based on user gestures and voice commands.
This system has been hosted in Palazzo Vecchio, Florence, showing
the changes that the city is going through (new train station, new
underground tracks…).
69
Alessandro Valli | Notes on Natural Interaction
golem
A real-time full body motion tracking and humanoid animation
system. Users don’t need to wear any special device, and can
animate 3D cartoon characters live. The system is MPEG-4
compliant.
voxbox
A real-time desktop voxelization and broadcast system. The
application builds a voxel representation of the user hands and
tools and sends it to local or remote computers for real-time 3D
visualization. It provides interaction with physical and virtual
objects in a seamless way.
70
Alessandro Valli | Notes on Natural Interaction
zooi
A context preserving zoomable user interface. This flash application
uses non homogeneous zooming in order to show details and
context at the same time optimizing the screen surface. It uses a
2D space browsing paradigm instead of the common hypertext
paradigm.
golem 2
A full unencumbered body tracker based on a model matching
approach in order to allow tracking of a large number of degrees of
freedom. A collaborative shared virtual environment based on these
systems has been tested too.
71
Alessandro Valli | Notes on Natural Interaction
ygdrasil vision
Interfacing a vision module to the shared networked environment
system ygdrasil to provide natural 3D navigation through deictic
gestures and body motions. Exposed in Palazzo Medici Riccardi and
Palazzo Vecchio, Florence.
iconoclastBody
Controlling an arcade computer game with intuitive gestures. The
game is inspired by Marcel Duchamp. An untrained user can play
walking around on the stage and performing smashing gestures.
72
Alessandro Valli | Notes on Natural Interaction
ambientClock
An experiment about ambient displays and peripheral perception.
An ordinary wall clock that gives non critical information such as
websites accesses, news and weather forecasts associated with
sound icons. The system was on 24 hours a day for some months in
the lab, and provided useful information when needed, while not
disturbing and interrupting the researchers.
visualBrowser
An automatically segmented and annotated movie or a collection of
images can be browsed using a multiresolution interface that
arranges thumbnails on screen in a number of ways depending on
user needs. An active search paradigm in an ordered universe is
used instead of the common query paradigm. The vision system is
based on near infrared light.
73
Alessandro Valli | Notes on Natural Interaction
flySwatter
In this installation, exposed in Palazzo delle Papesse museum,
Florence, multiple users could smash digital flies projected on the
walls using real flyswatters. The flies represented powerful men in
history.
Retina
A lightweight, robust computer vision module for interactive
natural interfaces. Performs early processing and sends features
through sockets or MIDI to other applications (C++, Java, Flash MX,
MAX/MSP…). Downloadable from the website.
74
Alessandro Valli | Notes on Natural Interaction
Epoque
A high performance 2D audio video library for C++. It can be easily
interfaced to Retina, and allows to build video based natural
interfaces in a very short time. Downloadable from the website.
Carmen
A high performance 3D library, based on the Irrlicht engine. It can
be easily interfaced to Retina, and allows to build 3D based natural
interfaces in a very short time.
75
Alessandro Valli | Notes on Natural Interaction
Acknowledgements
76
Alessandro Valli | Notes on Natural Interaction
Conclusion
77
Alessandro Valli | Notes on Natural Interaction
References
Perceptive Media
An interdisciplinary initiative to combine multimedia display and
machine perception to create useful, adaptive, responsive
interfaces between people and technology.
http://www.cs.ucsb.edu/%7Emturk/Papers/PerceptiveM edia.pdf
Embodied Interaction
Embodiment reflects both a physical presence in the world and a
social embedding in a web of practices and purposes. The outline of
a new foundation for the design and analysis of interactive systems
is presented.
http://www.dourish.com/embodied/embodied99.pdf
Spatial Computing
Spatial computing is human interaction with a machine in which the
machine retains and manipulates referents to real objects and
spaces. It is an essential component for making our machines fuller
partners in our work and play.
http://acg.media.mit.edu/people/simong/thesis/SpatialComputing.pdf
78
Alessandro Valli | Notes on Natural Interaction
DataTiles
Tagged transparent tiles are used as modular construction units.
These tiles are augmented by dynamic graphical information when
they are placed on a sensor-enhanced flat panel display.
http://www.csl.sony.co.jp/person/rekimoto/datatile/
Jeremiah
Jeremiah is based around two subsystems, a graphics system which
constitutes the head and a vision system which allows him to see.
There is also a simple, built in emotion engine which allows him to
respond to visual stimulus via expressions or emotions.
http://www.ee.surrey.ac.uk/Personal/R.Bowden/jeremiah/jeremiah.html
Leonardo
Leonardo is the Stradivarius of expressive robots. It is our challenge
to give Leonardo a computational brain that is worthy of its body.
http://robotic.media.mit.edu/projects/Leonardo/Leo-intro.html
Public Anemone
It is a robotic creature with an organic appearance and natural
quality of movement. It interacts with the audience by orienting to
their movements using a stereo machine vision system. But if you
get too close, it recoils like a rattlesnake.
http://robotic.media.mit.edu/projects/anemone/robot.html
79
Alessandro Valli | Notes on Natural Interaction
Reactrix
A visual display system that responds to people instantly and in
real-time with dramatic visual effects and entertaining gameplay,
to deliver an engaging experience people not only remember, but
seek out.
http://www.reactrix.com/
reacTable*
The reacTable* is a novel electronic music instrument with a
tangible user interface. The goal of the project is the creation of a
state-of-the-art interactive music instrument. It is collaborative,
intuitive, sonically challenging and interesting, totally controllable.
http://www.iua.upf.es/mtg/reacTable/
ToneTable
ToneTable is a sound and computer graphics installation which
enables up to four people to collaborate on exploring varied
dynamical relationships between media.
http://www.shape-dc.org/articles/pdf/DAFX-01.pdf
Open Window
An ambient virtual window for bolstering wellness and healing
potential during a hospital stay.
http://www.medialabeurope.org/hc/projects/openwindow/
EventScope Table
The medium we chose to pursue is a table that is front projected
from above. The interaction is done by an infrared stylus that is
tracked by a camera.
http://www.etc.cmu.edu/projects/eventscope/goals.html
Stomping Ground
It is a permanent installation consisting of a musical carpet and a
projection of live video with superimposed blobs.
http://acg.media.mit.edu/people/simong/stompingGround/index.html
Audiopad
It is a composition and performance instrument for electronic music
which tracks the positions of objects on a tabletop surface and
converts their motion into music.
http://web.media.mit.edu/%7Ejpatten/audiopad/
80
Alessandro Valli | Notes on Natural Interaction
One2One
We are developing a technological infrastructure for creating
personalized ambient communication links to enhance a sense of
presence and togetherness between two distant individuals. We are
exploring a variety of passive sensing and display devices to suit
individual taste and the character of the relationship.
http://www.medialabeurope.org/hc/projects/one2one/
Wanderful Alcove
The magic wand presents an interesting design opportunity as a
form for a tangible computer interface. In addition to exploring the
technology needed to build a magic wand interface, this project
focuses on role-immersion scenarios in which these interfaces can
have a socially tranforming effect on their users.
http://www.medialabeurope.org/hc/projects/wanderfulalcove/
Virtual Space
The aim has been to create intuitive user interfaces that blend in
with the surroundings of the user. The starting point of the project
to research and develop bodily and spatial user interfaces has been
the natural way in which people move and act.
http://www.vtt.fi/tte/projects/lumetila
Digital Seed
The Digital Seed is a virtual alter-ego of a real seed, he lives in a
cube. The physical actions on the cube affect the inner virtual
world where the seed lives and grows.
http://www.mle.ie/%7Emauroc/digitalseed/
UrineControl
The system uses computation to enhance the act of urination.
Sensors in the back of a urinal detect the position of impact of a
stream of urine, enabling the user to play interactive games on a
screen mounted above the urinal.
http://www.monzy.org/urinecontrol/
Audience Interaction
A variety of techniques that enabled members of an audience to
participate, either cooperatively or competitively, in shared
entertainment experiences. These techniques allow audiences with
hundreds of people to control onscreen activity.
http://www.monzy.org/audience/
81
Alessandro Valli | Notes on Natural Interaction
Calm Technology
Information technology is more often the enemy of calm.
Technologies encalm as they empower our periphery.
http://sandbox.xerox.com/weiser/calmtech/calmtech.htm
EyePliances
Appliances and devices that detect and respond to human visual
attention using eye contact sensors. EyePliances receive implicit
input from users, in the form of eye gaze, and respond by opening
communication channels.
http://www.hml.queensu.ca/papers/p550-Shell-CHI2003.pdf
HoloWall
The HoloWall is a wall-sized computer display that allows users to
interact without special pointing devices. The display part consists
of a glass wall with rear-projection sheet behind it. A video
projector behind the wall displays images on the wall.
http://www.csl.sony.co.jp/person/rekimoto/holowall/
Installation
The system consists of a window through which the scene is viewed
and a stylus with which objects are manipulated. The window is a
flat panel display with a tiny camera mounted on the back showing
a live video image of the room as seen through the window.
http://acg.media.mit.edu/people/simong/installationNew/intro.html
82
Alessandro Valli | Notes on Natural Interaction
Fisheye Menus
We are investigating techniques to support selection of an item
from a long linear list. The primary technique we are looking at is
the application of fisheye views to linear lists. Live demo.
http://www.cs.umd.edu/hcil/fisheyemenu/
Magic Lenses
Toolglass widgets are new user interface tools that can appear, as
though on a transparent sheet of glass, between an application and
a traditional cursor. These widgets may incorporate visual filters,
that modify the presentation of application objects to reveal
hidden information, to enhance data of interest, or to suppress
distracting information.
http://www2.parc.com/istl/projects/MagicLenses/doc/TGMLSiggraph93.ps
VisiPhone
VisiPhone is a communication object that opens a graphical as well
as an audio portal through space. It is designed to provide a
continuous, ubiquitous connection between people in different
places.
http://web.media.mit.edu/%7Ekkarahal/projects/visiphone/
iCom
It is a media installation that forms a bridge between different
locations. It operates in a continuous and background mode,
integrated with the surrounding space. The portal enables
awareness of remote activity and promotes a sense of connection
among those generating it.
http://www.medialabeurope.org/%7Estefan/hc/projects/icom/
Exertion Interfaces
An Exertion Interface is an interface that deliberately requires
intense physical effort. We designed, developed, and evaluated an
Exertion Interface that allows people who are miles apart to play a
physically exhausting ball game together.
http://www.exertioninterfaces.com/
83
Alessandro Valli | Notes on Natural Interaction
Jam-O-World
This is a multi-user interactive musical device. Intuitive input
devices with real-time computer graphics on a tabletop surface for
collaborative gaming and music-making.
http://www.etc.cmu.edu/projects/jamoworld/index.htm
Roomservice, AI-style
Can a room be intelligent? Thoughts and work of four people who
not only believe the answer is yes, but are working towards making
this happen.
http://www.ai.mit.edu/people/mhcoen/ieee.pdf
Tangible Bits
Tangible Bits allows users to "grasp & manipulate" bits in the center
of users’ attention by coupling the bits with everyday physical
objects and architectural surfaces.
http://tangible.media.mit.edu/papers/Tangible_Bits_CHI97/Tangible_Bits_CHI97.pdf
SenseTable
A system which tracks the positions of intelligent objects on a
tabletop surface, and projects information onto the objects
themselves.
http://tangible.media.mit.edu/papers/SenseTable_CHI01/SenseTable_CHI01.pdf
Narrative Spaces
Interactive narrative spaces. These spaces are orchestrated such
that people's presence and movement drives the presentation of
digital media.
http://xenia.media.mit.edu/~flavia/Papers/NarrativeSpaces.pdf
84
Alessandro Valli | Notes on Natural Interaction
Attentive Toys
We describe an attentive system that pay attention to people so
they can attend to people’s needs using visual and audio sensors.
We implemented it as a visually interactive toy robot.
http://www.umiacs.umd.edu/users/yaser/haritaoglu.pdf
Anthropos Project
The Anthropos project explores the fusion of AI research both as a
tool for investigating human machine interaction and human
cognition as well as a technological arena inspired by human
cognition. The objective is to understand how to explore the role of
anthropomorphism in HMI and balance issues of mechanistic vs.
humanlike capabilities.
http://anthropos.mle.ie/
ChatCircles2
ChatCircles2 is a chatroom that uses graphics in a different way
than most current graphical chatrooms. Instead of having pictorial
avatars, participants in ChatCircles2 use color and geometric form
to convey social presence and activity.
http://chatcircles.media.mit.edu/
85
Alessandro Valli | Notes on Natural Interaction
The KidsRoom
The KidsRoom was a fully-automated, interactive narrative
playspace for children. Using images, lighting, sound, and computer
vision action recognition technology, a child's bedroom was
transformed into an unusual world for fantasy play.
http://vismod.www.media.mit.edu/vismod/demos/kidsroom/kidsroom.html
SmartDesk
SmartDesk is a project of the Perceptual Computing group at the
MIT Media Lab and encompasses experimentation on a range of
computer-based perceptual input and output systems in a personal
work environment.
http://www-white.media.mit.edu/vismod/demos/smartdesk/
DreamSpace
The DreamSpace allows users to collaborate in a shared space. The
system "hears" users' voice commands and "sees" their gestures and
body positions. Interactions are natural, more like human-to-human
interactions.
http://www.research.ibm.com/natural/dreamspace/
Body Mnemonics
Body mnemonics is a meta tool for portable devices that enhances
their usability and makes them more responsive to our cultural
background on the basis of three principles: proprioseptic sense,
our outlook on our own bodies, and the "method of loci" mnemonic
device.
http://www.mle.ie/~jussi/projects/body_mnemonics/index.php
86
Alessandro Valli | Notes on Natural Interaction
Pfinder
The Person Finder (Pfinder) combined a traditional segmentation
framework with sound classification theory. The result was a
system that provided solid segmentation, plus a model of the body
that yielded more information about the human than classical
figure-ground segmentation alone.
http://www.cs.ucsb.edu/%7Ecs281b/papers/Wren.pdf
Facilitator Room
The facilitator room project is an attempt to observe, model, and
affect the interaction patterns of its users. This involves sensing the
user's motions and sounds using computer vision and audition
technologies, and then interacting with them using active
components in the room.
http://whitechapel.media.mit.edu/facilitator/introduction.html
MagicBoard
The MagicBoard project aims at augmenting a perfectly ordinary
whiteboard-like surface with electronic capabilities, via a video
projector and a pan/tilt/zoom camera.
http://iihm.imag.fr/demos/magicboard
Rubella
The idea with Rubella is to make people aware of their bodies and
the way they lead their lives. We visualize the lack of exercise by
reflecting it on an artefact, the dog Rubella.
http://www.rubella.tk/
KidStory
Tangible technologies were developed to support room sized
collaboration amongst groups. Groups of children could interact
using a magic carpet for navigation and bar coded and tagged
objects to insert story elements (pictures and sounds).
http://www.virart.nott.ac.uk/Projects_Kidstory.htm
Nebula
Nebula is an interactive projection system designed to enrich the
experience of going to bed, sleeping and waking up. It provides
intuitive and natural ways of physically participating in a virtual
experience, through simple body movements and gestures.
http://www.design.philips.com/smartconnections/nebula/
87
Alessandro Valli | Notes on Natural Interaction
Interactive Tapestry
Most of today’s interior decorations in home environments are
static. It is not possible to interact with and change them in real-
time. Our thought is that it should be possible to create a form of
informal interactive art that is easily accessible in ones homes.
http://outrun.idc.cs.chalmers.se/%7Eit3chje/uc/Projekt/index.html
88