Going Beyond the Display: A Surface Technology with an Electronically Switchable Dif

[krishnasai]

Going Beyond the Display: A Surface Technology with an Electronically Switchable Diff user
A Seminar Report
by
Prasanth.R.S
M105111
Department of Computer Science & Engineering
College of Engineering Trivandrum
Kerala - 695016
2010-11

Abstract
A new type of interactive surface technology based on a switchable projection screen is
introduced, which can be made di use or clear under electronic control. The screen can be
continuously toggled between these two states so quickly that the switching is imperceptible
to the human eye. It is then possible to rear project what is perceived as a stable image
onto the display surface, when the screen is in fact transparent for half the time. The clear
periods may be used to project a second, di erent image through the display onto objects held
above the surface. At the same time, a camera mounted behind the screen can see out into
the environment. It explore some of the possibilities this type of screen technology a ords,
allowing surface computing interactions to extend "beyond the display".Single self contained
system that combines these o screen interactions with more typical multi touch and tangible
surface interactions is presented in this paper. It describe the technical challenges in realizing
this system, with the aim of allowing others to experiment with these new forms of interactive
surfaces.

[attachment=8558]

1 Introduction
Interactive surfaces allow us to manipulate digital content in new ways, beyond what is
possible with the desktop computer. There are many compelling aspects to such systems,
for example the interactions they a ord have analogies to real world interactions, where it
manipulate objects directly with our ngers and hands. Some systems play on these analogies
further by associating tangible objects with the digital interface, again leveraging our skills
from the real world to interact in the digital. Many di erent interactive surface technologies
have been developed over the past few decades. One major category is the rear projection
vision system used in many research prototypes and now even commercial products. Such
systems support capabilities beyond that of regular touch screens, permitting multiple ngers,
and even other tangible objects near the surface to be sensed. Because the optical path is
behind the display, occlusions are greatly mitigated compared with front projection and vision.
The di user is a key part of these systems, displaying the projected image and ensuring that
the camera can only detect objects close to the surface. However, the use of a di user also
means that the displayed image, the sensing, and the hence user interaction will inherently be
bound to the surface.
This paper presents a new surface technology called SecondLight which carries all the bene ts
of rear projection vision systems, but also allows to extend the interaction space beyond the
surface. Like existing systems it can display, sense, and therefore interact on the surface, but it
can also simultaneously project and image through the projection screen. This key di erence
stems from the use of a special type of projection screen material, which can be rapidly switched
between two states under electronic control. When it is di use, projection and imaging on the
surface is enabled; when clear projection and imaging through is possible. This opens up the
three dimensional space above (or in front of) the surface for interaction. Because projection
is no longer limited to the surface, it can be used to augment objects resting on or held above
the primary display. Furthermore, both objects and user gestures can be sensed and tracked
as they move around in 3D space.
SecondLight represents a new approach to support these types of extended surface computing
interactions, bringing together ideas from diverse research areas, and integrating them into a
single self-contained system. It can feel that the use of switchable di users is particularly
relevant for the interactive surface and tabletop communities providing systems with the best
of both worlds. The ability to leverage the bene ts of a di user and rear projection-vision
for on surface interactions, with the potential to instantaneously switch to projecting and
seeing through the surface. In this paper describe the construction and operation of current
SecondLight system in full to allow practitioners to experiment with this new form of interactive
surface. It is worth reiterating that these novel features of SecondLight coexist simultaneously
with conventional surface-based interactions such as touch and tangible input. Rear projection
vision surface technology explained with gure 1.
5
2 New rear projection-vision surface technology
In gure 1 an image is projected so it appears on the main surface (far left). A second image
is projected through the display onto a sheet of projection lm placed on the surface (middle
left). This image is maintained on the lm as it is lifted o the main surface (middle right)
Finally, the technology allows both projections to appear simultaneously, one displayed on the
surface and the other on the lm above, with neither image contaminating the other (far right).
Figure 1:
2.1 Motivating Secondlight(A Trick of the Light)
The key technical component in SecondLight is an electronically controllable liquid crystal
material similar to that used in privacy glass, an architectural glazing material which can be
switched between transparent and di use states as shown in Figure 2. When transparent the
material is clear like glass, and most of the light passes straight through. When di use the
material has a frosted appearance, and light passing through in either direction will scatter. In
SecondLight this material is used as a rear-projection screen, resulting in a system which can
display digital content on the surface whilst it is di use, or project through the surface when
switched to its clear state.
It can continuously switch the screen between these two states so quickly that it does not

icker. It is then possible to rear-project what is perceived as a stable image onto the display
surface, when the screen is in fact transparent for half the time. During the clear periods a
second image can be projected onto any suitably di use objects held on or above the display.
As shown in Figure 1, by careful control in this way two di erent projections can be displayed
on and through the surface, seemingly at the same time. Further, the images projected through
the surface do not contaminate (or bleed onto) the ones on the main screen, and vice versa.
This provides two independent projection channels.
A camera placed behind the switching screen can capture images when the surface is di use.
The light scattering property of the screen in this state makes touch detection much easier.
Additionally, when the screen is clear, the camera can see right through the surface into the
space above (or in front of) the display. This allows the accurate up close touch input to be
augmented with richer data sensed at greater depths. A variety of diverse vision techniques can
then be used, for example recognizing hand gestures from a distance, tracking di use objects
with markers in order to project onto them, or detecting faces to "see" the number and position
of people around the surface. The ability to simultaneously project both onto a surface and
6
through it enables a number of interesting scenarios which explore further in the remainder of
this section.
Figure 2: A switchable screen, in clear state (left) and di use state (right). Note: the hand is
very close to the underside of the surface
2.2 On the Surface Interactions
Figure 3: Some of the typical on surface interactions a orded by SecondLight include multi-
touch input (left) and detection of physical objects such as a real paint brush and tangible
interface widget (right).
7
With the projection screen in its di use state, SecondLight exhibits the established properties
of multi touch and tangible surfaces. Two examples are shown in Figure 3. Aside from allowing
an image to be projected on the surface, the di user plays a key role in detecting when ngers
and other tangible objects are touching or very close to the display. This is an essential feature
for direct input surfaces because the interaction relies on robustly detecting touch. Since the
di user causes light to scatter, only objects very close to the surface will be clearly imaged
and this simpli es touch detection, mitigating the need for computationally expensive methods
such as stereo vision. It can detect a wide range of objects such as ngers, hands, brushes,
game pieces, mobile devices and so forth, and also support the unique identi cation of objects
using retro re
ective markers
8
3 Projection Beyond the Surface
SecondLight allows the combination of traditional surface interactions with more advanced
features that extend interaction beyond the surface. This is made possible because it can
project through the display surface, allowing physical objects that have suitable surfaces to be
augmented with projections emanating from the SecondLight unit.
Figure 4: Creating tangible layering e ects is extremely simple with SecondLight. A translu-
cent sheet of di use lm is placed above an image of a car to reveal its inner workings (middle)
or above the night sky to reveal the constellations (right). Plain images are shown on left.
Circular acrylic discs with di use topsides can be used to create a magic lens e ect.The magic
lens image is maintained even if the disc is lifted well away from the surface, can be seen in the
gure.
As shown earlier in Figure 1, one example is a thin sheet of di use lm which is augmented by
projection from below, as it rests on the surface and even when lifted up. This projected image
is maintained on the lm whilst projecting an entirely di erent image on the primary surface,
without cross contamination of the two images. The dual projection capabilities of SecondLight
can be used to create interesting layering and magic lens e ects. Instead of projecting two
entirely unrelated images, the image projected through is visually connected to the one being
projected on the surface. For example, as shown in Figure 4 top left, the image projected on the
surface could be a car, with an associated image that reveals its inner workings being projected
through (Figure 4 middle). In this scenario, if a user passes a piece of translucent material over
the display, otherwise hidden information is revealed, creating a two layer e ect.
3.1 Tracking mobile surfaces
In gure 5 mobile projection surfaces with passive and active markers being held at di erent
orientations above the surface. The markers de ne the location of the object, which is tracked
using the camera imaging through the surface. This allows correction of the through projection
enabling it to be distortion free and appear centered on the mobile surface. Top: a
exible
sheet with passive retro re
ective marker strips. Middle: an actively tracked surface with its
own battery powered IR light source which also allows multi-touch input on the mobile surface
to be sensed through the SecondLight surface (middle right). Bottom: the projected image
can be corrected for distortion as it is moved and tilted, thereby supporting quick and natural
reorientation into more conformable viewing positions. For example, tilted vertically towards
the user or oriented towards another user for viewing.
9
A zoom e ect can be applied as the lens is moved towards or away from the surface making
its behavior more analogous to a real magnifying glass or new layers of information could be
revealed as it rotates. SecondLight supports such tracking if the magic lens is augmented with
either passive (retro-re
ective) or active (powered infrared LED) tags. Indeed, by tracking the
position and orientation of a mobile display surface such as a magic lens, it is possible to alter
the projected image in real time so that it appears centered and without any foreshortening or
other distortion, even as this mobile surface is manipulated in three dimensional space.
Figure 5
This allows to create extremely cheap and lightweight peripheral surfaces that can be used in
conjunction with the SecondLight display. Users can shift their attention between the primary,
shared display and one or more smaller, mobile displays, viewing and interacting with content
on both as they please. The mobile surfaces can be tilted towards the user or even held in the
hand and the rendered content will track them accordingly, as illustrated in Figure 5. Further,
as also shown in this gure and described in detail later, a novel active tagging scheme is applied
to the surface which not only supports tracking of the mobile surface, but allows this object to
support multi touch input.

3.2 Other Interesting Tangible Possibilities
Systems such as Reactable and Microsoft Surface track the position of tangible objects placed
on the surface in order to project visual content immediately below or around the object. They
also support user input through direct manipulation of these tangible objects.Can support
this type of interaction with SecondLight but the additional ability to project light through the
surface allows to explore new designs for tangible objects. In the simplest scheme, a transparent
object with a di use top surface allows a projected image to be displayed on top of the object.
In a more complex embodiment, shown in Figure 6, circular prisms built into the object allow
the projected image to be totally internally re
ected onto the sides of the object.
3.3 Tracking mobile surfaces
Figure 6
The ability to shine light through the display, gives rise to other novel tangible object designs.
In gure 6, it demonstrate an object that uses internal prisms to project the incoming light onto
its sides. The prism inside the object is shown right. The e ect of the prism on the projection
is shown left. This middle image shows the screen in a non switching di use state illustrating
the behavior of a typical projection screen.
3.4 Gesturing and Input from a Distance
With the projection screen clear, a completely un-attenuated image can be captured by the
camera. With su cient illumination it is possible to track the users hands from a distance and
identify hand gestures and poses using computer vision techniques.
A stereo camera, 3D camera technology, or structured light could be used to support depth
based interactions (although have yet to explore these possibilities). Alternatively, as have
shown, tags and markers can also provide approximations for depth.Although interacting from
a distance breaks the direct manipulation metaphor, it does open up yet another input modality.
Feedback can also be provided during these depth based interactions by projecting onto the
underside of interacting objects. It may even be possible to provide coarse feedback to the
user when gesturing from a distance by illuminating the underside of hands, using for example
changes in color to indicate if a gesture has been recognized by the system (see Figure 7).
This concludes the broad overview of some of the possibilities that SecondLight enables. A key
technical challenge for interactive surfaces has been the low level sensing techniques employed
11
to detect the movements of ngertips and objects on the display surface. A variety of resistive,
capacitive and inductive schemes have been employed in the past. Both resistive and capacitive
touch screens have been scaled to support multi touch. However, one major issue is that these
systems cannot recognize a wide range of un tagged objects in addition to ngertips. Such
capabilities are an essential part of interactive surfaces, particularly ones that need to support
tangible input.
Figure 7: Secondlight allows gesture-based interactions with the primary surface from greater
distances than many back projected systems(left).
It is important to stress that SecondLight has not been developed to address a speci c
problem or application, but rather is an exploration of a new technology which believe has the
potential to deliver a range of interesting and compelling user experiences. Realizing the system
and even the simple proof of concept demonstrators presented above has been challenging.
12
4 Hardware
4.1 The switchable di user
A polymer stabilized cholesteric textured liquid crystal (PSCT-LC) optical switch is used.
PSCT-LC is similar to polymer dispersed liquid crystal (PD-LC), a material that is commonly
used as privacy glass in o ces. Both PD-LC and PSCT-LC are made from a special material
containing liquid crystal molecules which are normally randomly oriented and which therefore
scatter light in all directions. However, they become untwisted and therefore aligned in response
to a suitable electric eld which may be generated by applying a voltage across two parallel,
transparent substrates on either side of the screen.
4.2 Driving the di user
It can continually switch PSCT-LC screen between di use and clear states at 60Hz, which found
was a su cient frequency to avoid
icker perception when looking directly at the surface. Each
cycle consists of around 8.3ms when 150V is applied to the screen to make it clear followed by
8.3ms with no applied voltage, at which point it returns to its natural di use state.
4.3 Projector setup
Figure 8: The layout of the main Secondlight components; on the left is a cross-section
through the side of theunit and the right is a photo from front.
In order to project di erent content on the surface versus through it, it need to alternate
between two di erent images in sync with the switching di user. The"on" image is displayed
for 8.3ms followed by the "through" image for 8.3ms and so on. The frame rate for each of these
two images is 60Hz. It use two o -the-shelf Hitachi CPX1 60Hz projectors in combination with
fast optical shutters to create the two interleaved 60Hz images. Like the switchable di user,
the shutters are liquid crystal based, but in this case they switch to a black, light blocking state
when they are not clear. Blocking the light from the rst projector projection surface is clear
causes the image from the second projector to pass through the PSCTLC on the next part of
the cycle it reverse the shutters so that the "through" projector is blocked and the light from
the rst projector is displayed on the surface.
13
4.4 Camera con guration
In addition to projecting both onto and through the surface, it also image what is on the surface
and beyond using two ImagingSource DMK 21BF04 cameras mounted behind the di user.
Whilst SecondLight allows the capture of full colour images, to date tted IR pass lters to
limit imaging and sensing to the infrared spectrum.
It use both di use and FTIR IR light sources in conjunction with the rst camera to sense
multiple ngers and other objects. The FTIR light source consists of 264 Osram SFH 4255 high
power IR LEDs which are distributed on a 6mm pitch along the edges of a 490 x 390mm sheet
of4mm thick clear acrylic. The LEDs are wide angle (60), 850nm devices which are surface
mounted to a custom made PCB at right angles. They are driven continuously at around
80mA in chains of 6 devices from a 12V PSU. Here use the same LEDS as the source of di use
illumination. The second camera is triggered so that it grabs images when the PSCTLC is
clear, and can therefore also see far beyond the surface. Any IR sources in the eld of view,
such as IR LED markers, will be clearly visible.
4.5 Putting it all together a small matter of timing
All the components of SecondLight are held together using a lightweight frame made from a
modular extruded aluminium system from Bosch Rexroth. Note that it has adopted a tabletop
con guration, but a vertical setup is equally feasible. The PSCTLC, clear acrylic overlay and
IR LEDs are held in place using a black acrylic bezel which is secured to the top of the frame.
The various power supplies for the system rest on a shelf at the bottom of the frame.
14
5 Limitations And Future Work
Currently this system only image in the IR spectrum, but imaging visible light could enrich
the interactions that the system supports. For example, a high resolution digital stills camera
located behind the screen could be triggered to capture images when the surface is clear, in a
similar way to TouchLight. This would allow to image or scan objects such as documents in
color (both on the glass using a projector for illumination, and potentially o the glass given
enough ambient light). Have also veri ed the feasibility of imaging both "on" and "through"
at 120Hz using two cameras, each of which is triggered twice during the relevant 8.3ms time
slot, once at the start and once towards the end, although again, it has not yet incorporated
this into prescribed prototype.
In practice found the projector con guration described in the previous section, with the
"through" projection shining away from the front of the tabletop, limits the likelihood that
light is projected directly at the user. For other scenarios it may be possible to use the through
camera to detect and track users faces so that it can project black in the region of any eyes
detected. The use of two projectors which run continuously but whose output is optically
shuttered is very
exible for example it allows ne grained control of the duty cycle of the
"on" and "through" projectors. But a solution which uses a single, high frame rate projector
would be a lot less wasteful of light energy, and hence potentially brighter as well as being more
compact and probably cheaper.
It use projectors with a high depth of focus, and adjust the "through" projection to give a
crisp image anywhere within around 150mm of the surface beyond this it starts to gradually
blur. A motorized focus system, combined with depth sensing of the surface being projected
onto, would be useful. However, this would still be limited to focusing on a single surface,
whereas the image from a laser projector is always in focus, so this is an alternative technology
it would like to explore.
15
6 Conclusion
SecondLight, a novel surface technology is introduced, which allows interactions beyond the
display alongside more traditional on surface interactions. The technology brings together
diverse ideas and concepts from many di erent research areas and integrates these into a single,
self contained solution. The speci c contributions of this work are as follows:
The use of switchable di users for interactive surfaces.
Simultaneous projection on and through the surface without cross contamination of these
projections.
Tracking of and projection onto objects in real-time through a rear projected tabletop
whilst maintaining entirely di erent content on the primary surface.
Projecting images through a tabletop onto perpendicular sides of tangibles including non
planar surfaces using prisms.
Projecting images through a tabletop onto perpendicular sides of tangibles including non
planar surfaces using prisms.
Multi touch on the surface and on secondary displays above the surface with the sensing
and processing integrated in the tabletop unit.
Switchable di users present an exciting technology for interactive surfaces and tabletop
allowing to combine the bene ts of a di use display surface with the ability to project and see
through the surface, thereby extending interaction into the space in front of the display. The
hardware and software used to construct this prototype is described in some detail, allowing
others to explore these new forms of interactive surfaces.

References
[1] E. A Bier. Toolglass and magic lenses: the see-through interface'. In Proceedings SIG-
GRAPH, Dec 2009.
[2] P. Dietz and Leigh. D. DiamondTouch. Multi-user touch technology'. ACM, Dec 2008.
[3] Holman.D Borchers.J Smith.D, Graham.N. 'low-cost malleable surfaces with multi-touch
pressure sensitivity, tabletop'. ACM, Dec 2008.
[4] Apple iPhone Multi-touch. ' http://appleiphone/'.
[5] Izadi-S. Butler A. Rrustemi A. Hodges, S. and B. Buxton. 'thinsight: Versatile multi-touch
sensing for thin form-factor displays'. ACM, FeB 2009.