A project for Artists
Physical Computing
Extending the Human Computer Interface beyond point and click

"The Direct Manipulation (DM) style of user interface made popular by the Macintosh is becoming a de facto standard. Rather than being taken as a point of departure, it appears to be taken more as a standard to achieve." *1*


The goal of the project is to extend human computer interaction beyond the mouse and keyboard. Most artists initially find that drawing with a mouse on a pad while looking at a screen is an unnatural process. Eventually, they get used to it but this is the wrong approach. Instead of making the artist get used to the machine the machine should adapt to the needs of the artist. One of the primary goals of this research proposal is to begin to make the machine adapt to the human instead of forcing the human to adapt to the machine. This will be accomplished in stages. The first stage is begin to make the computer aware of the user. This stage will be accomplished utilizing existing sensors, transducers and detectors. There are already many commercially available sensors that have the ability to detect real world input. One example of such commercially available technology is the sonar on a polaroid camera that measures the distance from the camera to the subject. With slight modifications, this sonar could be used to detect whether or not a user is in proximity to the computer or art piece. The second stage is to begin to utilize this awareness of the machine in useful and productive ways. There are a variety of visually oriented multimedia applications available on the market that can interpret a mouse click and trigger an event such as a projector. Perhaps the best known authoring packages are Macromedias Director and mFactorys mTropolis. Utilizing these existing authoring packages, it is possible to have them interpret the input from sensors to produce real world actions. The goal of the project is to use existing hardware and software to create truly interactive art works that are aware of the user and the audience.

Historical Precedents

Throughout history, technology has influenced artists. From Leonardo DaVinci's drawings of machines to Nam June Paik's sculptures made of televisions, there has been a constant dialog between artists and technology. This relationship between art and technology has accelerated over the last century. From the futurists' interest in speed and machines, to the modernism's belief that man could invent a machine for every human need, this dialog has produced many important works of art.

Perhaps no other invention has influenced art more strongly in the past century than the camera. As a photographer I am very aware of the major changes the camera has had on the way that artists made their work and the way that mechanical visualization techniques such as photography have freed the visual artist from the necessity of rendering. It is my belief the computer is already beginning to have at least as profound an effect on art and visualization as the camera did on the passing century.

Already, there have been several exhibitions featuring art and technology. Many contemporary artists use some form of interaction in their work. Recent major exhibitions at the SoHo Guggenheim *2* and the Cooper Union for the Advancement of Art and Science *3* have exhibited and displayed these works. While established artists may have the funds necessary to hire a programmer, most artists do not have access to this kind of technical expertise. It is the goal of this project to allow artists to focus on art instead of technology.

Specifics of the research

While the area of human computer interaction is a large and broad field, I plan to research specifically the area as it applies to artists. Painters, photographers, dancers, sculptors and musicians have specific computing needs that are often overlooked by engineers and programmers. It is the goal of this project to address those needs. For example, a dancer may wish to control the sound of the music with the motion of their body, a sculptor may wish to have the object change depending on how far away the viewer stands, a painter may wish to use their entire body to make a stroke or mark in their work not just the forefinger, a musician may wish to have the sound change according to how many people are in the audience and how they relate to one another.

Utilizing existing technology, with some modifications, the goal of the research project is to construct a more visceral, physical approach to computing for artists who are not necessarily interested in engineering problems. Being an artist and not an engineer myself, it is not my interest to develop hardware that does not yet exist. Instead, the goal is to better use existing technology by adapting it to the way humans work and make art instead of forcing humans to work and think like machines. This is a design problem and an engineering one.

Using multimedia authoring software such as mFactory's mTropolis, Macromedia's Director and Opcode's Max and existing hardware such as the Infusion Icube system, the Handy Board and the ADB/IO device along with a variety of currently available projectors, sensors, transducers, actuators and detectors, I plan to develop a more natural computing environment for artists. The goal is to produce a system that is easy for artists to use that include a variety of "real world" data acquisition devices.

Currently, an artist who wishes to accomplish a multimedia installation must work closely with a programmer to develop the interactive part of the installation. It is the goal of this project to develop tools that help the artist create digitally derived works without the aid of programmers. The current stae of interactive projects is confined to the on-screen presentation paradigm. It is essential that artists break out of the screen and into the real world.

Whether the computer is a central element in an artists work, or simply a tool bridging a technological gap, the need for real world input and output is essential. This project will deal with two different issues in the design of a new interactive art work. The first is to choose a physical effect or sensor which detects the gesture to be interfaced to the computer. The second is to adapt that sensor to the practical and artistic needs of the equipment the artist wishes to control, i.e. computer software or sculpture. The use of light sensors, for example, would follow the dynamics of a crowd to detect their arrangement and movement. A microphone and filter would measure an envelope of the noise produced by the audience and trigger an appropriate response in the art work as determined by the artist. Alternatively, light beams or infrared sensors could be employed. To make a sculpture aware of the viewer as they approach the piece, one could attach an ultra-sound transmitter that measures the change in distance from the viewer to the sculpture. Either method results in changing analog voltages which the computer converts into an action or reaction of the art work. These messages may be used directly or passed to intelligent software to create a variety of responses dependant on the actions of the viewer.

The Final Result

The final results of the project will include, but not be limited to, creating interactive sculptures that "see" the viewer - making the art works as active as the audience. I will create sculptures that change as the viewer or viewers move closer to them. For example, a set of monitors with eyes on them that follow the viewer around the room or a sculpture that brightens or changes color as the viewer approaches it.

The Testing

I plan to use an "experimental" programming paradigm in which prototype systems are built and informally tested based on the needs of other artists as well as myself. The results of the project may be developed into commercial products as add-ons or extensions to existing hardware (peripheral devices) and software (similar to Photoshop plug-ins or Director Extensions or mTropolis modifiers). The results of the experiments themselves would be exhibited in galleries or museums and disseminated on the internet.

The Next Stage

I plan to approach software and hardware manufacturers for further R&D funding after the initial phase of research is complete.

*1* Buxton, W. (1993). HCI and the inadequacies of Direct Manipulation systems. SIGCHI Bulletin, 25(1), 21-22.
*2* Both the Mediascape show and the Hugo Boss prize exhibitions heavily featured artists and technology mixing.
*3* The TechnoSeduction show at the Cooper Union.

Some useful links:


Infusion systems makes the iCube A device for Macintoshes to be controlled by, and control, external devices. Be sure to check out their list of links to other hardware manufacturers and sensory input devices.

The source for info on the ADB/IO device which is similar to the iCube is here

Read My Mind
Other notable hardware includes revised versions of the IBVA brainwave hardware, with direct brainwave-to-MIDI output allowing the user to "think" new music. The new software has an open plug-in architecture that allows you to hook the brainwave hardware and software to nearly any Mac application through the addition of scripts and so on. The possibilities seem novel and exciting...

advanced robot kits allow you to build and construct moving sensing robots with relative ease.

Connect one of our lab interfaces to the serial port of your Macintosh and you are ready to collect and analyze data from any one of our many sensors. We have two different lab interfaces for Macintosh computers:

Sensor Developments Inc.designs and manufactures industrial transducers which measure force, torque, rotational speed, and temperature, as well as associated electronic systems.

digikey is a Manufacturer of hardware and gizmos

a discussion of touchscreen info with links to manufacturers

MIT professor Fred Martin's research into LEGO robotics for kids
More Research from MIT on a small programmable Lego device for interesting projects.

Makers of the QuickCam

Logitech makes wireless mice

Software and Wetware:

Dan O'Sullivan at the NYU ITP program offers a course in physical computing
the vision and modeling group at MITwas formed in 1987 to study problems in computer vision and scene modeling. Of particular interest is some of their research projects into recogniton of action and smart rooms,chairs and other objects that are now "dumb"

gestural recognition input devices are discussed here.

Apple computer academic researchers discuss human computer interface issues as well as educational issues in computing.

A few recent interesting exhibitions have utilized human computer interfaces that have gone beyond the forefinger.

Techno Seduction at the cooper Union
The Mediascape show at the Guggenheim SoHo

Computers in art design research and education was started in the 60s by luminaries such as Robert Rauschenberg to encourage artists and scientists to collaborate

The purpose of Art & Science Collaborations,Inc. (ASCI) is to raise the general awareness of art and artists using science and technology to explore new forms of creative expression.

There are a number of theoreticians at work on the problem of technology and human computer interface as well.

a panel on the imapct of technology on culture was recently held at the New School
Simon Penny discussion page

Tactile Augmentation: Enhancing presence in virtual reality with tactile feedback from real objects

Cyberspace and the Structure of Knowledge

Designing in Virtual space by Thomas A Furness of the HITLAB

CTHEORY is an international journal of theory, technology and culture. Articles, interviews, and key book reviews in contemporary discourse are published weekly as well as theorisations of major "event-scenes" in the mediascape.

Other resources online:

the ACM Transactions on Human - Computer Interaction page (ACMCHI) is here TOCHI. The educational survey is here

The Human-Computer Interaction Group was founded in 1984 to pursue research into the design and evaluation of interactive systems. The group has a unique interdisciplinary approach which integrates rigorous formal methods from the latest software engineering research, with theories of perception, learning and error developed within cognitive and social psychology, sociology and linguistics. The group's concern is not just with the usability of computers for the standard user in isolation, but also with usability in the context of the user's task and community, and for users with special needs, such as disabled users and operators of safety-critical systems.

The Mission of the Human Interface Technology lab is to empower people by creating better ways to interface with machines.
The Human Interface Technology Laboratory is a research and development lab in virtual interface technology. HITL was established in 1989 by the Washington Technology Center (WTC) to transform virtual environment concepts and early research into practical, market-driven products and processes. HITL research strengths include interface hardware, virtual environments software, and human factors. The Lab hopes to develop a new generation of human-machine interfaces to provide solutions to challenges in a variety of domains.

fineArt forum resource directory This directory is meant to serve as a resource and jumping-off place for people interested in art, and in the possible relationships between art and technology. Over 1000 art resources that can be accessed via the Internet; websites, gophers, ftp sites, mailing lists and other types of resources are included.

Siggraph online

Siggraph calendar of events

Please send any info you may have on similar issues to me