On Site | Wind on the Wing (M) & (MR)

Mobile and Mixed Reality Experiences for Simulated Nature Integration

Proposal Overview:

Wind on the Wing (M) & (MR) is an extension of a recent gallery installation, exhibited at Wave Hill in the Bronx. Part of a series of art installations intended to connect viewers with the natural environment, these two new works, a mobile app and a mixed reality experience, will offer a virtual tonic for the overstimulation of urban life(1).

Inspired by Charles Baudelaire’s statement “I have felt the wind on the wing of madness,” the project examines our increased loss of connection to the natural world and the potential physiological impact of that loss. Exploring how we respond to the interaction of the natural and the synthetic in urban environments, the goal is to alter the user’s surroundings through mobile and mixed-reality applications and bring the calming sites and sounds of nature into everyday urban experience.

Project Design:

Rationale
My recent work explores alternative models of experience, consumption, and exchange through participatory installations that invite the audience to question precarious environmental and societal structures that humans have become inured to. A common response to my work is that it makes participants feel calm, relaxed, and re-connected. My aim is to extend this experience beyond the gallery and into the everyday through the use of mobile and augmented reality applications.

Overlooking the Hudson River in the Bronx, “Wind on the Wing” was installed at Wave Hill's Glyndor Gallery, where wind-driven pinwheels influenced sound recordings of 12 commonly heard, local Bronx birds and their associated binaural tones. The speed of each wind-turned pinwheel altered the installation’s audio and visuals. Visitors to the installation frequently requested the ability to take the audio-visual experience with them – in their words “to take the calm with them.”

Virtual reality offers a potent escape from the stresses of the real world and has shown in some studies to be highly effective at treating post-traumatic stress(2). Other studies have suggested that exposure to the sights and sounds of nature- even looking at the colors of blue and green, can provide a calming effect on the body(3).

I am intrigued by the integration of the real and the virtual. Mobile applications and mixed reality have the potential to introduce simulated nature into an overstimulating urban landscape using spatial audio, integrated object renderings, a smart phone, and/or a mixed-reality headset. Wind on the Wing (M) & (MR) will permit users to incorporate the real built environment with simulated nature interventions.

Wind On The Wing (M)
A cross-platform mobile application, Wind on the Wing (M) puts the original “Wind on the Wing” audio and images into the user’s pocket. Currently in development, this portable version of the installation allows the user to adjust the volume of the specific birds and their associated binaural tones. The pure tones, which have slightly different frequencies, create an auditory illusion when played through headphones. The brain generates a third frequency, and the listener experiences a warble or tremolo as the brain tries to reconcile the pitches entering the left and right ears.

The on-screen visuals display close-ups of the two loudest birds heard in the audio at a given time. In contrast to the capriciousness of the wind-driven audio, these images will slowly fade into one another, creating a calming visual counterpoint to the audio.

The next phase of development will incorporate user input to simulate the original pinwheel generated wind input. When the user blows into or holds the microphone up to the wind, the bird sounds will decrease as the paired binaural tones increase, altering the audio/visual feedback. Users may find that their response to the audio changes as the wind blows. Depending on the input intensity, the resulting bird symphony may at times be soothing and calm or slightly maddening.

Wind On The Wing (MR)
 “Wind on the Wind (MR)” will be a portable, mixed-reality experience using a smart phone or a relatively inexpensive, mixed-reality headset. With the accurate gyro and camera on smart phones, the goal is to place virtual elements onto the user’s surrounding, as if they were physically there. In consultation with MR/VR programmers, open source, low-cost MR kits will be evaluated, along with other emerging technologies.

Three scenes are proposed for 360° video superimposition onto a user’s everyday built environment: a bubbling brook in a light rain, a sparkling snowfall at dusk, and a misty, light-drenched stand of trees. Markers, such as permanent elements in the NYC landscape or other objects, may be used to improve scene integration and allow for mapping of video onto a transparent virtual sphere. Spatial or binaural audio forked from the original “Wind on the Wing” audio will enhance the mixed-reality experience to bring the sights and sounds of nature into the urban landscape.

Future Development:

Depending on funding, additional development will include incorporating user input to influence the mixed-reality experience. This input could include EEG or similar technologies, which use signal processing to interpret the user’s mental activity or pulse oximetry signals. For example, the commercially available Muse headset provides feedback about a user’s meditation in real time by translating brain signals into the sounds of weather. Wind on the Wind (MR) would aim to build on this auditory response and include additional visual feedback that would alter the virtual experience, such as changes in light, color, or saturation.

References:

More information about this project will be posted as it progresses.

 


 

--

Special thanks to Jake Miles for initial programming assistance.

 

 

 

Mobile App - Work in Progress:

Wind on the Wing (M) - Audio & Images

 

App Dashboard - in progress

 

Very Rough Initial Sketches for MR:

Three scenes are proposed for object mapping into a user’s everyday built environment: a bubbling brook, a sparkling snowfall, and a light-drenched stand of trees. Spatial or binaural audio will enhance the visual mixed-reality experience to bring the sights and sounds of nature into the urban landscape.

Wind on the Wing (MR)- Snow Scene Sketch

Wind on the Wing (MR)- Stream Scene Sketch

Wind on the Wing (MR)- Forest Scene Sketch

 

Original Installation at Wave Hill:

Wind on the Wing - Installation Views