Case study / interactive sound
Sun and Us.
A field recording becomes a shared performance between the sun, a place, and the person listening.
Yiting Liu · Interactive concept, creative coding, sound and visual system · p5.js, Tone.js, PoseNet
Situation
Make distance audible.
Radio Aporee holds recordings from many places. I wanted to create a way to experience those places through changing celestial data, then invite a listener to affect what they heard.
Task
Let data and people play together.
The browser piece needed to bring field recordings, ISS and sun data, responsive visuals, and human movement into one understandable interaction. A visitor could explore with direct controls or choose camera-based movement.
Action / experience
A soundscape you can enter.
I built the audiovisual system in the browser. The tracking demonstration above shows how Us mode makes body movement an optional instrument. The manual demonstration below shows how anyone can shape the recording directly with Sun controls.
Action / system architecture
How the symphony is connected.
Listen to the world
Radio Aporee field recordings supply the source material. Local recordings keep the piece playable when its older data service is unavailable.
Locate the sun and ISS
Live position and solar data give the soundscape a changing sense of place and time.
Shape the sound
Tone.js controls looping, pitch, distortion, and filtering. Visitors can also move the Sun sliders directly.
Join with movement
Choosing Us loads the camera and pose tracker. Head and wrist positions become optional controls for the same sound parameters.
Simplified from the project source. Camera input begins only if the visitor chooses Us; the drawing and sound controls work without it.
Result
An exhibited, playable work.
Sun and Us was shown at the 2020 ITP Winter Show, One Mile Gallery, and the 2021 Museum of Wild and Newfangled Art Biennial. The browser demo makes the interaction available beyond those exhibitions.
For the current version, I repaired the start flow and reduced rendering work so visitors can reach the soundscape without loading camera tracking up front.
Explore the work
Make systems people can feel.
I design and build interactive work across sound, screens, and physical spaces.
Inquire about a project