01
Make sound spatial
Each prepared stem has its own circle. Moving it left or right changes its stereo pan; moving it vertically changes its level.
← Technical work
Case study / interactive learning
Spatial Music Lab.
An instrument becomes a movable object, making the sound of a mix easier to explore.
NYU Shanghai Creative Innovation Lab · Adjunct Researcher in Music and AI · Experience design, Next.js, Tone.js, MediaPipe
Situation
Hear where each sound sits.
Spatial audio can be abstract when the controls are hidden in a traditional mixer. For a research project at NYU Shanghai, the learning experience needed to make an audio decision visible and immediately audible.
The project uses prepared music stems so learners can focus on the relationship between a source’s position and the resulting mix.
Task
Turn listening into interaction.
As an adjunct researcher, I designed and developed a browser interface where students could move individual instruments, hear the effect in real time, and compare a passage. It also needed to be documented so the lab could continue developing it.
Action / working interaction
Move a circle. Change the mix.
The first recording shows me using both hands to move sound sources and use the on-screen controls. The second shows the same mix explored with a mouse. Play the videos with sound to hear the changes.
Action / implementation
Interaction with an audible response.
02
Make the system legible
Color, labels, and reactive rings help learners connect the visible source with the sound they hear. The stage scales with the window.
03
Support exploration
Playback, section navigation, looping, and mix recording let a class compare choices. Optional camera-based hand control adds another way to move a source.
System architecture / simplified
From gesture to sound.
- 01
Prepared stems
Audio files and section metadata define the instruments and timeline.
- 02
Browser stage
Dragging a circle updates its position; optional MediaPipe hand tracking can provide gesture input.
- 03
Tone.js mix
Each stem runs through playback, stereo panning, and level control into the master mix.
- 04
Listen and see
The browser plays the mix while audio analysis drives each circle’s reactive display; the mix can be recorded.
Result
A research tool people could try.
I delivered a working browser prototype and technical documentation, demonstrated it in university classrooms, and gathered educator and student feedback for further development.
The result shows both sides of my practice: shaping an approachable learning interaction and building the audio, visual, and gesture systems that make it work.
Explore the work
Build something people can feel.
I design and develop interactive systems for learning, live events, and public experiences.
Inquire about a project