Name: CuerpoSonoro
Author: Mara

The question

What if your body were a musical instrument? Not a metaphor, not a dance, an actual instrument where every gesture produces a unique sound, where your posture shapes the sonic landscape around you.

CuerpoSonoro was born from that question. It's a software project that explores the connection between body movement and real-time sound generation, turning the human body into an expressive musical interface.

How it works

The software captures body movement through a camera and translates it into musical parameters. Every gesture, every posture generates a different sonic response. There are no scores or pre-recorded sequences: you create the sound with your own body in real-time.

The key artistic principle: the body doesn't "play notes" — the body shapes sound. It's not an instrument that triggers discrete events, but an interface that continuously moulds a living soundscape. Any position of the body produces an interesting sonic state; there are no "wrong notes". It's like putting your hands in clay: there's always a shape, only which shape changes.

Under the hood

CuerpoSonoro runs a real-time pipeline that goes from camera capture to audio output in under 80 milliseconds. Here's how data flows through the system:

1

Capture

Camera feed at 640×480

→

2

Pose

33 body landmarks detected per frame

→

3

Features

17 movement descriptors extracted

→

4

Mapping

Features mapped to sound parameters

→

5

Sound

Real-time audio synthesis

Try the web demo

The camera records nothing. Only arrays of coordinates are processed — nothing is uploaded to any server.

Try the demo

What the system measures

The vision_processor module extracts dozens of descriptors from each frame. Each one captures a different dimension of movement and maps to a specific musical parameter. Here are the main ones.

Energy

Volume and filter resonance

Overall motion intensity: velocity of key joints (wrists, ankles, nose) between frames. High energy means the body is moving fast and with amplitude.

Symmetry

Stereo panning

Left–right balance of the body. Calculated from the horizontal deviation of both wrists relative to the centre of the frame.

Smoothness

Filter cutoff frequency

Fluidity vs. abruptness of movement. Measures how abruptly acceleration changes (jerk). Slow, sustained gestures score high; sudden ones score low.

Foot position

Chord selection (I, IV, V, VI)

Horizontal centre of both ankles in the frame space. Divides the capture zone into four harmonic regions.

Knee angle

Chord velocity (loudness)

Average knee flexion angle. Straight legs = maximum. Bent knees = lower value. Allows natural crescendos and decrescendos through the body.

Hand height

Melodic note selection

Vertical position of each hand relative to body height. Right hand controls the lower octave, left hand the higher octave.

Hand jerk

Note trigger

Detects sudden wrist movements. High jerk in a single frame is the trigger condition: it fires a note on event.

Elbow–hip angle

Pitch bend / vibrato

Angle between arm and torso for each side. Arm close to the body = stable pitch. Extended arm = glissando. Oscillating movement = vibrato.

Head tilt

Global tonal filter (CC74)

Lateral tilt of the head, measured from the height difference between both ears. Controls a global filter affecting all sound.

Web demo vs. installation

MediaPipe running directly in the browser (JS)

MediaPipe in Python — CPU, Metal or TensorRT (Jetson GPU)

Audio synthesis via Web Audio API

SuperCollider synthesis or MIDI/MPE to external synth

5 core motion descriptors

Dozens of motion descriptors, 12 MPE features

Accessible from any device with a webcam

Designed for live performance and physical installation

Why this project matters to me

This project grows out of a need to return to being. To gather the pieces and learn to inhabit a body that for a long time existed only as armor, as a place of pain and damage, as something wrong that had to change at any cost.

CuerpoSonoro is my way back to it. To listen to it. To let its movement (imperfect, real, mine) be the origin of something beautiful, not the consequence of the violences that run through it.

Technology stack

Computer vision

MediaPipe Pose

33-landmark body pose detection. Runs on CPU, Metal (Apple Silicon) or GPU (Jetson).

OpenCV

Camera capture and frame processing. Low-latency buffer, 30 FPS.

Audio synthesis

SuperCollider

Algorithmic sound synthesis via OSC. Real-time SynthDefs controlled by motion features.

MIDI/MPE

MIDI Polyphonic Expression output for external synthesizers (Surge XT). Two modes: classic and musical.

Core

Python

Core application logic. Feature extraction, OSC/MIDI communication, pipeline orchestration.

Infrastructure

Docker + Hetzner VPS

Portable, reproducible deployment. Two-service architecture to minimise inter-process latency.

Testing

pytest — 129 automated tests

Source

Open Source (MIT)

Explore the code

CuerpoSonoro is open source. Dive into the code, run it locally, or build on top of it.

GitHub — CuerpoSonoro Live demo

Got a similar project in mind?

Get in touch

hello@maramotto.com