Webcam Theremin: Play Music in Your Browser With Hand Gestures

I put a webcam theremin on the site: a browser instrument that turns hand movements into pitch and volume. You can play with two hands, simplify it to one hand, or use a mouse without enabling the camera. Here is how the controls work, which settings help you find notes, and what runs behind the screen.

Two hand movements control one continuous voice

The controls follow the familiar theremin arrangement: one hand changes pitch and the other changes loudness. In my version, the camera reads your hands against a mirrored video image, with a drawn pitch antenna on the right and a volume loop on the left. Move your right hand horizontally toward the antenna to raise the pitch; lift your left hand to make the sound louder.

The instrument uses the position of your palm in the image, so moving closer to the physical camera is not the pitch control. Lower the left hand toward the drawn loop to reduce the volume, or close that hand into a fist to mute the voice. Reverb can still leave a short tail after muting. Keeping the hands on their respective sides of the picture makes the controls easier to follow.

Start with the mouse, then enable the webcam

Open the instrument page and choose a starting mode. The current interface is in Russian: “Играть руками” means play with hands, while “Без камеры (мышь)” means without a camera, using the mouse. There is no account or application installation step.

Mouse mode is the quickest way to understand the playing area. Hold the mouse button down on the stage and drag: horizontal movement controls pitch, vertical movement controls loudness. Release the button to mute the voice. Once that makes sense, switch to camera mode, allow camera access, and position yourself so both hands fit in the picture.

Audio starts after your click because browsers restrict automatic sound playback. Camera access also needs permission and a secure page; MDN documents these requirements. The published instrument uses HTTPS.

A smaller note range makes the first melody easier

The default range is C3 to C6, spanning three octaves. Without note snapping, the pitch slides continuously between notes. The display shows the nearest note, frequency in hertz, and the difference in cents, where 100 cents equals one semitone.

For a first exercise, I recommend narrowing the range to C4–C5. The same hand travel then covers one octave, giving each interval more room on the screen. Find C, move slowly to the next note, and return without changing the volume hand. This separates learning the pitch movement from learning two controls at once.

Scale snapping helps you land on notes

The “Привязка к нотам” setting offers chromatic, major, minor, major pentatonic, minor pentatonic, and blues options. You can also choose the root note for the scales that use one. A pentatonic scale is a useful starting point for short improvisations because it limits the available note choices.

“Сила привязки” controls how strongly the calculated pitch is pulled toward the nearest allowed note. At 100%, the target pitch lands on scale notes; lower values retain more of the sliding movement. Portamento still smooths transitions between targets. Turn snapping off when you want to practise continuous pitch control and hear the spaces between notes.

One-hand mode removes the separate volume gesture

Enable “Одной рукой (кулак = тишина)” to control pitch with one hand while the instrument supplies a fixed playing level. Closing the pitch hand into a fist mutes the voice. The master volume remains available, so you can set a comfortable output level before playing.

The left-handed setting, “Левша (тон левой рукой),” swaps the control roles and mirrors the pitch layout. In the usual two-hand mode, the application assigns the hands by their screen positions when both are visible. Avoid crossing them during a phrase: keeping each hand in its own area makes the assignment predictable.

Four timbres share the same synthesizer

The sound choices are theremin, pure sine, flute, and cello, labelled “Терменвокс,” “Чистый синус,” “Флейта,” and “Виолончель.” These are synthesized timbres made with different harmonic mixtures. The flute and cello labels describe the intended character of the sound; they are not recordings of acoustic instruments.

I built the audio path with Tone.js: an oscillator passes through vibrato, a low-pass filter, hand-controlled gain, reverb, and master volume. Vibrato adds periodic modulation; reverb adds a sense of space. The portamento control sets how smoothly the pitch moves toward its next target. Start with modest effects so that you can hear your hand movements clearly.

Hand recognition runs locally in the browser

The camera controls use MediaPipe Gesture Recognizer, which returns hand landmarks and recognised gestures. My application uses a palm position calculated from the wrist and three knuckles, plus the closed-fist gesture. It translates those results into oscillator frequency and gain; the sound comes from synthesis rather than a music-generation model.

The application processes camera frames in the browser and does not implement a video-upload service. It requests video with microphone capture disabled. This is separate from ordinary page traffic: the deployed page also loads Google Analytics and an advertising script. Hiding the video preview only hides the image; camera tracking continues.

Tracking quality sets the practical limits

Give the camera a clear view of your hands and enough light to distinguish them. If the ordinary two-hand mode goes silent, check the pitch and volume hand indicators: both controls need to be recognised. Camera errors also distinguish permission denial, a missing camera, and a camera that another application has made unavailable.

The smoothing slider reduces jitter in the tracked position, while portamento smooths the sound’s pitch changes. Increasing either can make movement feel less immediate, so adjust them separately. The FPS indicator reports recognition activity, not a measured end-to-end audio latency. I have not published cross-device latency measurements or a browser compatibility matrix.

This project joins my other browser tools and music experiments, including SongGiftBot and my article about local music generation with ACE-Step. Here, the useful experiment is direct control: moving a hand changes a sound you are playing at that moment.

Try mouse mode first, then enable one-hand camera control with a one-octave range and strong pentatonic snapping. Once you can repeat a short phrase, add the volume hand and gradually reduce snapping. That gives you a manageable path from making the first sound to controlling a melody.