The fundamental question behind this project is whether a visual interface can represent the effects of key presses and the sustain pedal, then help young students determine the best time to change the pedal. Answering it begins with understanding what changes acoustically when the pedal is pressed.
What the pedal changes
When the sustain pedal is pressed, the dampers lift off the strings and allow them to vibrate freely. Sound from the strings that are struck transfers through the piano’s soundboard to other strings, which can then vibrate in response. This sympathetic resonance contributes to the warmth and depth of a pedaled piano.
The pedal also prolongs notes after the pianist releases their keys. Earlier notes continue to decay while new notes are played, increasing the overall acoustic energy. One way to describe this energy is RMS, or root mean square: a measure related to the effective amplitude of an audio signal.
Sympathetic resonance can also increase interharmonic energy—the energy at frequencies outside and between the piano’s quantized pitches. Imperfectly aligned vibrations and their interactions produce frequency content that does not exactly match individual piano notes. This makes interharmonic RMS a potentially useful signal for identifying pedaled sound.
The technical structure
Each version of the visualizer follows the same basic pipeline:
- Take a Fast Fourier Transform (FFT) of the incoming audio and analyze the frequencies being played.
- Create a circle for each detected frequency, with its size or decay rate controlled by the prototype’s current detection method.
- Let each circle contribute color to the background in proportion to its area.
- Continuously pull the background back toward white.
The detection method is the main part that changes between prototypes. It provides an observable value in the audio signal that estimates whether notes should continue bleeding into the background.
Why not use transcription AI?
Digital transcription models can turn audio into musical notation and may offer a route to detecting pedaling, but they introduce two problems. First, many are too computationally heavy for a standard tablet, the device many piano students would use. Second, transcription often depends on both past and future context, so it does not fit the requirement for immediate feedback. The visualizer therefore needs a lighter, purpose-built approach.
Prototype one: tracking RMS
My first prototypes relied on RMS. In testing, pedaled notes tended to remain around 30–40% of their original amplitude, so I stored the original amplitudes and used them to decide which notes should continue contributing to the background.
This method has important limitations. It is difficult to define a note’s original amplitude because piano sound undergoes acoustic beating: nearby frequencies can create periodic variations in volume. The program may misread these beats as new notes even when the pianist has not pressed another key. Plain RMS can also mistake a dense, rapid, unpedaled passage for pedaled sound because many notes overlap.
Prototype two: interharmonic RMS
The current approach monitors energy between piano notes. For example, if A is 440 Hz and B is 493.88 Hz, the program might inspect a band from roughly 450 to 483 Hz. I am testing whether this separates a sparse, heavily pedaled passage from a dense, unpedaled one.
The approach is promising because it is closer to a direct pedal detector than plain RMS, though it remains a deliberately lightweight approximation. The next step is to test whether this signal is reliable enough to support students in practice.
Terms
- RMS
- The square root of the mean of a set of squared values, used to describe the effective power of a signal.
- Interharmonic energy
- The power at frequencies that are not integer multiples of the fundamental frequencies in a system.
- FFT
- An algorithm that converts a signal from the time domain into a representation of its component frequencies.
- Acoustic beating
- A rhythmic rise and fall in amplitude caused by two sound waves with slightly different frequencies.