Aligning sheet music and audio

Designing to collect alignment data linking sheet music to performance audio faster

Timeline 14 weeks

TeamProduct designer,
Product lead,
User researcher,
Software developer

Tools Figma

DisciplineData-collection
Music
Generative AI

Context How to collect data faster?
As part of my Bachelor in HCI Thesis project, my team and I were challenged to help our client design a faster way to collect accurate alignment data linking sheet music to performance audio.

Our client Chris Donahue is researching machine learning methods for generative music. His goal is to enable a broader audience—inclusive of non-musicians—to harness generative music AI.

ChallengesCurrently, there does not exist a tool to align sheet music with audio performances.
Nuances like expressive timing and repeat signs make alignment challenging and time-consuming even for trained musicians.

Research
Understanding what our client is currently working with

(20 hours to align 13 pieces)

Project goalHow might we design an intuitive tool to help users to align sheet music and a corresponding recording faster so researchers can generate an accurate dataset to train models for generative music?

Solution

Design process
Labeling measures

Low fidelity prototype to test adjusting measure boxes

Users all found that dragging to label measures was intuitive.
We tested with three types of musicians
Music educators
Beginner musicians
Advanced musicians

An advanced musician pointed out that each measure is based on the number of beats rather than timestamps. We brought the feedback to our client who confirmed that beat timestamps were not necessary to align sheet music and audio.

From a research point of view, it's much more useful to align noteheads rather than beats.

Taking a step back to understand the user flow

Finalizing bounding boxes to label measures

Enabling annotators to add additional boxes as needed

Color coating to indicate algorithm confidence

Final measure segmentation design

Design process
Labeling repeats

How does a repeat work?

Originally, we designed finding repeats as tags on the measure boxes

However, after more user testing, we learned that users did not have a mental model on how to connect the repeat labels.

So, we zoomed out.
We designed a jump annotation workflow with connection points. Based on clicking the starting and ending measure of a jump.

Preview the logic-order measures based on jump labels

Because adjusting measure boxes and labeling jumps were two core steps in alignment, it led us to design a breadcrumb with all the steps.

Final solution
Putting all the pieces together

It takes just seconds to align 1 pieceOriginally, it took 20 hours to align 13 pieces, then with our new alignment interface, it took 30 minutes to align 13 pieces, and with improved measure detection, it just takes seconds to align 1 piece.

LAST UPDATED 2025-01-03

CHENSHERRY12 (AT) GMAIL.COM