Goal

Design a gestural virtual musical instrument that is easy to learn and doesn't require users to memorize note positions in 3D space.

Challenge

• No haptic feedback makes precise mid-air note selection extremely difficult

• Traditional instruments map notes to absolute physical positions — a concept that breaks down in 3D space

• Gesture systems suffer from false positives/negatives, latency, and ambiguity

Approach

• Replaced absolute note targeting with melodic intervals (relative pitch jumps), reducing the required gesture set to a small, manageable vocabulary

• Split interaction across 2 hands: dominant hand handles melody via strike gestures + finger count; non-dominant hand handles pre-programmed chord selection

• Used Leap Motion Controller for markerless, high-precision finger tracking

• Combined 4 gesture signals (velocity + distance + duration + direction) to minimize input errors

My Role

Co-researcher contributing to interaction design, gesture design rationale, and design sessions across ideation, prototyping, and iteration phases.

Findings & Reflections

• Interval-based input reduced cognitive load — users focused on musical direction rather than spatial targeting

• Visual feedback for the current base note identified as critical for error recovery

• Half-step intervals and intervals beyond 5 steps remain unsupported — clear directions for future iteration

• No formal user study conducted at publication; usability testing planned as next step