Gaze + Pinch User Interface

Designing the Interaction Model for Apple Vision Pro

COMPANY

Apple

ROLE

UX Research Lead | Technical Designer

EXPERTISE

UX Research | Experience Design

YEAR

2020

Overview

Apple Vision Pro introduced a new interaction paradigm without controllers.

The goal was to create an input system that is natural, precise, and scalable across all spatial interfaces.

I led prototyping and user research to define and validate the gaze + pinch interaction model, now foundational to visionOS.

My Role

I led prototyping and user research to define and validate the gaze + pinch interaction model, now used across visionOS.

As the UX Research Lead | Technical Designer, I

  • Led end-to-end prototyping of interaction systems  

  • Designed and implemented real-time interaction models  

  • Conducted large-scale user studies and behavioral analysis  

  • Collaborated across design, engineering, and hardware teams  

  • Presented prototypes to executive leadership including the CEO  

The Challenge

Traditional input methods do not translate to spatial computing. Carrying a 3D controller is cumbersome and using it is limiting, less intuitive and has a learning curve. We needed a system that:

  • Requires no learning curve  

  • Works across UI, text input, and object manipulation  

  • Minimizes fatigue during extended use  

  • Maintains precision without physical feedback  

Approach

The system was designed to feel invisible. Users should focus on content, not controls. We explored gaze as a primary targeting mechanism and micro-gestures for confirmation. The system was designed to:

  • separate intent (gaze) from action (pinch)

  • reduce physical effort

  • provide continuous visual feedback

Use Research

  • 18 internal studies, 400+ participants  

  • Mixed-method research: quantitative performance + qualitative feedback  

Key findings:

  • Eye gaze is fast but requires calibration and stabilization  

  • Micro-gestures reduce fatigue compared to large movements  

  • Visual feedback is critical for confidence and accuracy

These insights directly informed interaction thresholds, feedback design, and system behavior.

Impact

  • Shipped as the core interaction model of Apple Vision Pro

  • Enabled hands-free navigation, text entry, and object interaction  

  • Influenced hardware decisions (sensor placement, FOV)

  • Adopted across system-level and third-party applications

Reflection

Designing for spatial computing requires rethinking interaction from first principles.

The success of gaze + pinch comes from balancing human behavior, technical constraints, and system scalability, turning a complex interaction problem into a model that feels effortless.

Spatial interaction succeeds when complexity disappears. The goal is not to teach users a system, but to make it feel obvious.