Come Here – a Bare Hand Selection and Manipulation Technique
Goal
Design a Natural User Interface (NUI) for seamless bare-hand selection and manipulation of 3D virtual objects — covering rotation, translation, and scaling — in a large tracked VR space, without requiring users to hold devices or wear cumbersome tracking equipment.
Challenge
Prior approaches required users to hold "magic wand" devices or wear trackable gloves — unnatural and cumbersome
Bare-hand interaction lacks haptic feedback, making precise 3D manipulation difficult
Designing gestures that feel naturally related to their task while minimizing false positives and supporting both fine-grained control and large, quick movements
Application Context
The system targets 3D design tools and virtual showroom experiences. A key motivating scenario: a virtual store where users walk freely through the space, pick up products from shelves, rotate and examine them, and place them in a cart — all through natural gaze and hand gestures with no held controllers.
Approach
Implemented in the VisCube at Virginia Tech — a 40×48 ft tracked space — using a Qualisys motion tracking system and a Leap Motion mounted to the HMD for finger tracking
Hands rendered as avatars in the virtual environment for visual feedback
4 core interaction techniques:
Interaction | Gesture |
|---|---|
Selection | Index finger point — ray cast from eyes through fingertip |
Rotation | Index finger circle (X, Y, or Z axis in camera space) |
Translation | Push/pull (Z), swipe (X), up/down (Y) |
Scaling | Two-handed fists moved closer or further apart |
Combined multiple gesture properties (finger velocity, palm velocity, distance, duration) to determine magnitude of change — enabling both precise and large movements from the same gesture
Selection lock of ≥2 seconds after gesture recognition to prevent false positive re-selection
Users navigate the VE by physically walking through the tracked space
My Role
Lead designer and implementer of the interaction system and gesture recognition pipeline.
Findings & Reflections
Multi-property gesture detection reduced false positives while keeping thresholds accessible to avoid false negatives
Ray-cast selection provided a natural first-person pointing metaphor consistent with real-world object targeting behavior
Two-handed scaling introduced a second hand only when needed, keeping single-hand manipulation uncluttered
Hand-based selection was prioritized over gaze-based selection despite slightly lower accuracy — in keeping with the goal of natural, device-free interaction
No formal user study conducted — prototype designed as a design exploration and demonstration in a high-fidelity VR facility
