Abstract
TriMI is a tri-modal manipulation interface for direct demonstration, joint-level haptic teleoperation, and robotic execution. Its four-finger design provides 13 degrees of freedom: four in the thumb and three in each of the remaining fingers. With a bill of materials cost of approximately USD 800, excluding the tracking device and wrist camera, TriMI enables operators to collect demonstrations through direct physical interaction, teleoperate robotic hands with joint-level haptic feedback, and execute learned manipulation policies. We evaluated a diffusion policy trained on the collected direct demonstrations, achieving 18 successful trials out of 20 for cube pick-and-place (90%) and 12 out of 16 for cap unscrewing (75%). A full open-source release is coming soon.
*Tracking device and wrist camera excluded.
Three Modes, One Interface
Direct Demonstration
Manipulate objects by hand through the interface to record demonstrations.
02Joint-Level Haptic Teleoperation
Control the robot joint by joint and receive haptic feedback.
03Robotic Execution
Execute manipulation on the robot using the same interface hardware.
Hardware
Scroll through the TRIMI interface, or drag to rotate.
Assembly preview
Dexterity
Six tasks showcasing TriMI’s dexterity, from in-hand manipulation to tool use.
Cube rotation
Reorienting a cube within the hand.
Cutting and peeling tape
Coordinating contact to manipulate tape.
Multiple grasps
Grasping objects with different hand configurations.
Drill bit insertion
Positioning and inserting a drill bit.
Cup retention
Maintaining a grasp on a cup.
Screw sorting
Picking and sorting small screws.
Direct Demonstration
An operator manipulates objects by hand through TriMI to collect demonstrations. The two tasks below provide the demonstration data used to train the diffusion policy evaluated in robotic execution.
Cube pick-and-place
Direct demonstration
Cap unscrewing
Direct demonstration
Robotic Execution
TriMI carries the manipulation workflow from human demonstration to autonomous robotic execution.
TPU Hand Showcase
Explore the motion of the TPU hand before viewing the autonomous policy rollouts.
TPU hand in motion
A closer look at the hand’s movement.
Autonomous Policy Rollout: Cube Pick-and-Place
A diffusion policy trained on the collected direct demonstrations performs cube pick-and-place. This video shows an example rollout; the success rate summarizes 20 evaluation trials.
Cube pick-and-place
Autonomous diffusion policy
18 / 20 successful trials · 90%
Autonomous Policy Rollout: Cap Unscrewing
A diffusion policy trained on the collected direct demonstrations unscrews a bottle cap. This video shows an example rollout; the success rate summarizes 16 evaluation trials.
Cap unscrewing
Autonomous diffusion policy
12 / 16 successful trials · 75%
Wearability
TriMI is worn on the operator’s hand, allowing direct interaction with objects through the interface.
Wearing TriMI
The interface in use on the operator’s hand.
Teleoperation
One human interface for controlling robotic hands.
Teleoperation Across Robotic Hands
TriMI maps the operator’s hand motion to a robotic hand, providing an interface for teleoperation across different hand designs.
Cross-hand teleoperation
Transferring the operator’s motion to a robotic hand.
Joint-Level Force Feedback with the Midas Hand
TriMI teleoperates the Midas hand with joint-level force feedback returned to the operator, connecting commanded motion with haptic feedback.
Midas hand teleoperation
Joint-level commands to the robot; force feedback to the operator.