MATE: Mask-Guided Adaptive Object Template Estimation and Tactile-Confirmed Execution for Human-to-Robot Handover

Abstract

Geometry from masks, execution confirmed by touch.

Human-to-robot handover is challenging when transparent surfaces and hand occlusion produce incomplete or unreliable RGB-D geometry, while vision alone cannot confirm contact during execution. We present MATE, a Mask-guided Adaptive object template estimation framework with Tactile-confirmed Execution.

To recover reliable object geometry under transparency and occlusion, MATE fits an instance-specific object template to dual-view partial point clouds and refines its scale using cross-view mask agreement. A magnet-based tactile fingertip regulates gripper closure and release. Experiments on CORSMAL containers and household objects achieve average success rates of 89.8% and 91.7%, respectively, demonstrating the complementary benefits of geometric adaptation and tactile sensing.

Key Idea

Adapt the shape. Confirm the contact.

Reliable handover requires both robust geometric perception and contact-aware execution.

Before Contact

Geometric Uncertainty

Human-hand occlusion
Missing or distorted depth
Transparent surfaces
Incomplete object extent
Grasp-target uncertainty

After Contact

Contact Uncertainty

Fingertip–object contact
Secure gripper closure
Stable support contact

MATE process: perception, grasp, contact, and placement

01

Mask-Guided Template Adaptation

An instance-specific template is fitted to partial point clouds and scale-refined by cross-view mask agreement to recover geometry under transparency and occlusion.

02

Hand-Aware Target Planning

The refined template and estimated hand state define a grasp target with hand clearance that is continuously updated as the object moves.

03

Tactile-Confirmed Execution

A magnet-based tactile fingertip regulates closure at object contact and releases only after detecting support contact during placement.

Method

From partial RGB-D observations to contact-aware execution.

Method figure Perception pipeline of MATE Expected asset: static/images/perception_pipeline.png

Dual-view masks and depth form a merged partial point cloud. A SAM3D-generated template is fitted and scale-refined by multi-view mask IoU, then fused with the estimated hand state for grasp-target planning.

01

Dual-View Segmentation and 3D Lifting

A hand-object detector localizes the object in each RGB view, FastSAM extracts its mask, and valid masked depth pixels are lifted to 3D, filtered, and merged across both views.

02

Instance-Specific Template Fitting

For each new object, SAM3D generates a template from a masked RGB observation. Its initial scale is obtained by matching oriented bounding-box extents, followed by ICP registration to the merged point cloud.

03

Mask-Guided Scale Refinement

Candidate scales sampled around the initial estimate are projected into both camera views. The scale with the highest combined IoU between rendered silhouettes and observed masks defines the refined template.

04

Hand-Aware Grasp-Target Planning

MediaPipe estimates landmarks and handedness in both views, and the less-occluded hand observation is lifted to 3D. A target on the refined template is chosen for hand clearance and proximity to the object center, then updated online. During temporary detection failure, it is inferred from the tracked hand and the previous hand-to-target offset.

Tactile Execution

Tactile-confirmed grasping and placement.

The magnet-based fingertip monitors the norm n_t of its baseline-subtracted magnetic response. At the grasp target, the gripper closes until n_t exceeds the grasp threshold τ_g, records the contact position, and adds a small closing motion to secure the object. During descent for placement, MATE monitors Δn_t = |n_t − n_r| and stops to release when it exceeds the placement-contact threshold τ_p.

Hardware image Tactile fingertip static/images/tactile_fingertip.png

Magnet-based tactile fingertip.

Contact visualization Tactile response static/images/tactile_signal.png

Tactile response for grasp and support contact.

System / Hardware

Dual-view perception with close-range touch.

01 UR5 robotic arm
02 Robotiq 2F-85 gripper
03 Custom 3D-printed TPU tactile fingertip
04 2× Intel RealSense D435i RGB-D cameras

System image Real-world handover setup Expected asset: static/images/hardware_setup.png

The dual RGB-D cameras provide complementary views that reduce giver-hand occlusion. The TPU fingertip uses an eFlesh-style internal microstructure with micro-magnets above a magnetometer array to measure contact-induced magnetic-field changes.

Results

89.8% on CORSMAL containers and 91.7% on household objects.

Success requires stable upright placement, with bean spillage counted as a failure. Each CORSMAL condition includes 18 trials (2 participants × 3 grasp types × 3 locations); each household object includes 6 trials (2 participants × 3 locations).

CORSMAL Containers

C1 red cup · C2 small cup · C3 beer cup · C4 wine glass

C3 and C4 were evaluated both filled and as empty transparent containers.

Household Objects

O1 soap · O2 beaker · O3 candy-filled plastic bag · O4 food container

Evaluation figure Evaluation objects Expected asset: static/images/objects_for_handovers.png

Human-to-robot handover success rate (%)
Method	CORSMAL Containers							Household Objects
Method	C1	C2	C3	C4	C3 Empty	C4 Empty	CORSMAL Avg.	O1	O2	O3	O4	Household Avg.
w/o Tactile	66.7	88.9	88.9	61.1	55.6	72.2	72.2	100.0	66.7	100.0	33.3	75.0
w/o Shape Fit.	94.4	66.7	100.0	72.2	50.0	55.6	73.2	50.0	33.3	66.7	0.0	37.5
w/o Scale Ref.	83.3	94.4	94.4	83.3	77.8	77.8	85.2	100.0	100.0	100.0	66.7	91.7
MATE (Ours)	88.9	94.4	100.0	83.3	83.3	88.9	89.8	100.0	83.3	100.0	83.3	91.7

Without tactile feedback

The gripper could keep closing after object contact or release before stable table contact, causing excessive closure or object drops.

Without shape fitting

Missing depth on transparent objects left the raw merged cloud incomplete, sometimes eliminating valid grasp targets and reducing the household-object average to 37.5%.

Without scale refinement

Over-scaled templates placed targets outside the physical object boundary. The full system maintained at least 83.3% success in every condition, the strongest worst-case performance.

Conclusion

Complementary geometry and contact sensing for robust handover.

MATE addresses geometric uncertainty and contact-state uncertainty during human-to-robot handover. Across diverse object types and depth conditions, the complete system achieved at least 83.3% success, demonstrating robust grasping and placement under incomplete RGB-D observations.

Real-Robot Demos

Handover across geometry, transparency, and deformability.

Real-robot demonstrations on the CORSMAL containers and household-object categories used in the experiments.

Demo videoFood containerstatic/videos/food_container.mp4

Food container

Demo videoBeakerstatic/videos/beaker.mp4

Beaker

Demo videoCandy-filled plastic bagstatic/videos/candy-filled_plastic_bag.mp4

Candy-filled plastic bag

Demo videoSoapstatic/videos/soap.mp4