UniTac Gives Robots a Unified Model for Understanding and Generating Touch

A robot fingertip pressing into an orange, a plush toy, or the smooth handle of a metal bowl generates a very different signal depending on which sensor is doing the touching — and until now, no single AI model could both make sense of that signal and generate a realistic one in return. A new paper from researchers at Zhejiang University, Yale University, the University of Oxford, MIT, Shanghai Jiao Tong University, and UNIX AI introduces UniTac, described as the first unified multimodal model built specifically for the tactile domain, capable of both understanding touch and generating it across different sensor types.

The paper, posted to arXiv on June 30, 2026 (arXiv:2606.31451) and authored by Jiahang Tu, Fengyu Yang, Chenyang Ma, Xihang Yu, Ziyao Zeng, Shaokai Wu, Hanbin Zhao, Zhi Tao, Chao Zhang, Hui Qian, and Alex Wong, frames its central idea around the physical mechanics of touch itself. "UniTac models the tactile process as a transition from non-contact to contact, capturing the physical interaction between sensors and objects through a dual-level representation that encodes both sensor and object attributes," the authors write in the paper's abstract. That dual-level framing — separating what belongs to the sensor from what belongs to the object being touched — runs through nearly every component of the system.

The authors argue that prior touch-AI work has been held back by two problems. First, tactile-language datasets have historically been small and siloed — they note that PHYSICLEAR, a widely used benchmark, contains only 482 touch videos, even though the broader research community has released open datasets totaling more than 400,000 video clips and 1.6 million frames that are rarely combined for joint training. Second, sensors like GelSight, DIGIT, and Duragel each encode touch differently through their own lighting, gel deformation, and camera configurations, creating domain gaps that make models trained on one sensor fail on another. "Without object-level information, non-contact tactile data lack semantic meaning; without sensor-level information, generative models are unaware of the sensor configuration on which tactile signals should be synthesized," the authors explain, arguing that a unified model needs to track both levels simultaneously.

Architecturally, UniTac combines a pretrained touch encoder (built on the AnyTouch ViT-B/16 backbone), a Qwen2.5-VL multimodal language model backbone in 3B and 7B variants, a sensor-aware diffusion transformer projector, and a touch decoder built on the SANA diffusion architecture for images and Wan v2.2 for video. Understanding is trained through what the authors call Dual-Level Mixture Comprehension, which pairs an object-property description task (describing roughness, hardness, and texture) with a sensor-identification task (naming which physical sensor captured the signal). Generation follows a two-stage process — reconstruction, then sensor-aware alignment — paired with a Sensor-Prior Sampling Strategy that replaces the standard unconditional branch in classifier-free guidance with a sensor-conditioned prior representing the non-contact state, explicitly modeling the moment before contact occurs.

The results, evaluated on the PHYSICLEAR-Test benchmark and across four tactile sensors (Digit, GelSight, GelSight Mini, and Duragel), show UniTac-7B reaching an average understanding score of 66.51, ahead of GPT-4o (31.65), Qwen2.5-VL-7B (25.01), and the previous best tactile-specific model, Octopi-7B (57.31). On reasoning-heavy tasks — property comparison, property-object matching, and property superlative selection — UniTac-7B outperformed Octopi-7B by as much as 42.39 points on one metric. For generation, UniTac achieved an average 0.836 SSIM and 19.93 PSNR across the four sensors, ahead of comparable unified models and competitive with generation-only specialists. The team also deployed UniTac on a real robotic platform, where it compared two visually similar fabrics and correctly identified the smoother one as more suitable for contact with an infant's skin — a distinction invisible to camera-only vision. In a language-guided grasping trial, a vision-tactile-language-action system using real tactile input succeeded in 19 of 20 rollouts, compared to just 4 of 20 for a vision-only baseline.

Why It Matters

Vision and language have dominated the unified-multimodal-model wave of the past two years, but robots that manipulate the physical world need more than sight — they need to feel surface hardness, slip, and deformation to grasp fragile or visually ambiguous objects safely. UniTac's approach of treating touch as a two-stage physical process, rather than just another image modality, addresses a gap that has kept tactile AI fragmented across incompatible sensor hardware. By showing that a generative model can synthesize believable tactile data for a sensor it wasn't trained on — improving cross-sensor grasp classification accuracy from 50% to 99.37% in one experiment — the paper also points to a practical fix for one of embodied AI's quieter bottlenecks: the cost and slowness of collecting new tactile hardware data every time a lab adopts a new sensor.

What to Watch

Watch for whether UniTac's synthetic tactile data holds up as a substitute for real-world data collection at larger scale, and whether other labs adopt its dual-level, non-contact-to-contact framing as a standard for future touch-language-action models. The authors' own ablations show that removing the dual-level comprehension objective causes the sharpest performance collapse of any component tested, suggesting the sensor/object split is doing much of the model's real work. Also worth tracking: how quickly the approach migrates from GelSight-style optical tactile sensors, which produce image-like output well suited to this architecture, to other sensing modalities such as capacitive or barometric skins that do not.