浙江大学 April Lab

TARSphere A Robot-Oriented Full-Stack Framework for Long-Horizon Embodied Agents in Evolving Physical and Social Environments

1Zhejiang University 2Fudan University
arXivPaper GitHubCode Project

Abstract

Robots must act under physical and social constraints while adapting to changes in their environment and human needs. Existing embodied-agent systems often focus on specified goals or abstract execution into symbolic actions, limiting evaluation in an evolving world.

We introduce TARSphere, a robot-oriented full-stack framework for physically and socially grounded long-horizon embodied agents. TARSphere-Infra runs robot policies in a physics simulator with embodied NPCs and constrained dialogue. An event-driven finite-state machine (FSM) uses completed actions and delivered information to update task states and evaluate outcomes. TARSphere-Harness, its embodied brain harness, lets a frozen multimodal large language model (MLLM) act through multi-level executable skills, using observations, hierarchical memory, and execution feedback to revise its decisions.

On this foundation, agent-in-the-loop curation aligns task semantics, causal dependencies, and scene geometry to construct TARSphere-Bench: 360 episodes across 30 indoor scenes. Five paired conditions assess perceptual robustness, disruption adaptation, task update, human coordination, and intent understanding for navigate-and-report and fetch-and-deliver tasks. Our evaluation of 8 MLLMs shows that general-purpose models can outperform embodied models, while task updates and human interaction remain difficult. Frequent collisions and falls further motivate evaluating execution safety alongside task success.

Physically and socially grounded.
In a world that keeps evolving.

A full-stack framework for long-horizon embodied agents in changing physical and social environments.

Long-horizon tasks connect manipulation, social interaction, evolving task states, and physical contact and balance.
Long-horizon tasks connect manipulation, social interaction, evolving task states, and physical contact and balance.

See how TARSphere brings embodied people, policy-driven grasping, world updates, and physical constraints into the same environment.

Demonstration

One world. A continuous loop.

The mechanisms behind the demonstrations: a physical environment, an agent harness, and a paired benchmark.

TARSphere-Infra

A world that responds to actions.

TARSphere-Infra couples robot policies, embodied people, constrained dialogue, and an event-driven finite-state machine.
TARSphere-Infra couples robot policies, embodied people, constrained dialogue, and an event-driven finite-state machine.

4K · Narration & embedded subtitles · Three people, one Courtyard Home. An authored illustration of concurrent NPC activity, changing needs, and occupancy-planned paths. The grasp and delivery chapters use a separate recording with measured poses and stage credits.

Read the state-machine walkthrough
  1. Ground a request in the scene. Harold asks the robot to inspect a refrigerator. The robot navigates through connected rooms using the authored observation stance.
  2. Let people act independently. Harold walks to the meeting point while Owen moves between the dining area and passage, and Marcus visits the bedroom. Each has a live need, action, and state. Owen and Marcus’s concurrent activities are authored for this illustration; Harold follows the task-update episode.
  3. Check event guards. Target observation, completed NPC arrival, and the 2.5 m hearing-range condition must all hold.
  4. Deliver a changed need. Harold now requests the bedroom cover pattern. Only the delivered, authorized revision changes the active target and answer requirements.
  5. Follow the causal DAG. The robot observes the new target while Harold returns to the lounge; inspection, finding Harold, and reporting satisfy their own dependencies. Event nodes earn no score.
  6. Check physical evidence. A separate recorded Base delivery in the same home shows the full grasp at 1×, confirmed object retention, transport, and the accepted offer. Recorded credits are retained, including 0.9 for task understanding.
  7. Continue the loop. Public observations, dialogue, and execution feedback reach the agent. The evaluator privately checks requirements active at each evidence step.

TARSphere-Harness

From multimodal reasoning to physical execution.

TARSphere-Harness connects frozen models to observations, hierarchical memory, multi-level skills, and measured execution feedback.
TARSphere-Harness connects frozen models to observations, hierarchical memory, multi-level skills, and measured execution feedback.

TARSphere-Bench

Same task. Changing conditions. Paired episodes reveal how agents adapt.

360
episodes
30
household + commercial scenes
2 × 6
task families × conditions
8
evaluated models
Two task families share a Base condition and five paired capability conditions, with 60 episodes per condition.
Two task families share a Base condition and five paired capability conditions, with 60 episodes per condition.

See the task unfold.

Two tasks, six conditions, from the robot’s point of view.

Recorded demonstration

View the methodOpen full-resolution recording

Two scenes per condition. Expand a recording for task details and its connection to the method.

Benchmark curation

Aligning task semantics, causal dependencies, and scene geometry.

Agent-in-the-loop curation aligns task semantics, causal dependencies, and scene geometry, with review and execution checks.
Agent-in-the-loop curation aligns task semantics, causal dependencies, and scene geometry, with review and execution checks.

Explore paired episodes.

Explore all 360 episodes

Same scene, same scale. Compare both tasks across six conditions, then open an episode for its causal DAG, event guards and task definition. Details can switch across all 360 episodes. Paths are occupancy-planned references, not recorded executions.

To targetReturn / deliver

Causal DAG

Solid · outcome   Dashed · event

The initial request

Events & dialogue conditions

Authored episodeEpisode JSONCompare six conditions

Reference routes are planned on static occupancy, not recorded model trajectories. The map does not simulate moving people, perception changes or skill outages. Every DAG edge and live guard comes from the authored definition. Events gate dependent outcomes without adding score.

Comparison with existing benchmarks.

Physical execution, social interaction, and an evolving world in one benchmark.

Swipe to explore the full table. Select it to open the original.

Original submitted-paper comparison of BEHAVIOR-1K, GRBench, PARTNR, EmbodiedBench, HumanCLAW-Bench and TARSphere-Bench. Original supported, restricted-support and unsupported symbols are preserved.

Main benchmark results.

Eight models, evaluated across two task families and six conditions.

Swipe to explore the full table. Select it to open the original.

Original submitted-paper main results: eight models, two task families and six conditions, with progress, success, efficiency, inference cost and safety.

Progress / Success (%). Safety: collision / fall episode incidence (%). Original table from the submitted paper.

Harness ablations.

Gemini-3.8-Flash · balanced subset.

Swipe to explore the full table. Select it to open the original.

Original submitted-paper harness ablations: observation, memory and skill removals, context history and action horizon settings, and the full TARSphere-Harness.

Qualitative comparison.

Four models. The same task. Different paths through the world.

Four models on the same task: egocentric observations, reasoning and event-triggered milestones show successful reporting, failed recovery, wrong-person interaction and an unsupported completion claim.
Four models on the same task: egocentric observations, reasoning and event-triggered milestones show successful reporting, failed recovery, wrong-person interaction and an unsupported completion claim.

Build TARSphere with us.

Help us expand what robots can do and how people move, interact, and collaborate in simulation. We welcome research collaborations, new capabilities, and contributions of all sizes.

More versatile actions

Richer manipulation, flexible skills, and more ways to interact with the world.

More lifelike people

Natural NPC motion, expressive behavior, and believable social interactions.

More robot embodiments

Broader support for robot models, controllers, and physical capabilities.

Have an idea, a new scene, or something you would like to build?

12532055@zju.edu.cn

Citation

If this framework supports your research, please cite our paper.

BibTeX citation
@misc{xu2026tarsphere,
  title = {TARSphere: A Robot-Oriented Full-Stack Framework for Long-Horizon Embodied Agents in Evolving Physical and Social Environments},
  author = {Xu, Yicheng and Xu, Zhikai and Chen, Ziyuan and Dong, Yuhang and Zhang, Tao and Xue, Zhucun and Liu, Yong and Zhang, Jiangning},
  year = {2026},
  eprint = {2610.30043},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url = {https://arxiv.org/abs/2610.30043}
}