A new agentic paradigm for moving intelligence from the digital world into the physical world.
Physical intelligence is fundamentally about closing a causal loop with the world: understanding the current state, acting to change it, observing the real consequence, updating the underlying belief, and accumulating experience through continued interaction. Frontier agentic models have already begun to exhibit this capability in the digital domain, where they reason over goals, invoke tools, inspect execution results, and complete long-horizon tasks through feedback. What they still lack is a reliable way to extend that loop into the physical world.
OpenETA explores this transition from Digital Agent to Physical Agent as a new paradigm. Instead of reducing embodied intelligence to a single observation-to-action model, it places the Agent inside the world's causal chain so that perception, action, verification, and learning become one continuous process. Intelligence moves beyond processing digital information and begins to understand and change reality.
Architecture · Action Pipeline · Parallel Evaluation · Real-robot Deployment
- 2026-07-25 — Physical Hello World: OpenETA's first public release, extending the agentic loop from the digital domain into the physical world.
| Capability | OpenETA system boundary |
|---|---|
| Causal closed loop | Every world-changing action creates a fresh-observation obligation before reasoning, recovery, or completion can continue. |
| Replaceable Planner | Different LLMs and VLMs connect through an OpenAI-compatible interface with primary and fallback endpoints. |
| Composable physical capabilities | Perception, grasping, placement, motion, and future native policies compose through stable Tool and MCP interfaces. |
| Tool / Skill separation | Tools are host-owned atomic capabilities; Skills are readable and reviewable experience, never hidden execution. |
| Host-owned supervision | Schemas, provenance, safety gates, approval modes, and reviewers remain outside the Agent's authority. |
| Auditable Memory and Rollouts | Session working memory and immutable evidence are stored separately so success, failure, and uncertainty can be replayed. |
| Unified sim-to-real boundary | Simulators and robots share observation, action, result, and MCP lifecycle contracts. |
| Verifiable experience evolution | Experience begins as a session-local candidate and advances only after review, canary, and holdout validation. |
The core loop is:
observe -> reason -> act -> verify world change -> observe again -> adapt
Three boundaries constrain the system:
- Cognition / execution boundary: the Agent proposes intent, the Host decides whether to execute it, and the Simulator or Robot retains authority over environment truth.
- Action boundary: read-only capabilities may run in parallel, while world-mutating AtomActions execute one at a time and must satisfy the fresh-observation obligation before the next action.
- Evidence boundary: Tool completion, world-state change, and task success are distinct claims. Only trusted environment receipts, rewards, termination signals, or checkers can establish task completion.
| Agentic Planner | Tools | Simulator | Real World |
|---|---|---|---|
|
|
|
|
✅ Adapted. Unmarked items are not yet adapted. Simulator backends use isolated venvs; see Simulation Layer for installation, environment counts, and action spaces.
Requires Python 3.10+ and uv.
uv sync --extra dev
uv run openetaUse /provider in the TUI to configure an OpenAI-compatible Planner, then enter a task directly.
Common commands include /tools, /memory, /approvement, /run, /sessions, and /resume.
For a bounded single-task run:
uv run openeta --once "inspect the current scene" --max-turns 8| Entry point | Purpose |
|---|---|
uv run openeta |
Interactive Agent TUI for provider setup, Tool inspection, execution, and session recovery |
uv run openeta --once "..." |
Single-task execution with a bounded turn count |
uv run openeta-batch --manifest ... |
Parallel evaluation with isolated episodes and human recovery |
uv run openeta --command preflight/run/iterate/inspect ... |
Unattended experiments and experience iteration |
uv run openeta-replay EPISODE_DIR --output replay.mp4 |
Reconstruct a timeline and video from episode logs |
uv run python -m sim.mcp_server |
Simulator MCP, REST API, and Web Dashboard |
uv run openeta-robocasa-benchmark ... |
RoboCasa365 manifests, task suites, and benchmark utilities |
See uv run openeta --help, uv run openeta-batch --help, and each subcommand's --help for the
complete CLI reference.
Install one simulator backend, then start the MCP service and live Dashboard:
bash scripts/setup_envs.sh libero
uv run python -m sim.mcp_server
# Open http://localhost:8765/Validate a parallel-evaluation manifest without connecting to a model or MCP service:
uv run openeta-batch \
--manifest examples/parallel_libero_eval.json \
--validate-onlyThe repository includes LIBERO canary, train, holdout, and full-evaluation manifests. After connecting a Planner and the required MCP services, run isolated episodes in parallel:
uv run openeta-batch \
--manifest examples/parallel_libero_eval.json \
--concurrency 10 \
--approvement reviewed_autonomy \
--output outputs/parallel-libero.jsonUnattended experiments follow four stages: preflight -> run -> iterate -> inspect.
uv run openeta --command preflight \
--manifest examples/parallel_libero_eval.json
uv run openeta --command iterate \
--train-manifest examples/parallel_libero_six_train.json \
--validation-manifest examples/parallel_libero_six_holdout.json \
--experiment-id libero-skill-v1 \
--rounds 3 \
--approvement reviewed_autonomy \
--on-need-human fail
uv run openeta --command inspect \
--experiment-id libero-skill-v1Process completion does not imply task success. Final outcomes are established by trusted environment receipts, rewards, checkers, and retained rollout evidence.
| Document | Scope |
|---|---|
| Architecture | Boundaries across the Agent, Adapter, Simulator, and MCP layers |
| Agent command pipeline | Command schemas, Tool contracts, AtomActions, and safety gates |
| Simulation layer | Installation, environment registry, MCP, REST, and Dashboard |
| Parallel simulator evaluation | Concurrency, budgets, human recovery, and experience promotion |
| Rollout data contract | Session data layers and immutable evidence |
| Code policy runtime | Bounded code backends and sandbox boundaries |
| Calibration lifecycle | Calibration proposal, review, and publication |
| Real-robot deployment | Cameras, robot arms, real-environment MCP, and safety boundaries |
OpenETA remains under active research and development. New Planners, Tools, simulators, robots, and Skills should preserve the existing observation, command, and result contracts and include tests for side effects, failure semantics, and resource cleanup. Changes to schemas, Tool contracts, task success semantics, or safety gates must update the corresponding design documents.
uv sync --extra dev
uv run pytestOpenETA's initial simulator integrations were migrated and adapted from RLinf. The project builds on the open-source robotics ecosystems around LIBERO, MetaWorld, ManiSkill, RoboCasa, BEHAVIOR, MuJoCo, SAPIEN, and OmniGibson. Its perception and manipulation capabilities benefit from SAM 3, UniDepth, AnyGrasp, AnyPlace, Contact-GraspNet, GraspGenX, and Molmo.
Formal citation metadata will be added before the first public technical report. OpenETA is licensed under the Apache License 2.0. Third-party components and migrated code remain subject to their respective upstream licenses and retained source-file notices.