VERA policy provider¶

VERA (Video-to-Embodied Robot Action, MIT/CSAIL) is a two-stage, closed-loop video-to-action policy:

Video planner (DFoT / WAN) — a diffusion model that "dreams" the next frames from the current observation (+ optional text). Embodiment-agnostic.
Jacobian IDM — translates the dream into robot actions via a frozen visual backbone (VGGT/DINO) + a flow→action head. Embodiment-specific, data-efficient, swappable without retraining the planner.

One video planner, many IDMs — the route to zero-shot, cross-embodiment control.

The strands-robots vera provider is a thin, import-light WebSocket client + managed GPU server, mirroring the cosmos3 service pattern. The host venv never installs VERA's heavy/conflicting stack (PyTorch 2.6 / CUDA, VGGT, DFoT) — that lives in the strands-vera-server container.

host venv (numpy>=2)                 vera-server container (torch 2.6 / CUDA)
  VeraPolicy ─ VeraWebsocketClient ─ws─▶ vera.server.start_vera_server
  (no vera install)                      DFoT/WAN planner + Jacobian IDM + /ckpts

Quick start¶

from strands_robots.policies import create_policy

# Attach to a running server (see "Server" below) ...
policy = create_policy("vera", embodiment="mimicgen", auto_launch_server=False)
chunk = policy.get_actions_sync(observation, "stack the red block on the green block")

# ... or let the provider manage the container for you:
policy = create_policy(
    "vera", embodiment="mimicgen",
    server_mode="docker", ckpt_root="/abs/path/vera-ckpts",
)

The MimicGen → Panda path drives a real 7-DoF arm: the WAN planner + Jacobian IDM emit 6-DoF end-effector deltas, and the provider's IK bridge solves them onto the Panda's joints (auto-discovering the end-effector frame — no manual wiring). See examples/vera_mimicgen_panda/.

VERA MimicGen Panda rollout — VERA MimicGen policy on a Franka Panda — WAN dream + AllTracker + Jacobian IDM → eef-deltas → VERA IK bridge → joint targets → MuJoCo.

Embodiments¶

From VERA's adapter_factory._EMBODIMENTS:

Embodiment	action_space	dims	views	control	ports (policy/viz)	checkpoints
pusht	`velocity` (planar)	2, no gripper	`image`	10 Hz	8820 / 8821	experimental — IDM du path not wired end-to-end upstream
mimicgen	`eef_delta`	7 (6-DoF + grip)	`agentview_image`, `robot0_eye_in_hand_image`	20 Hz	8800 / 8801	✅ Wave-1 (+WAN base)
allegro	`joint_position`	16	12 cameras	15 Hz	8802 / 8803	🔜 Wave-2 (code only)
droid	`cartesian_delta`	7	`varied_1`,`varied_2`,`hand`	15 Hz	8804 / 8805	🔜 Wave-2 (code only)

Today, end-to-end: mimicgen (WAN planner; needs the frozen WAN base + a motion tracker) is the working, faithful embodiment — it exercises the whole eef-delta → IK path onto a real arm. pusht's server runs, but its IDM du action path is not wired end-to-end upstream (VERA's own configurations/dataset/pusht.yaml documents this gap), so it validates the provider → server → action plumbing rather than producing a solving rollout — treat it as experimental. allegro/droid are code-present but checkpoint-absent upstream (Wave 2).

The "generalist" claim, accurately¶

VERA's architecture is cross-embodiment: one embodiment-agnostic video planner + one cheap IDM per robot (frozen backbone, head trained from self-play). It is not a single checkpoint that drives every robot today — a robot is drivable iff (its action_space matches a served embodiment) and (a checkpoint exists) and (IK/validation is done for that arm). For eef_delta/cartesian_delta arms (mimicgen/droid), the provider includes an IK bridge that maps the 6-DoF end-effector deltas to joint targets and auto-discovers the end-effector frame from the compiled MuJoCo model — so any kinematically-compatible 6/7-DoF arm can be driven once a matching IDM exists.

Checkpoints¶

hf download sizhe-lester-li/VERA --local-dir ./vera-ckpts   # ~42 GB full; ~4 GB is Wave-1
export VERA_CKPT_ROOT=$PWD/vera-ckpts

MimicGen additionally needs the frozen WAN 2.1 base (text-enc + VAE + CLIP). Its IDM uses the AllTracker point tracker, which the container bundles (cloned at build time; weights auto-download). The WAN base:

hf download Wan-AI/Wan2.1-T2V-1.3B --local-dir ./Wan2.1-T2V-1.3B

The provider never auto-downloads — point it at pre-downloaded roots.

Server¶

The server holds the GPU and the two-stage model. Run it as a container:

docker build -f strands_robots/policies/vera/docker/Dockerfile -t strands-vera-server:latest .

# MimicGen (serves ws on :8800; needs the WAN base + offline resolver)
docker run --rm --gpus all --ipc=host -p 8800:8800 \
    -v "$VERA_CKPT_ROOT":/ckpts:ro -v "$PWD/Wan2.1-T2V-1.3B":/wan:ro \
    -e VERA_EMBODIMENT=mimicgen -e USE_OFFLINE_RESOLVE=1 \
    strands-vera-server:latest

The container entrypoint maps the single mounted /ckpts root onto VERA's per-embodiment checkpoint env vars; USE_OFFLINE_RESOLVE=1 resolves MimicGen's wandb-run-id IDM to the locally-mounted checkpoint (via provenance.json) so the server boots with no network. See policies/vera/docker/.

server_mode="docker" lets the provider build/run/stop the container itself; server_mode="subprocess" launches a local python -m vera.server... when VERA is installed in the same env.

Configuration¶

VeraConfig maps 1:1 to VERA's server flags and is env-overridable (deploy/CI wins over code defaults):

kwarg	env var	maps to
`embodiment`	—	`--embodiment`
`server_port` / `vis_port`	`VERA_SERVER_PORT` / `VERA_VIS_PORT`	`--port` / `--vis-port`
`algo_config`	`VERA_ALGO_CONFIG`	`--algo-config` (swap to the omni planner)
`dynamics_run_id`	`VERA_DYNAMICS_RUN_ID`	`--dynamics-run-id`
`text_prompt`	`VERA_TEXT_PROMPT`	`--text`
`ckpt_root`	`VERA_CKPT_ROOT`	container `/ckpts` mount
`sample_steps`	`VERA_SAMPLE_STEPS`	`--sample-steps`
`tracker_backend`	`VERA_TRACKER_BACKEND`	IDM tracker
`motion_plan_scale`	`VERA_MOTION_PLAN_SCALE`	live `configure`
`server_mode`	`VERA_SERVER_MODE`	`subprocess` \| `docker`

Both ports take the shared TCP-port domain every port-dialing provider applies: an int in [1, 65535], or None for the per-embodiment default. The value is checked once, on the config, because three consumers read it — the client dials it, the runner launches the server on it, and VeraConfig.server_uri reports it — so a value outside the range is not merely refused late but resolved differently by each of them. vis_port = 0 is the one exception and disables the live viewer (--vis-port is omitted). The VERA_*_PORT overrides go through the same check.

motion_plan_scale takes the same domain as the two IK scales below: a positive finite number, or None to leave the server's own scale alone. 0 is not the opt-out — it scales the plan to nothing — so None is the off switch and 0 is refused. It is checked on the config, not where it is used, because where it is used cannot refuse it: _ensure_started applies it after the server handshake with a best-effort configure call whose failure is logged at INFO and does not stop the rollout, so a value float() cannot convert is neither applied nor reported. VERA_MOTION_PLAN_SCALE goes through the same check; an unparsable spelling still falls back to None, as it does for the ports.

IK conversion knobs¶

Three keyword-only numbers shape every joint target the eef-delta path produces, and each is checked where it is supplied because each is applied rather than forwarded — nothing downstream can refuse them usefully:

kwarg	surface	domain
`rotation_dim`	`set_ik_target(...)`, `decode_vera_delta_chunk_to_targets(...)`	`3` (axis-angle) or `6` (rot6d) — the encodings the decoder implements (`None` on the setter keeps the embodiment's convention)
`translation_scale`	`set_ik_target(...)`, `decode_vera_delta_chunk_to_targets(...)`	a positive finite number (`None` on the setter leaves the current value)
`ik_smoothing`	`VeraPolicy(...)`	`[0, 1)` — `0` disables the smoothing

rotation_dim is an enumeration rather than a range: delta_to_matrix implements axis-angle and rot6d and raises for any other width, so there is no third encoding to ask for. It is refused at the surface that receives it because it is not refused usefully later — 0/2/4 reach that dispatch mid-rollout, inside get_actions, after the server handshake and the IK bridge build; a fractional or non-numeric width reaches the per-step slice step[3 : 3 + rotation_dim] and raises TypeError: slice indices must be integers, naming neither the parameter nor the surface. An integral float (3.0, what a config read produces) is accepted and normalized, since the width has to arrive at that slice as an index.

translation_scale multiplies every translation delta on top of the OSC position scale, so 0 discards the translation half of each action and returns a rotation-only chunk, a negative value inverts it, and nan/inf make every returned joint target non-finite — refused one layer later by send_action, where it reads as a wrong-embodiment action-key mismatch rather than as the scale that caused it. ik_smoothing weights the previous target in the EMA target = (1 - alpha) * solved + alpha * previous, so 1.0 freezes the arm at its first solved pose, above 1.0 the targets diverge away from the solution (measured at -5.9x the solved joint travel for 1.5), and a negative or nan value fails the alpha > 0 test the blend is gated on — silently applying no smoothing at all.

Wire protocol¶

The provider keeps a rolling context window of the last context_frames camera frames (width-concatenated across views) and calls the server's chunked infer when its local action queue drains — the same RemotePolicy contract VERA's own eval harness uses. The server returns {"action": [H, D]}; the provider maps each D-vector to robot actuator names (gripper binarized per the server's gripper_dim_index/gripper_is_raw), coercing to python floats per the Policy ABC.

QA the rollouts with a Cosmos 3 reasoner (closed loop)¶

VERA generates video-grounded actions; NVIDIA Cosmos 3 reads video and reasons in text — so it makes a natural automated QA critic for rollouts. Serve the reasoner, then have it grade a rollout MP4:

uv pip install "strands-cosmos[cosmos3]"
# serve Cosmos3-Nano on :8000 (vLLM + vllm-cosmos3); see strands-cosmos `c3-serve-reason`
python examples/vera_mimicgen_panda/critique_with_cosmos3.py     examples/vera_mimicgen_panda/artifacts/mimicgen_panda.mp4

from strands import Agent
from strands_cosmos import Cosmos3ReasonerModel

agent = Agent(model=Cosmos3ReasonerModel(base_url="http://localhost:8000/v1"))
print(agent("Grade this robot rollout — is the motion smooth and purposeful, "
            "any bugs? <video>/tmp/vera-critique/mimicgen_panda.mp4</video>"))

This closed generate → reason → fix loop surfaced (and fixed) real issues in the MimicGen→Panda example: an initial jittery critique drove the ik_smoothing EMA knob, and a "the arm is static" critique root-caused a near-singular default start pose with the motion off-camera — fixed with a tabletop-ready seed pose + camera framing. The reasoner's verdict moved NEEDS-WORK → PASS.

Testing¶

# offline unit tests (no GPU, no vera install)
hatch run test tests/policies/vera/

# gated live integration (needs a running server)
VERA_LIVE=1 hatch run test-integ tests_integ/policies/vera/

Install¶

# 1. Provider client deps. There is no `vera` extra: the provider needs only a
#    msgpack + websocket client, and VERA itself is distributed as a git
#    repository, which no extra can pull (PyPI rejects package metadata
#    carrying a VCS reference).
pip install strands-robots websockets msgpack 'numpy>=1.24'

# 2. VERA itself, for the managed-server (subprocess) mode.
pip install 'vera @ git+https://github.com/sizhe-li/VERA.git'

# 3. Only for the MimicGen sim example: MimicGen sim deps (also pulls the
#    experimental PushT env).
pip install 'strands-robots[vera-sim]'

Note on MimicGen. The vera-sim extra does not install NVlabs MimicGen: that project has no PyPI release, and the mimicgen name on PyPI is an unaffiliated package (a dependency-confusion risk), so it is intentionally not pinned here. The mimicgen VERA embodiment is just a config string and needs no such package. If you genuinely need NVlabs MimicGen for data generation, install it from source:
pip install "mimicgen @ git+https://github.com/NVlabs/mimicgen.git"
Note on robomimic. For the same reason the vera-sim extra does not pin robomimic: its highest PyPI release is 0.3.0, while VERA's examples target v0.5.0, which exists only as an ARISE-Initiative GitHub tag. Pinning robomimic==0.5.0 would be both unresolvable (it wedges uv lock) and a dependency-confusion vector. robomimic is not imported by strands-robots; VERA pulls it in itself. If you need v0.5.0, install it from source:
pip install "robomimic @ git+https://github.com/ARISE-Initiative/robomimic.git@v0.5.0"

For the docker path the host needs only websockets + msgpack (the client transport) — no vera, no torch.