Skip to content

Isaac Sim Backend (GPU)

The Isaac Sim backend runs the simulation on NVIDIA Isaac Sim (PhysX GPU physics + RTX path-traced rendering). It is a built-in, in-tree backend that lives at strands_robots.simulation.isaac, a peer of the mujoco and newton backends. It implements the same SimEngine contract as the MuJoCo backend, so the Robot() / Simulation / policy APIs are identical - only the physics and rendering run on the GPU through Isaac Sim.

strands-robots has no hard dependency on Isaac Sim: the sim-isaac extra provides the pip-installable helpers, and create_simulation("isaac") resolves the built-in backend, exactly like create_simulation("mujoco"). The Isaac Sim runtime itself (~30 GB) is provisioned separately - via its own pip wheels on Python 3.12, or out-of-band (see below).

When to use it

  • You have an NVIDIA RTX GPU (Ubuntu 22.04+, CUDA 12+) and want photoreal, path-traced observations for sim2real visuals or paper-grade frames.
  • You want USD-native scenes (real CAD assets, Nucleus, IsaacLab compatibility).
  • You want Replicator synthetic data - ground-truth depth, segmentation, and bounding boxes alongside RGB.
  • You want fleet RL on PhysX GPU with 1024+ parallel environments.

On macOS / Apple Silicon or CPU-only hosts, install the lightweight default strands-robots and use the MuJoCo backend instead - it runs everywhere and the agent contract is identical. Isaac Sim is a ~30 GB install and requires an NVIDIA GPU.

Install

Install the Isaac Sim runtime first, then the sim-isaac extra:

# Step 1 - install Isaac Sim 6.0 (Python 3.12) via one of:
#   - pip wheels (see caveats below):
#       pip install 'isaacsim[all,extscache]==6.0.*' --extra-index-url https://pypi.nvidia.com
#   - Omniverse Launcher -> Isaac Sim 6.0, OR
#   - Isaac Lab: git clone IsaacLab && ./isaaclab.sh -i, OR
#   - NGC Docker: docker pull nvcr.io/nvidia/isaac-sim:6.0

# Step 2 - install the sim-isaac extra (helpers for the built-in backend):
pip install 'strands-robots[sim-isaac]'

The sim-isaac extra lives in strands-robots (a peer of sim-mujoco and sim-newton). Requesting create_simulation("isaac") without the extra installed raises a ValueError whose message carries the exact install hint (pip install 'strands-robots[sim-isaac]'). Backend discovery is lazy, so MuJoCo-only users never pay the Isaac Sim import cost.

Installing Isaac Sim via pip - caveats

Since the cp312 wheels shipped for Isaac Sim 6.0.x, the runtime itself is pip-installable on Python 3.12. The extscache extra is required - the bare isaacsim[all] metapackage omits the isaacsim-extscache-* packages, and SimulationApp aborts resolving its extension graph without them. The pip install also degrades an existing dev environment in ways pip only warns about, so run this exact sequence:

# 1. Install the Isaac Sim wheels (NVIDIA index required):
pip install 'isaacsim[all,extscache]==6.0.*' --extra-index-url https://pypi.nvidia.com

# 2. Repair the coverage downgrade (see below):
pip install 'coverage>=7.6.1'

# 3. Accept the EULA for non-interactive first import:
export OMNI_KIT_ACCEPT_EULA=YES

Known collateral (observed with isaacsim 6.0.0.1 and 6.0.1.0):

  • coverage downgrade breaks robosuite/LIBERO with a red-herring error. isaacsim-kernel pins coverage==7.4.4, silently downgrading modern coverage. numba's tracer probe then fails, and the first visible symptom is far from the cause - robosuite's OSC controller import dies inside the LIBERO adapter with module 'coverage.types' has no attribute 'Tracer'. Verified remedy: pip install 'coverage>=7.6.1' after the isaacsim install. The reverse pip conflict warning (isaacsim-kernel requires coverage==7.4.4) is cosmetic: coverage is test tooling for the kit, not a runtime dependency.
  • torch stack bump vs lerobot pins. The isaacsim install upgrades torch/torchvision (and numpy/scipy/pyarrow), leaving pip conflict warnings against lerobot's torchvision pin. Expect those warnings; they do not by themselves indicate breakage. Validated combination as of 2026-07-31: isaacsim 6.0.x with torch 2.11 / torchvision 0.26.0 alongside lerobot 0.5.1 - GR00T-on-MuJoCo re-verified green post-install. The environment is outside lerobot's declared support, so re-verify your own policy path after installing.
  • EULA prompt on first import. Any non-interactive first import fails with Do you accept the EULA? ... EOF when reading a line unless OMNI_KIT_ACCEPT_EULA=YES is set.
  • Exit code 134 after successful work. Isaac Sim has a known atexit segfault that makes otherwise-clean scripts exit 134 after completing successfully. The drivers in this repo guard with os._exit(...) after SimulationApp teardown (see the isaac subcommand epilogue in examples/libero/run.py); user scripts that boot SimulationApp should do the same.

Usage

from strands_robots.simulation import create_simulation

# Kwargs flow into IsaacConfig. "isaac" resolves as a built-in backend.
sim = create_simulation("isaac", render_mode="rtx_realtime", headless=True)
sim.create_world()
sim.add_robot("so100")                          # procedural; no asset files needed
sim.add_object(name="cube", shape="cuboid",
               position=[0.4, 0.0, 0.05], scale=[0.05, 0.05, 0.05])
sim.add_camera(name="front", position=[1.2, 0.0, 0.6], target=[0.0, 0.0, 0.1])
sim.step(120)
frame = sim.render(camera_name="front")          # RGB + depth
sim.destroy()

Robot("so100", backend="isaac", ...) routes through the same factory, so the backend selection is identical whether you go through Robot() or create_simulation().

scale= above is an accepted alias for add_object(size=...), and it is the only extra keyword that method reads. Any other keyword is refused by name rather than dropped -- the same contract IsaacConfig applies to create_simulation kwargs, and the same verdict the MuJoCo and Newton backends give (they declare the same add_object parameters and no **kwargs, so an unknown keyword is a TypeError there):

sim.add_object(name="cube", heigth=0.3)
# {"status": "error", "content": [{"text":
#   "Unknown parameter(s) ['heigth'] for action 'add_object'. Valid: [...]"}]}

Configuration (IsaacConfig)

Keyword arguments to create_simulation("isaac", ...) (or Robot(..., backend="isaac", ...)) construct an IsaacConfig. Unknown keys are rejected eagerly. The commonly used fields:

Kwarg Type Default Description
num_envs int 1 Parallel environments. Set to 1024+ for fleet RL.
device str "cuda:0" CUDA device (cuda:N). Must be a CUDA device.
headless bool True Run without a GUI (required for cloud/CI).
physics_dt float 1/120 Physics timestep (seconds).
rendering_dt float 1/30 Rendering timestep (seconds).
render_mode str "headless" "headless", "rtx_realtime" (raster), or "rtx_pathtracing" (photoreal).
gravity tuple (0, 0, -9.81) Gravity vector (Z-up).
ground_plane bool True Add a ground plane on create_world().
stage_path str "/World" USD stage path prefix.
nucleus_url str \| None None Override Omniverse Nucleus URL (env-resolvable).
camera_width / camera_height int 640 / 480 Default camera resolution.
enable_rtx_sensors bool True Enable RTX-accelerated camera / LiDAR sensors.
verbose bool False Verbose Isaac Sim / Kit logging.

Environment variables

The Isaac backend reads three STRANDS_ISAAC_* variables (resolved when IsaacConfig is constructed). STRANDS_ISAAC_NUCLEUS_URL is read only when nucleus_url is not passed, so there the kwarg wins; the two switches override their field whenever they are set. Which of those two directions the switches should have is #2062.

Both switches accept four symmetric pairs, case-insensitively and ignoring surrounding whitespace:

on off
1 0
true false
yes no
on off

Unset -- or set to an empty value, which is what an undefined ${{ vars.* }} interpolation in a GitHub Actions env: block produces -- leaves the field alone. Any other spelling raises ValueError naming both vocabularies, rather than falling through to the off side: STRANDS_ISAAC_HEADLESS=enabled used to open a window.

Variable Description Default
STRANDS_ISAAC_NUCLEUS_URL Override the Omniverse Nucleus server URL when nucleus_url is not passed unset (Isaac defaults)
STRANDS_ISAAC_HEADLESS On forces headless; off forces windowed unset (uses headless kwarg)
STRANDS_ISAAC_RTX_PATHTRACING On forces render_mode="rtx_pathtracing"; off leaves render_mode alone unset

Capabilities and parity

IsaacSimulation exposes the same SimEngine shape as the MuJoCo backend:

  • World & lifecycle - create_world, destroy, reset, step, get_state, cleanup.
  • Robots - add_robot (procedural builders, or USD via usd_path=, or URDF), remove_robot, list_robots, robot_joint_names, send_action, get_observation.
  • Objects - add_object (cuboid / sphere / cylinder / capsule / mesh, dynamic or static), remove_object. A shape="mesh" add takes a mesh_path to an STL/OBJ/MSH asset (converted to USD once and cached under $STRANDS_BASE_DIR/asset_cache/usd_meshes/, content-addressed) or to a USD file (referenced directly). The asset defines the extent - size is ignored for a mesh, the MuJoCo read of that parameter (the Newton backend consumes it as a scale instead; see #2300) - and collision uses the mesh's convex hull, also the MuJoCo contract, with the same caveat for concave assets: the hull fills every cavity. A missing file, an unconvertible format, or an asset declaring a vertex coordinate that is not finite is refused up front, never realized as a fallback primitive.
  • Cameras & rendering - add_camera (look-at, FOV), render (RGB + depth). World-fixed only: parent_body (a body-mounted wrist camera, supported on mujoco/newton) is refused here with an error naming those backends, because camera prims are parented to the stage camera scope rather than to an articulation link.
  • Loaders - load_urdf / load_mjcf / load_usd resolve to a ProceduralRobot dataclass. Both XML loaders report each link's pose in its parent's frame. load_mjcf reads the rotation from whichever of MJCF's five spellings the body uses - quat, euler, axisangle, xyaxes or zaxis - under the model-global <compiler angle> and <compiler eulerseq>. The reported orientation is always a unit quaternion, the one MuJoCo's compiler stores: a non-unit spelling such as quat="1 -1 0 0" (the idiomatic quarter turn) is reported as the quarter turn it means, not as the components as written, which applied as a rotation would scale the frame by |q|^2 as well as turning it. load_urdf reads both halves from the <origin> of the joint that reaches the link, since URDF places a link on that joint rather than on the <link> element: xyz into position and rpy - fixed-axis roll-pitch-yaw, always radians - into orientation. A root link, reached by no joint, keeps the identity pose. A joint's axis comes from <axis xyz>, and each format's own default applies when the element states no vector the parser can read: a URDF joint that omits the optional <axis> acts about +X, an MJCF <joint> that omits axis about +Z. Both are valid axes, so a joint read under the other format's default would be reported acting in the perpendicular plane with the load still reporting success. Both of MJCF's spellings of a free joint are read - the dedicated <freejoint> element and <joint type="free">, which MuJoCo compiles to the same joint - so a floating base is reported rather than absent. <freejoint> is how every shipped quadruped and humanoid states its base, and it resolves no default class, because MJCF has no <default><freejoint> block: a <default><joint> class reaches the type="free" spelling only, exactly as MuJoCo applies it. Either spelling is reported with joint_type="fixed", since JointDef has no 6-DOF spelling, so a floating base is visible in joints without being counted as an actuated DOF by num_joints.

Because the joint-name and observation contract matches the MuJoCo backend, policies and observation mappings transfer unchanged between backends.

LIBERO scenes get the same treatment for their visuals: load_scene renders each task object with its real mesh (bowls, plates - the assets a pixel-conditioned policy was trained on) while keeping the validated collision-AABB box as the invisible physics proxy, so switching backends does not also switch what the cameras see. That box covers both MJCF spellings of a capsule or cylinder - pos plus size="radius half-length", and fromto plus size="radius", where the two endpoints carry the placement and the axis extent - so a fromto bar is proxied by its full length at its midpoint rather than by a ball of its radius at the body origin. An object whose mesh cannot be resolved keeps a visible box proxy, and the load_scene report then carries an explicit caveat that pixel-conditioned policy scores on that scene are not comparable across backends; when every object renders its mesh, the caveat disappears. A mesh asset that is declared but missing on disk fails the scene load loudly - never a silent box - and so does one declaring a vertex coordinate that is not finite: min/max order a NaN as neither smaller nor larger than anything, so the collision AABB measured from such an asset is the box of the vertices that are finite, numerically indistinguishable from a mesh that declared only those, while an infinite coordinate makes the proxy unbounded. MuJoCo refuses the same asset (vertex coordinate N is not finite), so the scene loader does too rather than sizing a proxy around it.

The accepted input domain matches too, so a call one backend refuses is refused by all three. For the setup methods that means the pose vectors, an object's color and mass, the camera fov and the pixel dimensions - and the entity name: add_robot, add_object and add_camera each require a non-empty string containing no NUL. That matters more here than on MuJoCo because the name is interpolated into the USD prim path ({stage_path}/Robots/{name}), so an unaddressable name does not just produce an entity you cannot look up - add_robot("") resolved to /World/Robots/, the container scope for every robot, and remove_robot prunes its cleanup registry by that prefix. Unlike the MuJoCo backend there is no "derive a label from the model" short form: name is also the procedural lookup key, so None / "" are refused rather than replaced with a generated label.

The one deliberate difference in that list is mass=0. The Newton backend documents it as an alternative spelling of is_static=True and honours it, so it stays accepted there; this backend documents no such spelling, so a zero mass is refused with is_static=True named as the remedy - the MuJoCo contract these docs otherwise mirror. A static object's mass is read by nobody on any backend, so it is not validated there.

Looking an entity up is the other half of that contract, and it answers rather than refuses: a name only addresses an entity here, so a name that cannot be a registry key is honestly absent. remove_robot, remove_object, remove_camera, send_action, move_object, get_body_state and the rest report it with the unknown-entity message they already had, robot_joint_names and get_observation keep answering empty, and get_frame / get_camera_params raise the KeyError their contract names. Previously the membership test itself raised TypeError: unhashable type for a list or dict name, so the miss escaped the envelope those methods document as their only failure channel - reachable with no entities registered at all.

render is the one lookup that cannot answer with a frame, so it reports the same verdict as its raw sibling. Given a camera_name that names a camera the scene does not carry it returns {"status": "error"} with Camera '<name>' not found. Available: [...] - the message get_frame raises for the identical name, and the one the MuJoCo and Newton render give. It used to report status="success" with an all-black frame instead, tagged Rendered (no camera), plus pixel_mean 0.0 as a measurement and the missing name in the camera field; because that envelope carries the PNG block the shared frame extractor reads, a rollout recording a mistyped camera wrote an all-black video and reported success. A camera_name that names no camera - None, "", "default" (the signature default) or "free" - still gets that blank frame: Isaac has no free camera to fall back to, unlike the two backends whose render entry points resolve those tokens to one, so for them it is a degradation rather than a mistake. Registering a camera under one of those names is accepted here and renders normally, since nothing on this backend routes them.

Fleet (IsaacLab-style) preview

sim = create_simulation("isaac", num_envs=1024, headless=True,
                        render_mode="headless")
sim.create_world()
sim.add_robot(name="panda", usd_path="/path/to/franka.usda")
# ... RL training loop ...
sim.destroy()

Where to go next

The Isaac backend was originally prototyped in the strands-robots-sim project, which still hosts a MkDocs site with additional architecture notes and troubleshooting. It is kept here as background reference; the backend itself now ships in-tree in strands-robots: