Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI Agents

Turning a simulation idea into a working application means assembling assets, connecting physics and rendering, and checking that the scene behaves as intended. Developers are combining frontier AI models with NVIDIA Omniverse libraries to help carry out that work — building applications for exploring scenarios, investigating failures and improving designs. Developers direct AI agents through natural-language instructions, review results and guide changes. Omniverse libraries provide GPU-accelerated physics, rendering and sensor simulation capabilities. Explore the projects below to see frontier AI models such as GPT-6 Astra at work, and check back for new examples from NVIDIA teams and developers across the ecosystem. Create Editable 3D Assets With Houdini and AI Agents

Creating animated 3D content requires connecting modeling, simulation, materials and rendering. Nick Scarcella, a senior technical artist on NVIDIA’s BondLab team, directed AI agents — primarily using Claude — to work with CUDA libraries and Isaac Sim to create animations. The BondLab team uses SideFX Houdini and the NVIDIA ovrtx library to develop materials, lighting and the final rendered appearance. In one experiment, Scarcella guided Astra to recreate an OP-1 portable synthesizer using product images, manuals and technical direction. He exported the model into the OpenUSD framework, tested it in Houdini and Blender, then used Astra and Opus to refine details before adjusting materials and textures for ovrtx rendering. Scarcella also used a publicly shared robot-hand checkpoint in Isaac Sim, directing the motion and developing materials and lighting in Houdini with Claude’s assistance. Working with team member Steven Dvornik, the team created an “Echos” visualization: translucent versions of the hand overlap and merge into a single hand animation, offering a visual interpretation of robot training. https://blogs. nvidia. com/wp-content/uploads/2026/10/Robot_Hand_Video_Omniverse. mp4

Figure: An artistic visualization of simulated robot-hand motion, developed with AI assistance and Houdini’s materials and lighting tools. These workflows bring agent-assisted creation into existing artist tools, giving teams editable content they can inspect, refine and render across applications. Explore Scarcella’s Houdini MCP tools and agent skill . Test Robotic Disassembly With Computer-Aided Design and Simulation

Before a robot can disassemble a product, developers need to know whether its tools can reach and remove the parts. Jens Jebens, a senior product manager for OpenUSD at NVIDIA, directed Astra to model a car suspension in PTC Onshape and configure it in NVIDIA Isaac Sim . Jebens used simulation feedback to revise the tooling, including a wrench designed to reach the suspension’s bolts, and reported successful removal of a suspension component in simulation. https://blogs. nvidia. com/wp-content/uploads/2026/11/Robotic_Dissasembly_Omniverse. mp4

PTC also connects computer-aided design (CAD) with simulation through Onshape’s SimReady Publisher, built with NVIDIA Omniverse libraries. Robotics startup Sensori uses it to transfer its Yuri robot into Isaac Sim, carrying geometry, joints and configuration data so the team can evaluate design changes without repeatedly rebuilding the simulation model. Read Sensori’s story . Explore the Onshape importer guide. Create an Editable Building Model From Site Scans

https://blogs. nvidia. com/wp-content/uploads/2026/10/relay-video-v14-1080p. mp4

Planning warehouse automation requires a model of the building itself, but shelving and machinery can obscure its structure in site scans. Omniverse brings together real-to-sim tools and agent skills to help developers turn real-world captures into editable starting environments for physical AI simulation. David Wehr, from NVIDIA’s simulation technology team, built on that work using Astra to develop a workflow for digitally reconstructing a building without its contents. The goal is an editable, simulation-ready building model where developers can add or reposition shelving and machinery to evaluate warehouse layouts and automation plans before making physical changes. Wehr guided Astra to interpret panoramic photos, create building geometry and align it with point-cloud measurements. OpenUSD stores the models, while Omniverse’s ovrtx library renders results for inspection. Explore NVIDIA Omniverse agent skills to help agents build with Omniverse libraries. Test Gas-Turbine Operating Scenarios With a Digital Twin

https://blogs. nvidia. com/wp-content/uploads/2026/10/Turbine-Lab-RTX-and-What-If-under-10MB-1. mp4

Before changing how industrial equipment operates, engineers need a way to explore the consequences. Pradeep Kaushik, a member of the NVIDIA Omniverse developer relations team, directed Astra to build a what-if analysis application around a simulated gas turbine and generator. The application lets users adjust conditions such as load, ambient temperature and fuel composition, then compare power, efficiency, fuel consumption and equipment response. Six test procedures explore scenarios including load changes, a stuck fuel valve and an exhaust-sensor error. The prototype gives engineers an interactive way to investigate how operating changes and faults could affect equipment behavior, supporting decisions about which scenarios warrant further testing. Astra used tools like Cantera for thermochemistry and SciPy for equipment dynamics. The application maps solver results to rotor animation and color-coded temperature, pressure and flow-velocity indicators in a 3D turbine. NVIDIA Omniverse libraries update the scene ( ovstage ), render it with RTX (ovrtx) and stream the view to the browser (ovstream). Engineers can rotate and zoom the cutaway, switch between indicators, and pause or scrub through a simulated response alongside charts and virtual instruments. Build Simulation Scenes With Open Source SimReady Assets

https://blogs. nvidia. com/wp-content/uploads/2026/10/kitchen_final_blog_with_music. mp4

With SimReady Central Library, announced at GTC Berlin , developers can access more than 2,000 simulation-ready USD assets from equipment manufacturers and asset providers for robotics simulations and AI factory digital twins workflows. Paul Kind, a member of NVIDIA’s Omniverse documentation team, pointed an AI agent to the library and guided scene assembly through conversation. His kitchen scene included 180 placements drawn from 63 unique library assets, alongside 18 custom geometry pieces. Kind reviewed renders produced with Isaac Sim and guided further adjustments. For robotics and factory workflows, developers can select SimReady assets with the simulation properties embedded to fit their required task then test in an assembled environment. Find assets in SimReady Central Library , and learn how to contribute . Stress-Test a Warehouse Simulation at Scale

https://blogs. nvidia. com/wp-content/uploads/2026/10/Infra-scale-demo-20260914_144614-Meeting-Recording. mp4

A warehouse simulation needs to keep running smoothly as more robot arms, conveyors and boxes fill the scene. When it slows down, developers need to identify what’s holding it back. Bhavik Sanganee, principal engineer at NVIDIA, used Astra with NVIDIA Omniverse libraries to build a warehouse simulation, increase its complexity and investigate slowdowns. Performance first fell below the target when the scene doubled in size. Running physics calculations, image rendering and video preparation concurrently restored performance and enabled further scaling. With 80 conveyors, 40 arms and 800 boxes, the simulation sustained 30 frames per second at full HD resolution during a two-minute test. The arms completed 640 successful box placements. Doubling the scene again reduced performance. Testing traced the slowdown to the physics and robot-control pipeline, giving Sanganee a specific area to improve. Start building with NVIDIA Omniverse libraries …