At CES 2023, Nvidia Corp. has announced major updates to its Isaac Sim robotics simulation tool that includes new people simulation, artificial intelligence (AI) capabilities, and cloud access. The new improvements will enable developers and researchers to train and optimize AI robots for a wide range of tasks from manufacturing and logistics to retail and agriculture.
Building and testing new AI-based robots requires simulation technology that places them in realistic environments, said Nvidia. The upgraded Isaac Sim simulation tool enables this testing across a range of operating conditions, and it is now accessible from the cloud.
Nvidia’s Isaac robotics platform also is comprised of several other tools including Replicator (a synthetic data generation tool), ROS (hardware-accelerated software modules that run on the robots), CuOpt (for route optimization), pre-trained models, TAO (train, adapt, and optimize) toolkit, and Sim-ready assets.
“The Isaac robotics platform is designed to accelerate the development and deployment of all manner of robots, and we have a number of software tools and SDKs that address different parts of this solution,” said Gerard Andrews, product manager for Nvidia’s robotics platform, during a CES briefing.
All of these tools are built on a foundation of two big pillars of technology – Nvidia’s AI suite of technologies and Omniverse, Nvidia’s platform for creating and operating metaverse applications, he added.
New features
Nvidia added several new features to the Isaac Sim that are designed to help both developers and researchers. These include new tools and assets for warehouse and logistics including a conveyor belt utility, new people behavior simulation for testing perception and safety systems, improved sensor performance for more accurate simulation, expanded ROS support, pre-integrated cuOpt with examples of logistics optimizations, new research tools (Isaac Gym, Isaac Cortex, Isaac ORBIT), and new pre-included robots (two mobile robots and 10 manipulators).
The driving force behind these developments center on advancing robotics simulation. “Simulation is the key technology to advancing robotics,” said Andrews. “It will be the virtual proving ground for all of your complex software in AI models that you might have in your new robot.”
Isaac Sim supports conveyor belt and people simulation. (Source: Nvidia Corp.)
Nvidia recognizes that the company’s work in simulation is not done, said Andrews. Some of the driving forces behind what Nvidia will be working on in the future is making tools more useful, which includes closing the sim2real gap, developing tools that are good at more than one thing, simulating robot fleets at scale, and building AI into the simulation tool, he added.
“Closing the sim2real gap means the more that the robot performs in simulation like it’s expected to perform in the real world then you are going to get more use cases, more utility, and more value, so we spent a lot of time focusing on how to make our simulations more realistic for that robot user or robot developer,” said Andrews.
Nvidia also focused on making the tool good at more than one thing, he said. “Specialized tools have a place in this world, but to be broadly adopted, the tool needs to be flexible, it needs to be modular, and it needs to be able to do multiple tasks,” he added.
“The third thing is envisioning a world of autonomous robots and AI is the key to enabling autonomy, and so the simulator itself needs to embrace AI in its DNA,” said Andrews. “We keep focusing on that and bringing in [different] parts of Nvidia into the platform, and then finally as robots get deployed more at scale, we need to be able to simulate these fleets at scale.”
One of the key drivers behind Nvidia’s simulation advances is the need to add people and their common behaviors to the simulations as humans increasingly work side by side with collaborative robots (cobots). The new people simulation capability allows human characters to be added to a warehouse or manufacturing facility, who are tasked with executing familiar behaviors— like stacking packages or pushing carts.
“We’re excited about people simulation – the ability to drop characters into the environment and issue commands to those characters and let them take part in a complex event-driven simulation where you can test the software on the robots,” said Andrews.
“We have a number of predefined supported behaviors in this initial release like go to a location with obstacle avoidance. You can issue commands to them to pick up or drop off an object, sit in a chair, push your cart, and stand idly, and over time we will be adding more and more behaviors,” he said.
It also is important to minimize the difference between what is observed in the simulated world versus what is seen in the real world, requiring physically accurate sensor models, said Nvidia. The Isaac Sim, using the NVIDIA RTX technology, can now render physically accurate data from sensors in real time.
“We improved our sensor performance, and specifically for LiDAR, we have ray tracing, which provides accurate performance where the sensor data generated in the simulator starts to mimic and mirror what you’ll get from the real-world sensor,” said Andrews.
For RTX-simulated LiDAR, ray tracing provides more accurate sensor data in a variety of lighting conditions, said Nvidia. It supports solid state and rotating configurations and new LiDAR models have been added including Ouster, Slamtec, and Hesai, Andrews said.
Isaac Sim improves RTX LiDAR and sensor support. (Source: Nvidia Corp.)
The upgrade also includes new simulation-ready 3D assets, which are needed for building physically accurate simulated environments. The tool provides a range of ready-to-go assets from warehouse parts to popular robots to help speed the process.
Also added to the latest upgrade are new capabilities for robotics researchers. These include the new Isaac Gym for reinforcement learning and Isaac Cortex for collaborative robot programming. A new tool, Isaac ORBIT, provides simulation operating environments and benchmarks for robot learning and motion planning.
For robot operating system developers, Isaac Sim upgrades support for ROS 2 Humble and Windows. All of the Isaac ROS software can now be used in simulation, said Nvidia.
Cloud access
The cloud has become a big focus area for Nvidia not only for Isaac Sim but for all of the company’s software to make it cloud available, said Andrews. This enables users to take advantage of the accessibility, scalability, and the collaborative nature when an application runs in the cloud, he added.
With Isaac Sim now available in the cloud, global teams working on robotics projects can collaborate much faster and easier for testing and training virtual robots. In addition, by using
Isaac Replicator developers can create massive ground-truth datasets that mimic the physics of real-world environments, and once deployed, they can use NVIDIA cuOpt, a real-time fleet task-assignment and route-planning engine, to improve operational efficiencies with automation, said Nvidia.
Replicator is Nvidia’s synthetic data generation tool, built on the Omniverse platform. A specialized version is available that runs in Isaac Sim for its robotics customers. “A lot of times collecting data to train AI models is difficult, costly, or even dangerous to do and so Replicator gives you a way to augment your data sets and to build more robust AI models,” said Andrews.
“We believe simulation is the critical technology to advance robotics and it will be the proving ground for robots,” said Andrews. “We have numerous customers that are working with us that have shared how they have been able to use Isaac Sim so far.”
Over one million developers and more than a thousand companies have used one or many parts of the Isaac robotics ecosystem, according to Nvidia. These include companies that have deployed physical robots, which were developed and tested in the virtual world using Isaac Sim.
Use case examples range from Telexistence’s beverage restocking robots and Sarcos Robotics’ robots that pick and place solar panels in renewable energy installations to Fraunhofer’s development of advanced AMRs and Flexiv’s use of Isaac Replicator for synthetic data generation to train AI models.
Nvidia announced a range of products, partnerships and offerings in autonomous machines, robotics, design, and simulation at CES 2023, January 5-8, Las Vegas, Nevada.
Advertisement
Learn more about NVIDIA
