The NVIDIA Isaac robotics platform is tapping into the latest generative AI and advanced simulation technologies to accelerate AI-enabled robotics.
At GTC 2024, NVIDIA announced Isaac Manipulator and Isaac Perceptor - a collection of foundation models, robotics tools and GPU-accelerated libraries.
During Monday’s keynote, NVIDIA founder and CEO Jensen Huang demonstrated Project GR00T, a general-purpose foundation model for humanoid robot learning. Project GR00T uses various new tools from the NVIDIA Isaac robotics platform to create AI for humanoid robots.
“Building foundation models for general humanoid robots is one of the most exciting problems to solve in AI today,” Huang said. “The enabling technologies are coming together for leading roboticists around the world to take giant leaps toward artificial general robotics.”
NVIDIA also announced a new computer for humanoid robots based on the NVIDIA Thor system-on-a-chip (SoC), and new tools for the NVIDIA Isaac robotics platform, including Isaac Lab for robot learning and NVIDIA OSMO for hybrid-cloud workflow orchestration. Both tools are instrumental in the development of Project GR00T and foundation models for robots.
Isaac Manipulator for robotic arms
NVIDIA Isaac Manipulator offers a collection of state-of-the-art motion generation and modular AI capabilities for robotic arms, with a robust collection of foundation models and GPU-accelerated libraries.
Robotics developers can use combinations of software components customized for specific tasks to perceive and interact with surroundings, enabling the building of scalable and repeatable workflows for dynamic manipulation tasks by accelerating AI model training and task programming.
“Incorporating new tools for foundation model generation into the Isaac platform accelerates the development of smarter, more flexible robots that can be generalized to do many tasks,'' said Deepu Talla, vice president of robotics and edge computing at NVIDIA.
Leading robotics companies Yaskawa, Solomon, PickNik Robotics, READY Robotics, Franka Robotics, and Universal Robots, a Teradyne company, are partnering with NVIDIA to bring Isaac Manipulator to their customers.
“By bringing NVIDIA AI tools and capabilities to Yaskawa's automation solutions, we're pushing the boundaries of where robots can be deployed across industries,“ said Masahiro Ogawa, President, Yaskawa. “This will significantly influence various industries.”
NVIDIA is introducing foundation models to augment existing robot manipulation systems. These will help develop robots to sense, adapt and reprogram for varied environments and applications in smart manufacturing, handling pick-and-place tasks, machine tending and assembly with the following:
- FoundationPose is a foundation model for 6D pose estimation and tracking of previously unseen objects.
- cuMotion taps into the parallel processing of NVIDIA GPUs for solving robot motion planning problems at industrial scale by running many trajectory optimizations at the same time to provide the best solution.
- FoundationGrasp is a transformer-based model that can make dense grasp predictions for unknown 3D objects.
- SyntheticaDETR is an object detection model for indoor environments that allows faster detection, rendering and training with new objects.
Isaac Perceptor for AMRs and visual AI
Manufacturing and fulfillment operations are adopting autonomous mobile robots (AMRs) to improve efficiency and worker safety as well as to reduce error rates and costs.
Isaac Perceptor provides multi-camera, 360-degree vision capabilities, offering early industry partners such as ArcBest, BYD and KION Group advanced visual AI for their AMR installations that assist in material handling operations.
The NVIDIA Nova Orin DevKit - created in collaboration with Segway Robotics and Leopard Imaging - allows companies to quickly develop, evaluate and deploy Isaac Perceptor.
“ArcBest is collaborating with NVIDIA to bring leading-edge machine vision technology into the logistics space,” said Michael Newcity, chief innovation officer of ArcBest and president of ArcBest Technologies. “Using the Isaac Perceptor platform in our Vaux Smart Autonomy AMR forklifts and reach trucks enables better perception, semantic-aware navigation and 3D mapping for obstacle detection in material handling processes across warehouses, distribution centers and manufacturing facilities.”
Project GR00T for humanoid robotics development
Also demonstrated at GTC 2024, GR00T-powered humanoid robots can take multimodal instructions - text, video and demonstrations - as well as their previous interactions to produce the desired action for the robot. GR00T was shown on four humanoid robots from different companies, including Agility Robotics, Apptronik, Fourier Intelligence and Unitree Robotics.
Humanoid robots are complex systems that require heterogeneous computing to meet the needs of high frequency low level controls, sensor fusion and perception, task planning and human-robot interaction. NVIDIA unveiled a new Jetson Thor-based computer for humanoid robots, built on the NVIDIA Thor SoC.
Jetson Thor includes a next-generation GPU based on the NVIDIA Blackwell architecture with a transformer engine delivering 800 teraflops of 8-bit floating point AI performance to run multimodal generative AI models like GR00T. With an integrated functional safety processor, a high-performance CPU cluster and 100GB of ethernet bandwidth, it significantly simplifies design and integration efforts.
Project GR00T uses Isaac tools that are available to robotics developers for building and testing foundation models. These include Isaac Lab, a new lightweight simulation app built in Isaac Sim to train this humanoid robot model at scale, and OSMO, a cloud workflow orchestration platform for managing the training and simulation workloads.
Isaac Lab and robot learning
Robots that require advanced locomotion skills, whether with walking or grasping, need to use deep reinforcement learning in a simulated environment and be trained repeatedly in a virtual environment to learn skills. However, this utility becomes more useful when the model transfers to the real robot deployment, which has been demonstrated with Project GR00T.
As the successor to Isaac Gym, Isaac Lab benefits from NVIDIA Omniverse technologies for physics-informed, photorealistic, perception-based reinforcement learning tasks. Isaac Lab is an open-source, performance-optimized app for robot learning built on the Isaac Sim platform. It incorporates reinforcement learning APIs and a developer-friendly tasking framework.
Cloud-native robotics workflow scheduling with NVIDIA OSMO
NVIDIA OSMO scales workloads across distributed environments. For robotics workloads with complex multi-stage and multi-container workflows, the platform provides a location-agnostic deployment option and dataset management and traceability features for deployed models.
“Boston Dynamics employs a range of machine learning, reinforcement learning and AI technologies to power our robots,” said Pat Marion, machine learning and perception lead at Boston Dynamics. “To effectively manage the large training workloads, we’re using NVIDIA OSMO, an infrastructure solution that lets our machine learning engineers streamline their workflows and dedicate their expertise to tackling the hard robotics problems.”
OSMO supports GR00T, for example, by concurrently running models on NVIDIA DGX for training and NVIDIA OVX servers for live reinforcement learning in simulation. This workload involves generating and training models iteratively in a loop. OSMO’s ability to manage and schedule workloads across distributed environments allows for the seamless coordination of DGX and OVX systems, enabling efficient and iterative model development. Once the model is ready for testing and validation, OSMO can uniquely orchestrate software-in-the- loop workflows on OVX (x86-64) as well as hardware-in-the-loop workflows with NVIDIA Jetson (aarch64) compute resources.
Supporting the ROS Ecosystem of Developers
NVIDIA joined the Open Source Robotics Alliance (OSRA) as a founding member and platinum sponsor. OSRA is a new initiative by the Open Source Robotics Foundation to foster collaboration, innovation and technical guidance in the robotics community by supporting several open-source robotics projects, including the Robot Operating System (ROS).
“The increasing capability of autonomous robots is driving a rise in demand for more powerful but still energy-efficient onboard computing,” said Vanessa Yamzon Orsi, CEO of Open Robotics. “The ROS community is experiencing this demand firsthand, and our users are increasingly taking advantage of advanced accelerated computing hardware from industry leaders such as NVIDIA.”
NVIDIA Isaac Perceptor with Nova Orin evaluation kit, Isaac Manipulator, Isaac Lab and OSMO will be made available to customers in the second quarter of this year.
Article topics
Email Sign Up
Related Artificial Intelligence News
Related Companies
Related Resources
