NVIDIA's Jim Fan: The Robotics Endgame is Closer Than You Think

And up first, I'm delighted to introduce And up first, I'm delighted to introduce my friend Jim Fan. Uh Jim leads the my friend Jim Fan. Uh Jim leads the my friend Jim Fan. Uh Jim leads the embodied autonomous research uh group at embodied autonomous research uh group at embodied autonomous research uh group at NVIDIA, otherwise known as NVIDIA NVIDIA, otherwise known as NVIDIA NVIDIA, otherwise known as NVIDIA Robotics. Um I think that robot robots Robotics. Um I think that robot robots Robotics. Um I think that robot robots are just one of the most thrilling are just one of the most thrilling are just one of the most thrilling things that's going to happen. Uh a car things that's going to happen. Uh a car things that's going to happen. Uh a car basically is a big robot, but I'm basically is a big robot, but I'm basically is a big robot, but I'm excited for robots can go beep boop and excited for robots can go beep boop and excited for robots can go beep boop and lift things for us. And so Jim was Jim lift things for us. And so Jim was Jim lift things for us. And so Jim was Jim was a standout at last year's AIN, and was a standout at last year's AIN, and was a standout at last year's AIN, and we're delighted to have you back. we're delighted to have you back. we're delighted to have you back. >> Thanks [applause] everyone. Thanks.

>> Thanks [applause] everyone. Thanks. >> Thanks [applause] everyone. Thanks. So, it was a summer day in 2016. So, it was a summer day in 2016. So, it was a summer day in 2016. Actually, right in this office that Actually, right in this office that Actually, right in this office that we're sitting in, [snorts] there's a guy we're sitting in, [snorts] there's a guy we're sitting in, [snorts] there's a guy in shiny leather jacket, you know, big in shiny leather jacket, you know, big in shiny leather jacket, you know, big biceps, hurling in this large metal biceps, hurling in this large metal biceps, hurling in this large metal tray. And on this large piece of metal, tray. And on this large piece of metal, tray. And on this large piece of metal, he wrote, "To Elon and the OpenAI team, he wrote, "To Elon and the OpenAI team, he wrote, "To Elon and the OpenAI team, to the future of computing and humanity, to the future of computing and humanity, to the future of computing and humanity, I present you the world's first DGX1." I present you the world's first DGX1." I present you the world's first DGX1." So, that was the first time I met So, that was the first time I met So, that was the first time I met Jensen. And as any good intern would do, Jensen. And as any good intern would do, Jensen. And as any good intern would do, I rush to get in line to sign my name on I rush to get in line to sign my name on I rush to get in line to sign my name on it. So, can you spot it? My name is it. So, can you spot it? My name is it. So, can you spot it? My name is here. And can you spot another? That's here. And can you spot another? That's here. And can you spot another? That's Andre right there. So, Andre, we're Andre right there. So, Andre, we're Andre right there. So, Andre, we're going to the computer history museum. I going to the computer history museum. I going to the computer history museum. I feel like a dinosaur. feel like a dinosaur. feel like a dinosaur. You know, back then, I had no clue what You know, back then, I had no clue what You know, back then, I had no clue what I was signing up for. And then, no one I was signing up for. And then, no one I was signing up for. And then, no one can describe what happened next better can describe what happened next better can describe what happened next better than Ilia himself. If you believe in than Ilia himself. If you believe in than Ilia himself. If you believe in deep learning, deep learning will deep learning, deep learning will deep learning, deep learning will believe in you. And oh boy, did deep believe in you. And oh boy, did deep believe in you. And oh boy, did deep learning believe in all of us big time. learning believe in all of us big time. learning believe in all of us big time. Three-step functions, six years. That's Three-step functions, six years. That's Three-step functions, six years. That's how all it took to bring us here today. how all it took to bring us here today. how all it took to bring us here today. The first take GP3 pre-training The first take GP3 pre-training The first take GP3 pre-training next token prediction is really about next token prediction is really about next token prediction is really about learning the rules of grammar, the shape learning the rules of grammar, the shape learning the rules of grammar, the shape of language. It's about simulating how of language. It's about simulating how of language. It's about simulating how thoughts and code and strings in general thoughts and code and strings in general thoughts and code and strings in general should unfold. should unfold. should unfold. 2022 instruct GPT supervised fine-tuning 2022 instruct GPT supervised fine-tuning 2022 instruct GPT supervised fine-tuning align the simulation to do useful work align the simulation to do useful work align the simulation to do useful work or one reasoning using reinforcement or one reasoning using reinforcement or one reasoning using reinforcement learning to surpass imitation learning learning to surpass imitation learning learning to surpass imitation learning and finally auto research accelerating and finally auto research accelerating and finally auto research accelerating the whole loop beyond what's humanly the whole loop beyond what's humanly the whole loop beyond what's humanly possible. So, as Andre said, all the

possible. So, as Andre said, all the possible. So, as Andre said, all the labs are getting to the final boss labs are getting to the final boss labs are getting to the final boss fight. So, for LMS, they're in the thick fight. So, for LMS, they're in the thick fight. So, for LMS, they're in the thick of the end game. And honestly, I'm very of the end game. And honestly, I'm very of the end game. And honestly, I'm very jealous. Look at how happy Andre was. jealous. Look at how happy Andre was. jealous. Look at how happy Andre was. Big smile on his on his face. The OM Big smile on his on his face. The OM Big smile on his on his face. The OM folks are having the party of their folks are having the party of their folks are having the party of their lifetime. They're speedrunning AGI on lifetime. They're speedrunning AGI on lifetime. They're speedrunning AGI on mythical creatures literally called mythical creatures literally called mythical creatures literally called mythos. So, why can't robotics get a mythos. So, why can't robotics get a mythos. So, why can't robotics get a piece of fun? piece of fun? piece of fun? So as any self-respecting scientist So as any self-respecting scientist So as any self-respecting scientist would do, I copy homework and I give it would do, I copy homework and I give it would do, I copy homework and I give it a new name. I call it a great parallel. a new name. I call it a great parallel. a new name. I call it a great parallel. So instead of simulating strings, can we So instead of simulating strings, can we So instead of simulating strings, can we simulate next physical world state and simulate next physical world state and simulate next physical world state and then we can align through action then we can align through action then we can align through action fine-tuning onto a thin slice of that fine-tuning onto a thin slice of that fine-tuning onto a thin slice of that simulation that matters for real robots. simulation that matters for real robots. simulation that matters for real robots. And we let reinforce learning carry the And we let reinforce learning carry the And we let reinforce learning carry the last mile. last mile. last mile. And that's it. the great paralo coping And that's it. the great paralo coping And that's it. the great paralo coping the LM success. If you can't beat them, the LM success. If you can't beat them, the LM success. If you can't beat them, join them. So, please join me in a new join them. So, please join me in a new join them. So, please join me in a new episode, robotics, the endgame. And episode, robotics, the endgame. And episode, robotics, the endgame. And sorry, I just couldn't resist. Nano sorry, I just couldn't resist. Nano sorry, I just couldn't resist. Nano banana is too good. Thanks, Dennis. banana is too good. Thanks, Dennis. banana is too good. Thanks, Dennis. So, how do we play the end game? It

So, how do we play the end game? It So, how do we play the end game? It boils down to two things. Model strategy boils down to two things. Model strategy boils down to two things. Model strategy and data strategy. Let's look at a model and data strategy. Let's look at a model and data strategy. Let's look at a model first. first. first. The last three years were dominated by The last three years were dominated by The last three years were dominated by VAS or visual language action models and VAS or visual language action models and VAS or visual language action models and models like pi and Groot fall in this models like pi and Groot fall in this models like pi and Groot fall in this category. So we assume that the category. So we assume that the category. So we assume that the pre-training is done by a VM and we pre-training is done by a VM and we pre-training is done by a VM and we simply graph an action head on top of simply graph an action head on top of simply graph an action head on top of it. But really if you think about these it. But really if you think about these it. But really if you think about these models they are LVAS because the most models they are LVAS because the most models they are LVAS because the most amount of parameters are dedicated to amount of parameters are dedicated to amount of parameters are dedicated to language. So language is first first language. So language is first first language. So language is first first class citizen followed by vision and class citizen followed by vision and class citizen followed by vision and action. And by design VAS are great at action. And by design VAS are great at action. And by design VAS are great at encoding knowledge and nouns but not so encoding knowledge and nouns but not so encoding knowledge and nouns but not so much at physics and verbs. It's kind of much at physics and verbs. It's kind of much at physics and verbs. It's kind of head heavy in the wrong places. head heavy in the wrong places. head heavy in the wrong places. This is my favorite example from the This is my favorite example from the This is my favorite example from the original VA paper. Move the coke can to original VA paper. Move the coke can to original VA paper. Move the coke can to a picture of Taylor Swift. Yes, it has a picture of Taylor Swift. Yes, it has a picture of Taylor Swift. Yes, it has not seen Taylor Swift before. Yes, it's not seen Taylor Swift before. Yes, it's not seen Taylor Swift before. Yes, it's able to generalize. But this is not able to generalize. But this is not able to generalize. But this is not quite the pre-training ability that quite the pre-training ability that quite the pre-training ability that we're looking for. So what's the second we're looking for. So what's the second we're looking for. So what's the second pre-training paradigm? And I always pre-training paradigm? And I always pre-training paradigm? And I always thought that it would be something thought that it would be something thought that it would be something glorious. glorious. glorious. Unfortunate, it turns out that this is Unfortunate, it turns out that this is Unfortunate, it turns out that this is AI video slop that we call. You know, I AI video slop that we call. You know, I AI video slop that we call. You know, I can watch these um cats playing banjo on can watch these um cats playing banjo on can watch these um cats playing banjo on security cam all day. It's peak security cam all day. It's peak security cam all day. It's peak internet. internet. internet. But really, look at this. No one can But really, look at this. No one can But really, look at this. No one can take this seriously [laughter] until we take this seriously [laughter] until we take this seriously [laughter] until we realize that these video models are

realize that these video models are realize that these video models are learning to simulate next world state learning to simulate next world state learning to simulate next world state internally. So these are some rods from internally. So these are some rods from internally. So these are some rods from V3. You can see that the models they V3. You can see that the models they V3. You can see that the models they pick up gravity, buoyancy, lighting, pick up gravity, buoyancy, lighting, pick up gravity, buoyancy, lighting, reflection, refraction all by reflection, refraction all by reflection, refraction all by themselves. None of this is coded in themselves. None of this is coded in themselves. None of this is coded in physics emerge by predicting the next physics emerge by predicting the next physics emerge by predicting the next blob of pixels at scale. blob of pixels at scale. blob of pixels at scale. And even visual planning emerges. Look And even visual planning emerges. Look And even visual planning emerges. Look at how VO solves these mazes. It solves at how VO solves these mazes. It solves at how VO solves these mazes. It solves them by running simulation forward in them by running simulation forward in them by running simulation forward in pixel space. And draw attention to the pixel space. And draw attention to the pixel space. And draw attention to the lower right corner here. This is my lower right corner here. This is my lower right corner here. This is my favorite example. Let's watch. And you favorite example. Let's watch. And you favorite example. Let's watch. And you blink if you miss. How V3 solves this blink if you miss. How V3 solves this blink if you miss. How V3 solves this one. [laughter] one. [laughter] It's super smart. You know, V3 figures It's super smart. You know, V3 figures out that if you're not looking, geometry out that if you're not looking, geometry out that if you're not looking, geometry is optional. is optional. is optional. I call this physics slop. I call this physics slop. I call this physics slop. So how do we make these world models So how do we make these world models So how do we make these world models useful? Well, we do action fine-tuning. useful? Well, we do action fine-tuning. useful? Well, we do action fine-tuning. We align this superp position of all We align this superp position of all We align this superp position of all possible future states and [snorts] possible future states and [snorts] possible future states and [snorts] collapse that onto a thing size that collapse that onto a thing size that collapse that onto a thing size that matters for real robots. matters for real robots. matters for real robots. Introducing dream zero. is a new type of

Introducing dream zero. is a new type of Introducing dream zero. is a new type of policy model that dreams a couple of policy model that dreams a couple of policy model that dreams a couple of seconds into the future and acts seconds into the future and acts seconds into the future and acts accordingly. accordingly. accordingly. And you know that motor actions there And you know that motor actions there And you know that motor actions there are highdimensional continuous signals. are highdimensional continuous signals. are highdimensional continuous signals. So that looks just like pixels. We can So that looks just like pixels. We can So that looks just like pixels. We can render it at the same time as we render render it at the same time as we render render it at the same time as we render the videos. So dream zero zero jointly the videos. So dream zero zero jointly the videos. So dream zero zero jointly decodes the next w states and next decodes the next w states and next decodes the next w states and next actions and as a result it's able to actions and as a result it's able to actions and as a result it's able to zeroot solve task and verbs that it has zeroot solve task and verbs that it has zeroot solve task and verbs that it has never seen in training never seen in training never seen in training and as a robot executes we can visualize and as a robot executes we can visualize and as a robot executes we can visualize what it's dreaming about and the what it's dreaming about and the what it's dreaming about and the correlation is very tight. If the video correlation is very tight. If the video correlation is very tight. If the video prediction works the action works if the prediction works the action works if the prediction works the action works if the video hallucinates the action fails. So video hallucinates the action fails. So video hallucinates the action fails. So once again vision and action are now once again vision and action are now once again vision and action are now first class citizens. first class citizens. first class citizens. And we have a lot of fun with Dream And we have a lot of fun with Dream And we have a lot of fun with Dream Zero. So we just wrote a robot around um Zero. So we just wrote a robot around um Zero. So we just wrote a robot around um in our lab and then type random things in our lab and then type random things in our lab and then type random things into the prompt box. And of course Dream into the prompt box. And of course Dream into the prompt box. And of course Dream Zero is not going to get all of these Zero is not going to get all of these Zero is not going to get all of these tasks 100% robust, but it's kind of like tasks 100% robust, but it's kind of like tasks 100% robust, but it's kind of like GPU 2. It's trying to get the shape of GPU 2. It's trying to get the shape of GPU 2. It's trying to get the shape of the motion correct in every case. So the motion correct in every case. So the motion correct in every case. So Dream Zero is our first step towards Dream Zero is our first step towards Dream Zero is our first step towards open-ended open vocabulary prompting for open-ended open vocabulary prompting for open-ended open vocabulary prompting for robotics. robotics. robotics. And we [snorts] call this new type of And we [snorts] call this new type of And we [snorts] call this new type of model world action models or wh. model world action models or wh. model world action models or wh. So let's all take a moment of silence So let's all take a moment of silence So let's all take a moment of silence for our dear friend VAS. for our dear friend VAS. for our dear friend VAS. They've served us well. Rest in peace. They've served us well. Rest in peace. They've served us well. Rest in peace. Long live world action models.

Long live world action models. Long live world action models. [clears throat] [clears throat] [clears throat] And next data strategy. And next data strategy. And next data strategy. This is Nvidia's chief scientist Bill This is Nvidia's chief scientist Bill This is Nvidia's chief scientist Bill Dali operating teley operation inside Dali operating teley operation inside Dali operating teley operation inside our lab. And given his salary, I think our lab. And given his salary, I think our lab. And given his salary, I think this is by far the most expensive teleop this is by far the most expensive teleop this is by far the most expensive teleop trajectory ever collected in our data trajectory ever collected in our data trajectory ever collected in our data set. set. set. The past three years have been dominated The past three years have been dominated The past three years have been dominated by teley operation. It's the golden era, by teley operation. It's the golden era, by teley operation. It's the golden era, right? VR headsets, right? VR headsets, right? VR headsets, extremely optimized extremely optimized extremely optimized latency for streaming and these complex latency for streaming and these complex latency for streaming and these complex rigs that look like medieval torture rigs that look like medieval torture rigs that look like medieval torture devices, you know, so much investment in devices, you know, so much investment in devices, you know, so much investment in industry, so much pain and suffering. industry, so much pain and suffering. industry, so much pain and suffering. And yet for teleyop it's upperbounded by And yet for teleyop it's upperbounded by And yet for teleyop it's upperbounded by 24 hours per robot per day. The 24 hours per robot per day. The 24 hours per robot per day. The fundamental physical limit. And actually fundamental physical limit. And actually fundamental physical limit. And actually who am I kidding? It's more like three who am I kidding? It's more like three who am I kidding? It's more like three hours per robot per day and only when hours per robot per day and only when hours per robot per day and only when the robot god is merciful because they the robot god is merciful because they the robot god is merciful because they throw tantrums all the time. So how can throw tantrums all the time. So how can throw tantrums all the time. So how can we do better? Well, how about this? You

we do better? Well, how about this? You we do better? Well, how about this? You just wear the robot hand on your own just wear the robot hand on your own just wear the robot hand on your own hand. hand. hand. So this is called UMI or universal So this is called UMI or universal So this is called UMI or universal manipulation interface and it's a manipulation interface and it's a manipulation interface and it's a deceptively simple idea. You wear the deceptively simple idea. You wear the deceptively simple idea. You wear the robot actuator on your hand and directly robot actuator on your hand and directly robot actuator on your hand and directly collect the data as humans while getting collect the data as humans while getting collect the data as humans while getting the rest of the robot body out of the the rest of the robot body out of the the rest of the robot body out of the loop. Yet I would say UMI is perhaps one loop. Yet I would say UMI is perhaps one loop. Yet I would say UMI is perhaps one of the greatest papers ever written in of the greatest papers ever written in of the greatest papers ever written in robotics data and it spawned two unicorn robotics data and it spawned two unicorn robotics data and it spawned two unicorn startups. On left hand side is startups. On left hand side is startups. On left hand side is generalist improving this design. So you generalist improving this design. So you generalist improving this design. So you can wear the gripper here and on right can wear the gripper here and on right can wear the gripper here and on right hand side Sunday made these three-finger hand side Sunday made these three-finger hand side Sunday made these three-finger data gloves. So last year we took it one data gloves. So last year we took it one data gloves. So last year we took it one step further. We designed this step further. We designed this step further. We designed this exoskeleton that has a onetoone mapping exoskeleton that has a onetoone mapping exoskeleton that has a onetoone mapping with five finger dextrous robot hands with five finger dextrous robot hands with five finger dextrous robot hands and we call it dex ooi. Let's look at it and we call it dex ooi. Let's look at it and we call it dex ooi. Let's look at it in action. On the left the human in action. On the left the human in action. On the left the human directly collecting data always the directly collecting data always the directly collecting data always the fastest. on the right. Look at how fastest. on the right. Look at how fastest. on the right. Look at how difficult teleop is right the human difficult teleop is right the human difficult teleop is right the human operator here one of our most skilled operator here one of our most skilled operator here one of our most skilled PhDs he has to align very carefully PhDs he has to align very carefully PhDs he has to align very carefully right and then it's super slow also the right and then it's super slow also the right and then it's super slow also the success rate is very low as well and in success rate is very low as well and in success rate is very low as well and in the middle you just wear these the middle you just wear these the middle you just wear these exoskeleton and you collect data exoskeleton and you collect data exoskeleton and you collect data directly directly directly and we train a robot policy on this data and we train a robot policy on this data and we train a robot policy on this data so here what you see is a fully so here what you see is a fully so here what you see is a fully autonomous autonomous autonomous row out of a policy that's trained on row out of a policy that's trained on row out of a policy that's trained on zero t operation data zero t operation data zero t operation data So, we're able to break the curse of 24 So, we're able to break the curse of 24 So, we're able to break the curse of 24 hours per robot per day and see how hours per robot per day and see how hours per robot per day and see how happy these robots are because they no happy these robots are because they no happy these robots are because they no longer need to be in the loop for data longer need to be in the loop for data longer need to be in the loop for data collection. So, is this the answer? Have collection. So, is this the answer? Have collection. So, is this the answer? Have we solved scaling for robotics? we solved scaling for robotics? we solved scaling for robotics? Anyone driving Tesla or Whimo here? Anyone driving Tesla or Whimo here? Anyone driving Tesla or Whimo here? Anyone? Right. You know, when you're Anyone? Right. You know, when you're Anyone? Right. You know, when you're driving, you're actually contributing to driving, you're actually contributing to driving, you're actually contributing to the biggest physical data flywheel. the biggest physical data flywheel. the biggest physical data flywheel. And the beauty is you don't even feel it And the beauty is you don't even feel it And the beauty is you don't even feel it during FSD because the data upload is an during FSD because the data upload is an during FSD because the data upload is an ambient process. Yet wearing these UMI ambient process. Yet wearing these UMI ambient process. Yet wearing these UMI or data wearables, it's still or data wearables, it's still or data wearables, it's still cumbersome, right? It's intrusive. It's cumbersome, right? It's intrusive. It's cumbersome, right? It's intrusive. It's not as seamless as just driving to work. not as seamless as just driving to work. not as seamless as just driving to work. So we need an FSD equivalent. So we need an FSD equivalent. So we need an FSD equivalent. The data collection needs to get out of The data collection needs to get out of The data collection needs to get out of the way, fade into the background so we the way, fade into the background so we the way, fade into the background so we can capture the full glory of human can capture the full glory of human can capture the full glory of human dexterity across all walks of lives, dexterity across all walks of lives, dexterity across all walks of lives, across all labors of economic value. across all labors of economic value. across all labors of economic value. So we are going all in on human So we are going all in on human So we are going all in on human egocentric videos that come with these egocentric videos that come with these egocentric videos that come with these detailed annotations like hand position detailed annotations like hand position detailed annotations like hand position tracking and dense language annotations. tracking and dense language annotations. tracking and dense language annotations. introducing ego scale where 99.9% of the

introducing ego scale where 99.9% of the introducing ego scale where 99.9% of the training that goes into this is based on training that goes into this is based on training that goes into this is based on human egocentric videos and the result human egocentric videos and the result human egocentric videos and the result is an end to end policy that maps is an end to end policy that maps is an end to end policy that maps directly from the camera pixels here to directly from the camera pixels here to directly from the camera pixels here to 22 degrees of freedom high dexterity 22 degrees of freedom high dexterity 22 degrees of freedom high dexterity robot hands which you see here is fully robot hands which you see here is fully robot hands which you see here is fully autonomous. autonomous. autonomous. We pre-train eagle scale on 21k hours of We pre-train eagle scale on 21k hours of We pre-train eagle scale on 21k hours of in the wild egocentric human data with in the wild egocentric human data with in the wild egocentric human data with zero robot data whatsoever. And during zero robot data whatsoever. And during zero robot data whatsoever. And during pre-training we predict these hand pre-training we predict these hand pre-training we predict these hand joints and wrist poses. Then in action joints and wrist poses. Then in action joints and wrist poses. Then in action fine-tuning we collect only 50 hours of fine-tuning we collect only 50 hours of fine-tuning we collect only 50 hours of high precision moap data gloves and four high precision moap data gloves and four high precision moap data gloves and four hours of teleyop. That's four hours of hours of teleyop. That's four hours of hours of teleyop. That's four hours of teleyop less than 0.1% of our training teleyop less than 0.1% of our training teleyop less than 0.1% of our training mix. mix. mix. And with this ego scale is able to And with this ego scale is able to And with this ego scale is able to generalize to these very dextrous tasks generalize to these very dextrous tasks generalize to these very dextrous tasks like sorting card or manipulating like sorting card or manipulating like sorting card or manipulating syringe right over transferring the syringe right over transferring the syringe right over transferring the liquid. You know someday we might have liquid. You know someday we might have liquid. You know someday we might have robot nurses at home. Might as well try robot nurses at home. Might as well try robot nurses at home. Might as well try this. And for these tasks it takes only this. And for these tasks it takes only this. And for these tasks it takes only one shot demonstration at test time to one shot demonstration at test time to one shot demonstration at test time to learn different shirt folding learn different shirt folding learn different shirt folding strategies. strategies. strategies. And perhaps the most fascinating finding

And perhaps the most fascinating finding And perhaps the most fascinating finding from the paper is that we discovered from the paper is that we discovered from the paper is that we discovered this neuroscaling law for dexterity. this neuroscaling law for dexterity. this neuroscaling law for dexterity. It's a very clean relationship between It's a very clean relationship between It's a very clean relationship between the amount of hours we put into the amount of hours we put into the amount of hours we put into pre-training and the optimal validation pre-training and the optimal validation pre-training and the optimal validation loss. In fact, it's a clean log linear loss. In fact, it's a clean log linear loss. In fact, it's a clean log linear mathematical equation six years after mathematical equation six years after mathematical equation six years after the original neuroscaling law for the original neuroscaling law for the original neuroscaling law for language models. language models. language models. So if we put all of these data So if we put all of these data So if we put all of these data strategies on this chart, X-axis is strategies on this chart, X-axis is strategies on this chart, X-axis is alignment to the robot hardware. Y-axis alignment to the robot hardware. Y-axis alignment to the robot hardware. Y-axis is scalability. This is what it looks is scalability. This is what it looks is scalability. This is what it looks like. Teleyop the least scalable data like. Teleyop the least scalable data like. Teleyop the least scalable data wearables. You can go up to hundreds of wearables. You can go up to hundreds of wearables. You can go up to hundreds of thousands of hours. And egocentric video thousands of hours. And egocentric video thousands of hours. And egocentric video if we're able to spin the FSD flywheel if we're able to spin the FSD flywheel if we're able to spin the FSD flywheel easily 10 million hours in the next year easily 10 million hours in the next year easily 10 million hours in the next year or so. And if we draw a line here, or so. And if we draw a line here, or so. And if we draw a line here, everything to the left of this line is a everything to the left of this line is a everything to the left of this line is a new paradigm sensorized human data. So new paradigm sensorized human data. So new paradigm sensorized human data. So let me make a few predictions. In the let me make a few predictions. In the let me make a few predictions. In the next year or two, we'll see teleyop next year or two, we'll see teleyop next year or two, we'll see teleyop dropping and dropping to almost dropping and dropping to almost dropping and dropping to almost negligible amount. And then there will negligible amount. And then there will negligible amount. And then there will be an ensemble of data wearables custom be an ensemble of data wearables custom be an ensemble of data wearables custom designed for different hardware and use designed for different hardware and use designed for different hardware and use cases. And finally, the main diet for cases. And finally, the main diet for cases. And finally, the main diet for robotics will be egocentric videos. robotics will be egocentric videos. robotics will be egocentric videos. So, a moment of silence for our dear So, a moment of silence for our dear So, a moment of silence for our dear friend Tal. You have served us well. friend Tal. You have served us well. friend Tal. You have served us well. Rest in peace. Long live sensorized Rest in peace. Long live sensorized Rest in peace. Long live sensorized human data. human data. human data. Are we done with the data strategy yet? Are we done with the data strategy yet? Are we done with the data strategy yet? Did you notice I put two rings on data Did you notice I put two rings on data Did you notice I put two rings on data strategy? What's the outer ring here? strategy? What's the outer ring here? strategy? What's the outer ring here? All the OM frontier labs have spent

All the OM frontier labs have spent All the OM frontier labs have spent significant budget now on acquiring significant budget now on acquiring significant budget now on acquiring millions of coding environments to do millions of coding environments to do millions of coding environments to do reinforcement learning. So, robotics is reinforcement learning. So, robotics is reinforcement learning. So, robotics is the same. were in urgent need to scale the same. were in urgent need to scale the same. were in urgent need to scale up environments. up environments. up environments. And of course, you can always do And of course, you can always do And of course, you can always do reinforcement learning directly on the reinforcement learning directly on the reinforcement learning directly on the real robot. So in our lab, we use RL to real robot. So in our lab, we use RL to real robot. So in our lab, we use RL to push certain tasks to almost 100% push certain tasks to almost 100% push certain tasks to almost 100% success rate so you can do these success rate so you can do these success rate so you can do these continuous execution for hours on end. continuous execution for hours on end. continuous execution for hours on end. You know, it's kind of therapeutic to You know, it's kind of therapeutic to You know, it's kind of therapeutic to see these robots assembling GPUs just by see these robots assembling GPUs just by see these robots assembling GPUs just by themselves or as a wise man will say, themselves or as a wise man will say, themselves or as a wise man will say, "Good boy, this task has been approved "Good boy, this task has been approved "Good boy, this task has been approved by my boss." by my boss." by my boss." Yet we can't [snorts] get to one million Yet we can't [snorts] get to one million Yet we can't [snorts] get to one million environments because that will require environments because that will require environments because that will require one million robots if you do it the one million robots if you do it the one million robots if you do it the previous way. So we need a better way previous way. So we need a better way previous way. So we need a better way here. Let's say you take an iPhone here. Let's say you take an iPhone here. Let's say you take an iPhone picture and you can pass it through this picture and you can pass it through this picture and you can pass it through this 3D wall scan pipeline to extract all the 3D wall scan pipeline to extract all the 3D wall scan pipeline to extract all the objects and then automatically objects and then automatically objects and then automatically synthesize them again inside a classical synthesize them again inside a classical synthesize them again inside a classical physics simulator. So all these objects physics simulator. So all these objects physics simulator. So all these objects are actually interactive after the scan are actually interactive after the scan are actually interactive after the scan and then you can augment this infinitely and then you can augment this infinitely and then you can augment this infinitely in simulation with variations that we in simulation with variations that we in simulation with variations that we call digital cousins. call digital cousins. call digital cousins. So now iPhone basically become a pocket So now iPhone basically become a pocket So now iPhone basically become a pocket world scanner. In this process that we world scanner. In this process that we world scanner. In this process that we call real to sim to real and in this way call real to sim to real and in this way call real to sim to real and in this way we have a scalable way to port the we have a scalable way to port the we have a scalable way to port the physical world into the digital world. physical world into the digital world. physical world into the digital world. But still this method relies on a But still this method relies on a But still this method relies on a classical graphics engine. Can we do classical graphics engine. Can we do classical graphics engine. Can we do better? Introducing Dream Dojo.

better? Introducing Dream Dojo. better? Introducing Dream Dojo. So, it's always been on video world So, it's always been on video world So, it's always been on video world model and turning them into full-fledged model and turning them into full-fledged model and turning them into full-fledged neural simulators. Dream Dojo takes as neural simulators. Dream Dojo takes as neural simulators. Dream Dojo takes as input these continuous action signals input these continuous action signals input these continuous action signals and outputs the next RGB frames as well and outputs the next RGB frames as well and outputs the next RGB frames as well as sensor states in real time. Not a as sensor states in real time. Not a as sensor states in real time. Not a single pixel you see here is real. And single pixel you see here is real. And single pixel you see here is real. And Dream Dojo is able to capture and learn Dream Dojo is able to capture and learn Dream Dojo is able to capture and learn the mechanics of different robots the mechanics of different robots the mechanics of different robots through a purely datadriven approach. through a purely datadriven approach. through a purely datadriven approach. There's no physics equation, no graphics There's no physics equation, no graphics There's no physics equation, no graphics engine involved in this process. engine involved in this process. engine involved in this process. So the new post- training paradigm for So the new post- training paradigm for So the new post- training paradigm for robotics is a massively parallel RO robotics is a massively parallel RO robotics is a massively parallel RO system that runs on a few real robot system that runs on a few real robot system that runs on a few real robot stations on a bunch of graphics cores stations on a bunch of graphics cores stations on a bunch of graphics cores running world scans and heavy inference running world scans and heavy inference running world scans and heavy inference compute running world models. compute running world models. compute running world models. Or as this equation goes, compute now Or as this equation goes, compute now Or as this equation goes, compute now equals environment now equals data. Or equals environment now equals data. Or equals environment now equals data. Or as a wise man would say, the more you as a wise man would say, the more you as a wise man would say, the more you buy, the more you save. And this message buy, the more you save. And this message buy, the more you save. And this message has been approved by my boss. has been approved by my boss. has been approved by my boss. So that's it. Putting it together, the So that's it. Putting it together, the So that's it. Putting it together, the great parallel that robotics will great parallel that robotics will great parallel that robotics will follow. And it's happening as we speak. follow. And it's happening as we speak. follow. And it's happening as we speak. And we're looking at the beginning of And we're looking at the beginning of And we're looking at the beginning of the endgame. the endgame. the endgame. You guys play the video game You guys play the video game You guys play the video game Civilization. Civilization. Civilization. Still my favorite. I like to think of my Still my favorite. I like to think of my Still my favorite. I like to think of my research as unlocking game achievements research as unlocking game achievements research as unlocking game achievements on this civilizational technology tree. on this civilizational technology tree. on this civilizational technology tree. >> [clears throat] >> [clears throat] >> [clears throat] >> And there are three more achievements to

>> And there are three more achievements to >> And there are three more achievements to unlock for robotics and then we're done. unlock for robotics and then we're done. unlock for robotics and then we're done. I can retire and I can't wait for that. I can retire and I can't wait for that. I can retire and I can't wait for that. The first is passing the physical The first is passing the physical The first is passing the physical touring test. Across a wide range of touring test. Across a wide range of touring test. Across a wide range of activities, you cannot tell the activities, you cannot tell the activities, you cannot tell the difference between a human doing the difference between a human doing the difference between a human doing the task or a robot doing it. Maybe not task or a robot doing it. Maybe not task or a robot doing it. Maybe not drunk humans, but you know, physical drunk humans, but you know, physical drunk humans, but you know, physical touring test is about unit energy in and touring test is about unit energy in and touring test is about unit energy in and unit labor out. And just by judging at unit labor out. And just by judging at unit labor out. And just by judging at the sexy pose of this robot, I think the the sexy pose of this robot, I think the the sexy pose of this robot, I think the work is cut out for us. So maybe it's work is cut out for us. So maybe it's work is cut out for us. So maybe it's two to three years away. two to three years away. two to three years away. And next physical API, you have a whole And next physical API, you have a whole And next physical API, you have a whole fleet of robots and they can be fleet of robots and they can be fleet of robots and they can be configured just like any other software configured just like any other software configured just like any other software using APIs and command lines using APIs and command lines using APIs and command lines orchestrated someday by Opus 9.0. Z and orchestrated someday by Opus 9.0. Z and orchestrated someday by Opus 9.0. Z and if we have this physical API we'll be if we have this physical API we'll be if we have this physical API we'll be able to realize lysot factories those able to realize lysot factories those able to realize lysot factories those are essentially printers of atoms they are essentially printers of atoms they are essentially printers of atoms they take as input design in markdown files take as input design in markdown files take as input design in markdown files and then output fully assembled products and then output fully assembled products and then output fully assembled products completely autonomous or these wet labs completely autonomous or these wet labs completely autonomous or these wet labs that automate scientific discoveries in that automate scientific discoveries in that automate scientific discoveries in chemistry biology and medicine chemistry biology and medicine chemistry biology and medicine and the final stop physical auto and the final stop physical auto and the final stop physical auto research when robots research when robots research when robots start to design, improve, and build the start to design, improve, and build the start to design, improve, and build the next iteration of themselves far beyond next iteration of themselves far beyond next iteration of themselves far beyond what's humanly possible. what's humanly possible. what's humanly possible. So, you might ask, is this too science

So, you might ask, is this too science So, you might ask, is this too science fiction? Like, are we going to see this fiction? Like, are we going to see this fiction? Like, are we going to see this in our lifetime? in our lifetime? in our lifetime? Well, it took the AI community 14 years Well, it took the AI community 14 years Well, it took the AI community 14 years to go from the first forward pass of to go from the first forward pass of to go from the first forward pass of Alexet in 2012, a model that barely Alexet in 2012, a model that barely Alexet in 2012, a model that barely recognized cat versus dog to AI ascent recognized cat versus dog to AI ascent recognized cat versus dog to AI ascent today 2026. Well, we talk about agentic today 2026. Well, we talk about agentic today 2026. Well, we talk about agentic auto research auto research auto research and let's just add another 14 years. How and let's just add another 14 years. How and let's just add another 14 years. How about that? about that? about that? 2026 is right in the middle of 2012 and 2026 is right in the middle of 2012 and 2026 is right in the middle of 2012 and 2040. And technology does not advance 2040. And technology does not advance 2040. And technology does not advance linearly. It advances exponentially. linearly. It advances exponentially. linearly. It advances exponentially. So, [snorts] I can say with 95% So, [snorts] I can say with 95% So, [snorts] I can say with 95% certainty that we'll get to the end of certainty that we'll get to the end of certainty that we'll get to the end of the endgame, the end of the technology the endgame, the end of the technology the endgame, the end of the technology tree by 2040. tree by 2040. tree by 2040. And we'll still be all we'll still be And we'll still be all we'll still be And we'll still be all we'll still be on. on. on. If you believe in robotics, robotics

If you believe in robotics, robotics If you believe in robotics, robotics will believe in you. And to all of us will believe in you. And to all of us will believe in you. And to all of us here sitting here, I think our here sitting here, I think our here sitting here, I think our generation was born too late to explore generation was born too late to explore generation was born too late to explore the earth and too early to explore the the earth and too early to explore the the earth and too early to explore the stars. But we are born just in time to stars. But we are born just in time to stars. But we are born just in time to solve robotics. solve robotics. solve robotics. [applause]