Rivian’s AI Master Plan: Hands-Free Driving, R2 + LiDAR, and the Road to Personal Level 4 Autonomy
The InEVitable
Rivian’s AI Master Plan: Hands-Free Driving, R2 + LiDAR, and the Road to Personal Level 4 Autonomy
Annotations will appear as you listen
0:00
68:50
Term
GPUs
GPUs are powerful computer chips that help process graphics and data quickly. In cars, they help with things like self-driving technology and advanced safety features.
Term
microchips
Microchips are tiny pieces of technology that help cars work properly. They control things like music systems and safety features in the car.
Term
sensors
Sensors are tools that help cars understand what's around them, like other cars or obstacles. They help with safety features and can even help the car drive itself.
Car
Rivian R2
The Rivian R2 is a new electric SUV that Rivian is planning to release. It's designed to be smaller than their previous models and will include modern tech for self-driving capabilities.
Term
lidar
Lidar is a technology that helps cars see their surroundings by using lasers to measure distances. It's important for self-driving cars to navigate safely.
Term
autonomy
Autonomy in cars means that the vehicle can drive itself without needing a person to control it. This is done using special technology that helps the car understand its surroundings.
Term
AI
AI stands for artificial intelligence, which is technology that allows computers and machines to think and learn like humans. In cars, it helps them make decisions and navigate on their own.
Car
Rivian R1S
The Rivian R1S is a new electric SUV designed for adventure and outdoor activities. The 'Gen 2' means it's the second version of this model, which has better features than the first version.
Car
Proton Gen2
The Proton Gen-2 is a small car made in Malaysia that is usually cheaper than many other cars. It's designed to be practical and easy to drive, making it a good choice for people who want something affordable. It's important because it shows how local companies can make cars that compete with bigger brands.
Car
Cadillac Escalade
The Cadillac Escalade is a big, fancy SUV that lots of people use for luxury and comfort. It's known for having a lot of space inside and nice features, making it a popular choice for families or those who want to travel in style. The new Escalade IQ is an updated version that will include electric options.
Term
Tesla Full Self-Driving
Tesla Full Self-Driving is a system that helps Tesla cars drive themselves. It can change lanes, recognize traffic lights, and follow highways without much help from the driver.
Concept
ground truth
Ground truth is basically the real and accurate information about what's around a car. It's important for things like self-driving cars to know where everything is so they can drive safely.
Term
world model
A world model is like a detailed map that a car uses to understand everything around it. It helps the car know where things are so it can drive safely and avoid accidents.
Brand
Ford
Ford is a well-known car company from the United States that makes many types of vehicles, like trucks and cars. It was started by Henry Ford over a hundred years ago.
Brand
Daimler
Daimler is a famous car company from Germany that makes luxury cars, including Mercedes-Benz. They have been around for a long time and were involved in creating some of the first self-driving cars.
Concept
autonomous vehicles
Autonomous vehicles are cars that can drive themselves without needing a person to control them. They use special technology to see and understand their surroundings, making it possible for them to navigate roads safely.
Brand
Volvo
Volvo is a car company from Sweden that is famous for making safe cars and working on new technologies like self-driving features.
Term
L4
L4 is a level of self-driving technology where the car can drive itself without any help from a person, but only in certain areas or situations.
Term
Waymo
Waymo is a company that works on self-driving cars. They are known for creating technology that allows cars to drive themselves safely.
Car
Tesla Model Y
The Tesla Model Y is a type of electric car that looks like a small SUV. It's important because it can go a long distance without needing to be charged often, and it has cool tech features like autopilot. Many people like it because it's good for the environment and has a lot of space inside.
Term
sensor arrays
Sensor arrays are groups of sensors that help the car see and understand what's around it. They can detect things like other cars, pedestrians, and obstacles to help the car drive safely.
Term
level four
Level four means the car can drive itself without needing a person to take control in certain situations. It can handle driving all on its own in specific places like cities or highways.
Term
prognostics
Prognostics is like a health check for your car. It helps predict when parts might fail so you can fix them before they cause bigger problems.
Part
12 volt battery
A 12 volt battery is what powers your car's electrical systems, like lights and radio. Keeping an eye on its voltage can help you know when it needs to be replaced.
Term
proactive maintenance
Proactive maintenance means fixing things on your car before they break down. It helps keep your car running smoothly and prevents bigger problems later.
Concept
CASE
CASE is a term that describes a new way cars are being made and used. It includes features like being able to connect to the internet, driving themselves, being shared with others, and running on electricity.
Term
EVs
EVs stands for electric vehicles, which are cars that run on electricity instead of gas. They are better for the environment and can save you money on fuel.
Term
EV
An EV, or electric vehicle, runs on electricity instead of gas. This means they don't produce exhaust fumes like regular cars do, making them better for the environment.
Term
ICE
ICE means internal combustion engine, which is the type of engine found in most cars that run on gasoline or diesel. These engines burn fuel to create power, which produces exhaust fumes.
Term
L2
L2 is a level of self-driving technology where the car can help with some driving tasks, but the driver still has to pay attention and be in control. If something goes wrong, the driver is responsible, not the car manufacturer.
Concept
autonomous cars
Autonomous cars are vehicles that can drive themselves without needing a person to control them. They use special technology to see and understand their surroundings.
Brand
BMW
BMW is a well-known car company from Germany that makes luxury cars. They are famous for their high-quality engineering and advanced technology.
Term
Transamerica Trail
The Transamerica Trail is a long path that goes across the United States. It's meant for people who like to drive on rough roads and explore different places.
LIVE
Hi there and welcome to The Inevitable. This is Motor Trends Podcast, our podcast about
the future of the car, the future of the automobile, the future of autonomy and AI at a little
California-based, sort of California-based startup called Rivian. You might have heard
of them.
I hate the fact that their genius CEO is younger than me. It really just is really…
And he looks like Superman. He's taller than you. He's better looking than you.
Yeah, it's just…
Yeah, he's smarter than you.
He's worth a billion dollars or more. It's just endlessly depressing. However, and I
went up there for something they called autonomy and AI day and we learned a lot. We actually,
like, I didn't think we'd learned a lot. We learned a lot. And before we get to our
interview, Ed has a message just for you.
Yes, the Inevitable podcast is brought to you by nobody currently. I'm going to ring
the bell.
Yeah, yeah, do it.
There you go.
So, if you want to sponsor, shoot us a note, Edward.loh.herst.com or slide into our DMs. Indeed,
we are fresh back from Rivian's autonomy and AI day. It was a full, basically, half-day devoted
to everything they're doing within the space. We saw microchips, GPUs, we saw sensors, we
saw where the lidar is going to be positioned on the all-new R2 smaller Rivian SUV.
We took a drive with a robot driving a car.
We went to a hands-free, hands-free drive around Rivian's campus in Palo Alto, California.
But like a big 19-mile kind of drive around.
And then at the end of the day, we had a chance to sit down with James Philbin, who is their
vice president of autonomy and AI at Rivian. This guy, SuperSmart, he was very casual and
modest about saying that he's actually Dr. James Philbin. He's a PhD.
Yeah, who built Waymo's.
Yes, he was previously at Waymo for a couple of years as director of software engineering.
And but prior to that, he was at Zoox for four years, senior director of perception
and computer vision. Prior to that, he spent a number of years at Google. He was a staff
software engineer, core computer vision, machine language algorithms, and image annotation and
understanding. So yeah, so once again, Ed and I are two dummies in a room with someone who's
probably too smart to talk to us. Yeah, very smart, very kind to answer all of our dumb questions.
Philbin worked on such products he would know as Google goggles, Google image search,
and Google photo search. So this is a guy who really knows perception and imaging and is now
applying it to what they're doing, what Rivian is doing with autonomy and AI. It's a fascinating
conversation. And we're interrupted halfway through with a special guest. There's a special
and actually watching and listening because our guests kind of barged in.
Yeah, and did very rudely, gave us some great information, but wasn't wearing a one of the
mics that we had. A lavalier. Yeah, so our producer, we got a directional mic on. Yeah,
I think Brent figured it out. So if the audio levels are too bad, but it's a great conversation,
wide ranging. So let's go right to it. James Philbin, VP of autonomy at Rivian.
James Philbin, Vice President, autonomy and AI at Rivian. Thanks for coming on.
Yeah, thanks for having me. Today is apparently your day. It was it was Rivian's autonomy and AI
day. So why don't we start there? And can you just tell us what was the big, what was the,
what were the headlines from this big day? Because normally it's, it's been what Rivian
autonomy day. And now the AI piece was, is kind of the new, the new hotness. Is that right?
I don't know about that. So I think, you know, that's always been sort of part of
the vision. So I would say, you know, today, we had a few announcements, it's quite a lot to take
in to be there. So I think on the autonomy side, we announced universal hands free, which is our
big scaling up in the hands free system we launched actually earlier this year. So the current system
is enhanced highway assist that covers 135,000 miles of highway, you know, hands free highway
miles. And then universal hands free, which is what's coming out this month to our one gen two
customers scales at up to three and a half million. So it's a massive, massive growth.
And that was a huge map you showed of North America. Like what?
Yeah, it's crazy right.
US and Canada and like everything was green. Everything is right. Except for Wyoming. Yeah.
Universal hands free. Okay. And then
we talked through some of the roadmap for next year. So
big piece is a point to point system that we've been developing based on this very large driving
model. The point to point means that like I want to go to 711 from my house. So I say 711 in the
car without me doing anything drives to 711. That's a full driving task following the routes,
turning, you know, responding to traffic lights, stop signs, other traffic, the whole thing.
So yeah, we, we're not releasing that today. We were, we have been working on it and we
actually gave some demos here in Palo Alto sites. We took a demo ride.
You can tell me how it was. It worked. Here we are. Yeah. So that's been,
yeah, like really incredible progress we've seen there. And a large part of that is really
powered by this sort of data flywheel that we, we launched on actually our one gen two all the
way back last year. Right. So what that has done is essentially create this huge database of driving
scenarios from all of our customers. So we get this sort of incredibly rich kind of long tail
edge case scenarios streaming back into the cloud. That thing gets fed into the model and the model
can then learn those driving behaviors. So some of the drives today, it's kind of like the median
Rivian driver, right? That's kind of what it's learned in terms of how it can respond to things.
And then on the hardware sides, we announced, yeah, two, two important things. So one is
that R2 from next year, later next year, we'll have LiDAR and we showed the integration of
that, which looks amazing. So, and that's really a huge collaboration with the design studio and
all of the, you know, the headline design folks and everything. And then also R2 from next year
will have a new autonomy chip developed in-house. That's our Rivian autonomy processor. And we
went into some of the details of that chip. So we're in December, mid December of 25.
And I don't know if you've given a hard day, but when does the R2 launch?
Can you say even what quarter?
First half of next year.
First half of next year. And I know that LiDAR is an optional thing. So you would buy
a R2 that's equipped with LiDAR and that comes later after the launch. But the chipset, that dual
Rivian chipset that we saw, is that on every R2 or only the ones that are LiDAR equipped?
It will only be on the second phase of R2, which will be later next year. And every
second phase R2 will also have the LiDAR. So there'll be a kind of a flag day where they
switch over and now those R2s will have the new chip and the LiDAR.
The new chip said is Gen 3 autonomy.
Yes.
Gen 3 compute. So the R2 will launch with Gen 2 initially and then it'll move to,
you could, if you want, get the Gen 3.
Exactly. So, you know, it's a very capable system still and it will get all of the features that,
you know, R1, Gen 2 owners will have as well.
And we took a ride, just to be totally clear, we took a ride in a Rivian R1S generation 2.
Yes. Yeah, exactly. Yeah. And the demo rides you saw today, production vehicles, nothing
special about the sensors or the compute in them, exactly what every customer has in their own.
Right. So it will be capable of the hands off, point to point.
Yes, exactly.
Just trying to get the terminology.
Absolutely.
We got a lot of questions about LiDAR and their, but let's just quickly wrap on that.
The other nuts when it's on AI, right?
Yes. And then, you know, Waseem went into some of the details of the new Rivian Assistant,
which will be in every Gen 1 and Gen 2 vehicle and obviously R2.
And that has this sort of amazing integration throughout the stack.
So you can, you know, Waseem showed some demos, you know, integrated with Calendar,
where you can actually change events and then you can kind of have contextual queries.
So you can text someone and it will actually include context in that text.
Yeah, the example, it was a cool example was like, hey, we're going to this somewhere in San Francisco,
give me three restaurants nearby or give me a list of restaurants nearby.
And then the voice command was take those top three, text them to somebody else
and ask them which one they want to eat at, send. And that was, yeah, it's cool.
That's like, it's definitely a real world scenario.
So and then you also mentioned what I thought was interesting was that
RJ said it was going, you're moving from a software-defined vehicle to an AI-defined
vehicle. And I bring that up for a couple reasons. A, it's interesting. B was I quoted that,
I put it up on my Instagram post and a lot of people wrote back like, oh, everything's now AI.
I wish they'd stopped doing that. What does revision mean when you say an AI-defined vehicle?
Yes, I think it's more that the autonomy and AI are kind of infusing everything about the vehicle,
right? And so autonomy is obviously handling whole driving function and more and more of it.
And then AI can sort of provide that, the kind of the fabric for the vehicle,
so that different components that used to be kind of separate can actually interact together.
I think that's the way I would see it, sort of like almost like a connective tissue that ties
all of these systems together. And so AI allows you much more flexibility in terms of combining
the output of this system and that system and how it interacts. And then autonomy, of course,
is about handling more and more of the driving function. So we really see that trend happening
where autonomy is getting more and more powerful. And then on the AI side, it's breaking down these
sort of barriers between systems and allowing a much more fluid sort of user experience.
So I think one of the examples today was changing the seat-heat settings. And you didn't have to
say, you know, turn on the heat of a seat, you know, one, two, and three, you said, like,
turn it on for everyone but me, right? So it's a very contextual kind of query. And then the results
are super useful. Let's take it from the top and go back to what you were saying. So your
current system of E8, EHA. So enhanced highway assist.
Enhanced highway assist has 135,000 miles of mapped roadways. And the new one, which you're calling
Universal Hands Free. Universal Hands Free. I heard some guys talking about it. Are you,
what's the internal, is it just UHF or is it RUF? So we've shown it to kind of UHF,
but yeah. Universal Hands Free. I've been updating our team at Motor Trend during the
press conference in our technical editor. Eric Tingwell was like, where did they get the 3.5
million miles from? Where did, is that from the flywheel? From the, or no?
Yeah, no. So actually, so it's a good question. So the system basically works on any roads with
well marked lane lines. And so we were like, actually, how many roads do have well marked
lane lines? So we didn't get a precise answer. We know that it's more than 3.5 million. So that's
why we said more than 3.5 million. And we got that from like some government statistics and
some other things. It may be quite a bit more, but we didn't want to say that without knowing it.
So that's why we ended up with the 3.5 million mile bigger.
So UHF basically works on any, any paved road.
Marked, marked. So there should be, you know, lane lines that you can see.
Lines or line? Does it need both? No, it doesn't need both, but it needs at least one.
So in other words, like a one mile paved driveway to somebody's cabin, it's not going to work on that.
Yeah. And actually, you know, so we can obviously test it in those conditions and the system mostly
actually works fine in those cases, but we want to build that confidence before we, you know,
it won't stop. It won't read like the ending of the asphalt or bitumen to the, to the dirt
shoulder as a line. So we, we actually predict road edge that would be a road edge. And we also
predict something called a virtual lane center. And that's actually how the system works internally.
So it tries to estimate, you know, here is where a driver would actually position.
Now, what we see is that on those types of roads, the uncertainty in that actually gets quite high.
And so right now, you know, we're just being cautious and basically limiting it to cases
where there's a lane line and that tends to like really tighten the some certainties up.
It gives a lot more confidence that the feature is going to work well.
Have you to make it real provincial? Have you checked out Carmel Valley Road,
which is like one and a half lanes wide with no markings.
But yeah, exactly. So there's the sunroasing trucks coming at you.
Yeah. So the sunroasing California like that. And, you know, right now we don't want people
using the system there. But yeah, I think for me, it's like, you know, and I've been testing
this system for, you know, almost six months. And for me, it's just really the road trip
champ is just fantastic at doing these long journeys on half on highway off highway.
It just it's so much more relaxing than having to do those long drives yourself.
And okay, so you you've been doing the testing in the last six months and you've been doing it
for road trips, but also the point to point like starting from from the parking lot here
Rivian to wherever. So so point to point is actually a different system.
So in parallel, we've been working on this, this extension to that model called the large
driving model. And that kind of goes a step further. So what you see on UHF is that we have
essentially the world model that's predicted out of it. And what we also had ego trajectories,
but we had like a basic form of those. And with the large driving model, we've now like really
been developing that ego prediction head. And so what we've done is we've fed all of this Gen
two customer data where we have the sensor, we have, you know, sequences of sensor data.
And we see how the customer was driving during that sequence. And that gives us the ground
truth that we can train this model so that you know, oh, I saw this, and then here's how the
human drove and the model should try and do something similar. So that's basically the
core of that that system. So that's something that we we've been developing, you know, while
we've been productionizing UHF, we've been working on LDM, you've actually seen some
demos of the point to point on LDM today. LDM will start coming into that stack next year,
and we'll be making all of the autonomy features better. So we'll have universal hands free,
and it will start to have some of that LDM kind of magic being brought into it to make it more
capable. And from a user point of view, it'll just kind of magically get better. Exactly. So
but there's cases actually where you kind of like that for it to be kind of
smartly increasing, decreasing the speed. The large driving model is really good at
actually humanistic speed selection. So that's one of the areas we're thinking we'll bring
the large driving model in first into, you know, enhancements to universal hands free.
I mean, one thing I want to point out is that all of these features, they get better with
every release. So you'll review it now. And literally, every OTA gets better, and we add
more capabilities, more features, and we've been doing that the entire year.
Okay. LDM, is that a trademark? Is that a Rivian trademark?
I don't think so. Probably not. Because like LLM, everybody knows LLM has now
become sort of in the parlance of AI. Is LDM like a tweak? That's not an industry,
like you're building a large driving model. That is unique. That is not an LLM. It's not
because my understanding is your LDM is not language based. It's taking in visual
sensor. Yep. Sorry. It's not rules based. Camera, radar, LiDAR.
Camera and radar today, I want to add next year, we'll add LiDAR.
Okay. So for that reason alone, it's not just, it's not like it's reading maps or whatever,
like language data per se. Yeah. So it also has a map input, right? So the LDM demo you took
today, it actually is following a route. So it actually has an input, an SD map.
And that's, it's kind of, you kind of input geometry. So this kind of speaks to the
generality of the approach. We, in the same kind of architecture, we can input camera data,
we can input radar data, we can input this kind of geometry, which is very different from those two.
But the model is able to incorporate all of these different cues and contacts.
So then I'm trying to, I struggle with, I'm trying to learn AI, and I'm pretty good at
solving and understanding what an LLM is. And I've been hearing world model, but is an LDM
a world model then? Is it, which one's bigger, would you say? So yeah, so LDM is, I mean,
we've intentionally named it, of course, to sound a little bit like LLM. And that's,
that's because it shares so many of the characteristics of an LLM. So the large
driving model, it uses transformers a lot. So in the same way that large language models do,
we use actually tokenization, kind of like large language models, but instead of word tokens,
right, you're sampling little bits of trajectory, right? So normally with an LLM, you feed in some
contexts, and then you sample, you know, texts out of it. For us, we feed in the sensor data,
and then we sample trajectories out of it. That's kind of how it works.
Okay, so then is it not, so it's closer to an LLM than a world model?
Well, so the world model is kind of the base model that is, that it's trained on top of,
if that makes sense. So the way it works is we actually pre-train the model so that it can
detect objects and lane lines and traffic lights and other things. And then we fine-tune this
large driving model head on top of that. So like the world model is like the playing field?
Yeah, so you can imagine that these signals are floating around in some kind of abstract space.
So we have, you know, what you'll see with LLM is it actually handles traffic lights pretty well.
And the model in some sense knew about traffic lights because we'd already been training it
for the perception task of traffic lights. Now, there's no rules that we're putting in
to LLM, but it's sort of that feature space exists, and LLM is able to exploit it well.
So it kind of helps the learning process. Back to the presentation. So point to point,
and then after that, there's eyes off. And that's where you can then start texting or
taking a nap or whatever. That's the idea. Yeah, so point to point will be, you know,
on R1 Gen2 and the R2 Phase 1. And then with the LiDAR and in-house chip that comes on the R2 Phase
2, the Gen3 chip, that will, you know, at a later date stretch to kind of eyes off driving.
Okay. And that's really, is that, because I know RJ said there was Level 4. He said it was
Level 4 personal, which we'll get to in a second. But with eyes off, is that like Level 4 in terms
of like, my eyes are off, now it's just doing the driving and I'm... So that's typically considered
Level 3. Level 3? Yeah. Okay. Yeah. So it's where you, you know, you can take your eyes off within
that defined ODD. Now, we may need to bring you back in for some cases, right? So you, for example,
if there's a construction zone coming up, and we can just take that in a far enough in advance,
we may bring you back in, right? Got it. Obviously, as the system gets better, we can handle more of
those cases. Okay. Level 3 is kind of the way you would do that. ODD means operational design
domain. Yes, exactly. It's what they design the system to work in. And if you puncture or if you
leave that domain, then you have to take control. So is the ODD the 3.5 million miles you're talking
about, or is it more restricted in Level 3? But eyes off, it would be more restricted to begin with.
To begin with. Level 4 means you go back to the full... Well, Level 4 means there's driver out,
right? So there's not even a driver in the seat. So there's no kind of backup. So the challenge
with Level 4 is you, you know, handing over to the driver to say handle construction zone,
that's no longer an option, because the driver's not even there, right? Right. So you essentially
have to, within the routes, handle everything. Okay. So that's how far off... How far off is
Level 4 with... Yeah, RJ mentioned it. So how far off is that in years? So I actually think the gap
from to kind of Level 4 from eyes off is probably smaller than the one to eyes off.
In the sense that once your eyes off, in those ODDs, you essentially have the confidence that
you're human level in terms of safety and performance and comfort and everything else,
and you just need to continually expand the ODD until boom, now you can do the L4 task. I mean,
there's a bit more to it than that, but you know, at a high level, that's kind of...
That's the work that goes into it. So I am intentionally trying to pin you down because
you're going to launch the end of 26, you're going to launch the R2 with LiDAR.
Does that mean eyes off comes with LiDAR or that vehicle is capable of eyes off? Eyes off
happens when eyes off is ready. Yeah, so we won't have eyes off at launch. Okay. We will have...
We'll start to roll out point to point later next year, and yeah, eyes off will be kind of
eyes off by 2028 with an adult debut with Cadillac Escalade IQ. Do you think you'll be
before that or after that? 2028? I don't even know what that means when they say 2028.
Yeah, I mean, I think the key for me is like, is it in a useful ODD? Is it in a useful enough
scale that customers actually will use it? Because I think, you know, people, OEMs have
announced eyes off systems in the past in such a limited way that it's essentially not a real
product in my opinion. So I think for us to do it, I actually want it to be genuinely useful.
And I do think that eyes off is a game changer. Think how many hours people spend commuting every
day. And, you know, you may love driving in certain scenarios, but most people don't enjoy
their commute in the morning. And so I think if you can give people that time back in a real way,
I think that's massive, right? And I think, you know, we'll sell an R2 to you and your wife or
and all your friends at OEM because it has this incredible capability.
That gives you great time back. And then you're saying eyes off is level three. And in my
understanding, the way I was taught was level three means that, you know, Rivian's assuming
the liability. That's in other words, like the, or is that that's with eyes off?
Yeah, I mean, that's, that's a, that's more of a legalistic question. I don't actually don't
know the full answer to. I mean, I think you should assume that if
if you're telling someone it's okay to like, you know, no, no, I agree with you.
Then, then, you know, you need to be very confident that the system is going to behave well.
Okay. Okay. Well, that means that assumes
that Rivian, the OEM is going to assume the liability. I would think
that's my understanding of the definition of level three is that, that when you, when the car
says I'm in level three, you're free to watch a video. You're not driving anymore. The car, I
know that level three is just like, hey, take over. There's like, you know, again,
there's a construction site or there's an ambulance or something.
I think the right edge case is of course, right? Because you're still required to be,
you know, attentive. So it's not like you can sit there and get drunk, right? So that,
so, so that's why it's a nuanced part of the announcement on the hardware side
was this that you're putting LiDAR on the car. And we saw, we saw the sensor demo.
You have 10 cameras, you have five radars, you have a LiDAR and the headliner. I want to,
I want to talk about that, but I want to ask this philosophical question, which is,
you know, I've been doing a lot of testing with Tesla full self-driving the latest version 14.
It's really good. And it is, it is shockingly good point to point. It's, and it's seven cameras.
And I look at the fact that you have 10 plus five radar plus LiDAR.
Why do you need, like, do you, that is, Elon's been very vocal about how, hey, humans have two
eyes. These are seven cameras. It's doing better than humans because I got this neural network.
You guys are talking about, very similarly about, you use a neural net, you're having,
you're having all this back and forth with the cloud and reviewing millions of miles of roads
already traveled. Like, why, why do we need, why do we need additional, this additional sensor
array, the complexity and the cost for that, for a radar and letter? Yeah, I think a couple of
things to think about, right? So one is like speed to market, right? So it is true that if you have
multiple modalities and the same amount of data, you have a much more performance system, or you
can get to the same level of performance with less data. So I think that's one thing is, like,
we can get to a product which is extremely good, much faster. I think the second thing is, like,
that the, the cost of these sensors has really come down massively. So I think you could also
argue, like, why not LiDAR? Why, why actually have a camera and not a LiDAR? So I think there's,
like, there's some asymmetry to, to the argument I hear there, which actually doesn't fully make
sense to me. I think it's assuming that LiDAR is this, this $10,000 spinning thing that breaks down
all the time. I mean, that's not true anymore. That was true 10 years ago. And the world has
really moved on. And LiDAR is really now in that price point where it's similar to a radar.
So to me, it's, it's, you know, I would argue, like, why not put it in there,
especially if it's getting you to that product faster. And the final product has more redundancy
and it's going to be safer and allow that kind of eyes off experience.
So, I mean, I, I mean, you could drive by like covering your eye, but you're a better driver
if you have two eyes. I think it's the same, essentially the same argument.
Yeah. I mean, the way I look at it is that you get, you know, the edge cases get safer. It's like,
it's going to just work better in low visibility and weird lighting situations than cameras will.
And this LiDAR data is important for the ground truth. So, you know, all of these OEMs
have LiDAR based ground truth vehicles, right?
Can you talk about, because ground truth, and I'm sorry if this, if everybody knew what ground
truth was, but when, when they dropped, oh, here we go. Here's, here's the truth right now.
Yes, you're rolling and we're going to capture it anyways.
You're going to get a grill rolling coming in. Yes. Yes.
Yeah, absolutely. I'm sizzling.
I'm trying to, I'm trying to, should we mic him up or are you going to just roll with it?
Okay. Well, we were doing, we were doing great. I just asked him why LiDAR?
Yeah, I was asking. So, I think you, you covered it, but I, I struggle with how good
FSD 14 is. We've been driving it and it's got seven cameras. You guys have 10
and five radar and I will LiDAR. That only points forward. I was a little, I was like,
because you got up on stage and said, we're not taxi cab, we don't got the tiara.
Like, and I was like, oh yeah, this thing's only, the slider is only aimed at this direction.
It's not actually giving you the full 360 degree point cloud, but let's go back. Why,
why all the additional sensors? I know the cost has come down, but still, if the, if the cost is,
you know what you said already, but I can, I can, I can answer it if you haven't said anything.
I did it already. He did, but I, you know, and Ed was like, I think he was suddenly happy.
Yeah. Well, let me, let me, let me give it a try. How are you?
Well, first of all, there's a few things to talk about. So, before we jump into
why LiDAR, I think it's really important to recognize how the model's being built and
how different this is than historically, how LiDARs were used. And so historically,
you talk about LiDARs having turned 60 degree hue and generating this, this three-dimensional
map is one club in a, you know, like think of it as like an 81.0 model. The way that the
systems were designed is you'd had a perception platform, which depending on whether you're
going for level two or level four, it would be cameras and radar or cameras and radar plus
LiDAR in the case of a level four. And it would identify and classify all the objects around you.
And then it would handle all those objects or beyond classifying that associated actors to
those objects. And then it would hand those to a planner and the planner would be rules-based
environment. So, you know, obviously written by humanists that were trying to codify how
what the rules of the road were to say that I drive, is that you drive. And then the system would
make a whole host of decisions around what to do in terms of operation and vehicle.
What the whole world has shifted to and what the, what's clearly going to win in terms of the
approach is what we now often call end-to-end. But what does that actually mean? It's a model
that's trained. It's a, it's a large, very large model that's trained using all the data of the
vehicles that are being driven. Of course, human operating and your training at end-to-end and
running off of this planner that's a rules-based human described environment. In fact, the models,
they have to do a bunch of pre-processing, some are post-processing, but the model's being trained
around behavior. And it's, and it's creating, this is hard for a brain to imagine, but it's
creating a neural net. Like it's creating an understanding of how the world works. And so,
the more information you provide it, the more nuance it can pick up. And so, this is the reason
why you often will use ground-tree fleets to, to add additional nuance to train your cameras. This
is why if you go stand in the corner, you'll see vehicles from, from Tesla that are running with
LiDARs on them. They're not using the LiDARs as production sensors are using as a ground-tree fleet.
And so, our vehicle is a camera-heavy system. So it's primarily camera where I use radars for
additional object detection. The radars also help the model train better. So not only are they there
to identify these corner cases of poor visibility, which are important when you look at the long tail
of safety. Like today, we're driving on a sunny day. So it's, this is not a hard day to drive on.
So you can easily do it today with just cameras. But they cover that long tail of corner cases,
but they also help provide context for the cameras when you're training your camera platform. Like,
is there an object there and it helps you identify, create this, identify what are the
important areas for the, for the model to focus on. In a similar way, LiDARs, which have a non-overlapping
set of strengths relative to a camera or to a radar, of course LiDAR doesn't work well
if you've got things that are blocking its line of sight. I mean, I should stand in front of you,
see, obviously, you block it. But it works really well at providing a three-dimensional image of the
world, which of course, cameras are not three-dimensional devices. This is a 2D set of pixels.
And so you can create deeper understandings of the world that you wouldn't otherwise have,
or you wouldn't as easily have, that you can get with a LiDAR. And so for us, the forward-facing
LiDAR is there because it gets very long range. So it helps cover some of these corner cases for
haspene. I think nitrogen, feeders. Yeah, and it's very helpful for training the model.
And this whole, I think, concern around whether the sensors are confusing each other,
competing. The moment you have more than one sensor, whether it's a camera, a LiDAR or LiDAR,
you have to deal with the fact that you're going to have different cameras or different sensors have
different vantage points, views, or perspectives, and therefore have noise. And in a late fusion
process that's an issue, but in an early fusion process, we're feeding it into it like into a
neural net. It just is a, it doesn't really apply in any of the same ways that we would
classically think about it. And so all that being said, we absolutely, like, and we shouldn't
try to witness, we absolutely are taking more perception heavy approach than Tesla. We have
more, as you said, seven versus 10 cameras, LiAR versus no LiDAR, or Marcus LiAR, Tesla no LiAR.
But it allows us, we believe, to train the model much faster with fewer vehicles
and then to have a higher ceiling on the performance capability
with better corner case coverage. And so that's not in any way of you saying something negative.
I thought the FSD 14 is very, very, very good. It's super impressive what they built architecturally
in terms of building a neural net based approach of those end to end model approaches be very much
aligned to that. In fact, they're very similar in that way. The difference is if you saw the
chart genes had on the front app before you go to the transformer, raise the coding,
we just have more stuff feeding into it. Right. And you have to do that because
they've had a longer head start on the training of their model. They have six billion miles.
There's an important distinction here, which I do want to make. I think we often confuse,
we often overlay time as being equally valuable. So if you're working on a 1.0 architecture,
like a few walls based approach, it's not really that applicable to something that's
AI central approach. And so you actually have like, and this is true of Tesla. So if you look at
autopilot versus FSD, they're completely different model architectures, right? That autopilot is
their historic platform. And FSD is their end to end AI central platform. They're not, they're not
FSD. All the pallets, it's asymptotic to what it is, asymptotic to what it is. And so in our case,
we took the decision in the 2021 to get in 2022 to shift all effort on a complete queen
ship approach, which is our Gen 2 platform, which is an AI central approach. But it's going to get
better with better sensors and better compute. But the model doesn't go away. I'm going to try
to use the analogy if I learn to drive without my glasses on, I would be a less capable driver.
If you suddenly gave me dropped glasses, I wouldn't forget all my skills, right? I would become
better at understanding complex situations and seeing nuance that's hard to see with poor vision.
I meant if you could put radar on my head so I could see if you're fog or seeing the perfect
dark of a night, I'd be even better. And if you have LiDAR that allowed me to train my own
perception models in ways that I can't today, I'd be even better. And then if you said I want to
increase the compute capability of your brain by 20x, something I would do things that are
hard to imagine, but I'd start to notice patterns that I don't see today. You know,
if there's certain things happening in my peripheral vision, maybe that may mean there's
something they need to change or do different. So much more head room, much more potential.
I think that's a fight. Yeah, so yeah, thank you for one question.
Before it leaves us, I don't want to ask this RJ, but is the news here, and this is where I
started before you walked in. Ground truth is sort of a revelation to me. Can you first of all explain
the concept of car manufacturers and ground truth, what it is for our totally ignorant
audience? We're ignorant. Every OAM needs to produce what we call a world model, right? So
I mean, all that is essentially where are the objects around the vehicle? And even in end
to end approaches, they still need that perception signal. It's used for active safety. It's typically
actually used as some of the supervision, even in a more end to end approach. And so typically the
way you do that, because you don't get the depth from a camera bow system, or it's still not clear
often in the camera plus radar based system, just alone, you overlay it with LiDAR and then you can
sort of see the edges of things. So you basically, traditionally, you'd have people like literally
draw in boxes. There's a car there, there's a human there. And so that's what these fleets
traditionally have done, right? So they drive around, they collect this data, humans are annotating
them. You then use those labels to train systems that just use the production senses to produce
those objects. So that's kind of what the ground truth. That's ground truth. That's the traditional
ground truth. Traditional ground truth in every car manufacturer that is attempting anything ADAS
related, anything with cameras. Will have these fleets. Or do they contract with other companies
that do ground truth? I think there are certain, yeah, we have an internal channel where people
post photos of like all these different vehicles. And I remember seeing this, I mean, I remember
seeing this as a kid, my dad had worked for this robotics company, and they were doing
something for the Air Force, and they had like early radar, but yeah, we just put boxes around
like dots on things. Yeah, exactly, right, targets basically. And then I saw like Ford in like 2008
or something, they were tech day at Ford, and it was like slightly better looking. And that's when
Ford had like, what's it called, you know, when you're backing up and it slams on the brakes,
and they were showing, yeah, there's a name for it. It doesn't matter. They were showing boxes
drawn around things. Right. But hang on. So ground truth teams started essentially
early 2000s, kind of the, is it the same time as Google putting cameras on cars and driving around
town for mapping? Does it come from that? And basically, I'm trying to make this real for
the people at home, right? When you see a car on the freeway, and it's got four or five spinning
arrays. So if we really go into the history of it, I mean, I think it was actually Daimler,
and a German professor called Dickman's, very in the actually 80s. And they built some of the
very, very first autonomous vehicles, super early tech, you know, they were, you know,
barely even figuring out, you know, how to build any of this stuff. And he drove vans from, I
think, you know, they had one Berlin to Munich. This was back in the 80s. So that was really the
start of this. Even back then, there was a desire to, you know, we need to know where objects are,
because that's, you know, the most basic thing about driving is you need to know where the lanes
are into where the objects are. How do we know that? So, yeah, so this idea of needing ground
truth. So then is the news, the news portion that when RJ got up and spoke about ground truth and
using the fleet with LIDAR to help, is that the industry first? Are you the first manufacturer
that will be using consumer vehicles to help inform? Was that sort of the... So in the west,
yes, that'll be true. We'll have the largest, you know, LIDAR-equipped consumer fleets
in North America, for sure. Okay. And meaning that in China, this has happened already?
Yes. Okay. Interesting. Okay. And in China, you know, these three modalities, that's not even
really a controversy anymore. Yeah. When you go there, you see all sorts of difference. We talked
about it, like, because we talk to these Chinese manufacturers and they love, it's hilarious. They
send us the spec sheets and they are bragging about the processor inside. They love telling you
they got NVIDIA or they got Snapdragon and they'll tell you how many radars they got,
how many LIDARs. In the halftime, you're like, what do you need? What do you got for them?
You know, we're future-proofing. Right. This is ridiculous.
So what it means is that we get, you know, we get this incredible coverage from that fleet.
And now whenever we see an edge case that, you know, and we can go into some of the ways we
actually would find those, but now we can also, you know, label the perception data in that case.
Right. So it's very, very powerful. So it means that all of that data, we can immediately,
you know, have full label coverage for it. Okay.
So it's not only the driving scenario. We can also do, you know, the perception piece and
everything else. And it just is, it gets a much more powerful data loop that you can build when
you have that ability. So you'll be the, you'll be arguably the first manufacturer. I think you
will be the first manufacturer if this comes out per your plan to deploy LIDAR in wide use. You
were not supposed to be, because Volvo was supposed to have a bunch of cars running around
already. Why has LIDAR been such a challenge for North American or Western?
Yeah, for the industry.
For the Western automobile. Like, what's, what's...
What's tricky about LIDAR?
It's hilarious that Volvo released this vehicle and they put a big blister on the top and they said,
it's there. It's not active yet. We'll turn it on at some point.
And then China has been using it for years.
Nope, we're not doing it. We pulled out with the supplier. We've broken up with that arrangement.
What, why? Like, why is LIDAR still such a... You say it's settled law essentially in China.
Why is the West still having problems with like rolling out LIDAR in vehicles?
I don't know.
If you don't know.
Honestly, I don't because it's a puzzle to me because LIDAR has been used in the L4 domain for
so long, right? So it's, it's not a new sensor in that sense. So I think what is new is the
resolution of it and obviously the cost point of it. And now you can actually bring it to
when you say L4 real quick, you mean like Waymo.
Yeah, the rubber types.
We're also talking about the giant spinning, the spinning things that everyone's like,
there's no way those are coming to a consumer like, like personally on vehicles because...
Yes, because apparently putting spinning things on vehicles is impossible.
But anyway, you try it.
Well, I can imagine my five year old going up and going, what's this? I've watched this,
you know, and that's...
Yeah, yeah. But, but yeah, these, these, the lightest we're talking about, you know,
they're different. They're obviously, you know, have gone through several generations and they're
really, you know, very robust and, you know, capable for their consumer vehicles and all
that. And, you know, the stuff, you'll see in the real world.
All right. I'm sort of sold, I guess. I, you know, I was, it was curious, I'm curious as to
in the time that you've been developing this, which I think RG reference 2021.
So that's the same time as FSD has been launched and we were been super vocal,
super critical about like V12 and how terrible and dangerous it's been in some situations.
But it's gotten really good lately.
What Ed's trying to say is, does your ability to solve edge cases with LiDAR and radar justify
the $500 extra hardware costs that R2 would have that a Model Y won't have?
To me, I think if it accelerates our progress towards getting to those state-of-the-art features,
then I think it justifies it. And I think, you know, if you look at it as a proportion of the
bomb, the overall bomb for R2, it's all still very small, you know, this one.
Bomb is bill of materials. Bill of materials. I'm learning these phrases.
So, you know, for all of the benefits you get and then the headroom you get,
you know, it does, it does feel like it's, you know, the time is right now for putting this into
and, you know, if it had been a few years ago, the answer would probably have been different
because the price has come down so much more in that time.
And I also argue like, you know, you know, these edge cases often involve like hitting people.
So if you can like prevent that, like that, how do you put a price?
Yeah, there's way too many, you know, accidents in the US and
yeah, better sensing and better active safety can really move the needle.
I was, again, a pro at this Instagram post and people are like, I would never like
little robot driving around and I'm like, you know, there's 43,000 people die every day,
every year in the US in non-autonomous vehicles, fully human driven.
So like, you know, if we can improve on that in any way, that's a good thing.
I don't know if this is in your, you're not, you're not a car designer, but the last question
just on LiDAR before, I want to jump into AI with the time we have remaining, but
we saw the sensor array, we saw where you put it and for those of you listening at home,
it is above the mounting point of the rear view mirror and where the two,
Rivian has 10 cameras, there are two forward facing, it's above that array.
It's in the roof.
It's in the right above the windshield, the border of the windshield and it's,
it's recessed slightly. It's an A surface, meaning it's the top, it's the top surface,
but it is not, I thought it was going to be, I thought when they showed it, it was actually
covered, like it would get a nice sweep of the windshield wiper, but it does not to clear
foreign debris. Like, can you just talk about, and again, RG got up on stage, mentioned the,
the blister, the taxi cab, like signal bump, which is sort of what Volvo has, and then the
TRO, which is like the crown of, of, of sensor arrays at a lot of the, the level four, Google
Waymo, if you're not familiar, right? You guys have put this thing now in this spot above the
windshield and it's essentially the 180 degree. I mean, it's 120 degrees. 120 degrees. So it's,
it's not 360. It is a forward view. I get it. That's the direction of travel. Most people
go in, but like, why, what were some of the other concerns? What about things like salt spray?
I'm off-roading. I get mud on it. Getting mud on it or bugs, bugs building up in this high
velocity, high, you know. Yeah. So, yeah. So one of the benefits about putting it up there is
actually you, you do minimize those types of things. So arrow because of arrow and because
you just, you just hire off the ground. So, you know, stuff that's like splitting up, it tends to,
tends to not go that high. And so we, we did a lot of studies on this, actually, you know,
looking at like fascia positions behind the windshield, you know, up in the roofline,
and that, that definitely seems to be kind of the sweet spot. You're also, you kind of want the
sensor higher because then you're above stuff and you can see further, right? If it's always
occluded by the vehicle in front, it's actually not that useful. Okay. I see. You got to get it over.
Okay. Yeah. And then what about, what about maintenance? Will there be an alert in the
car to tell the owner that you need to clean it? They need to wipe it. Yeah, exactly. Yeah. So
actually we, you know, something we built in Gen 2 already is basically like blockage detection,
we call it. So it's essentially every sensor has, you know, another neural network that runs and
actually is telling you, okay, is it blocked? Is it a smear? Is there a bug on the thing?
And, you know, we put that together. And actually a lot of those kind of minor things, we still
enable all the features, but when you get a more significant blockage, then we will tell you,
okay, you need to clean your sensor. And they were mentioning, it meant just for the radars,
but there's some kind of heating. So if something's like ISIS frozen on it,
yeah, does the LiDAR have a heating element? The LiDAR, I actually don't know the answer to that.
We can get you down. All right, let's change gears in the time we have left and talk a little
bit about this, the AI, the Rivian assistant. What is, is that Rivian unified intelligence? Is
that what it's based on? RUI? Did I hear that? Yes, I think so. So I think this is more of a
SIEM's domain. I can talk at a very high level, but I think...
You're the VP of autonomy and AI. Oh, the assistant is not...
Yeah, the assistant is on the SIEM side of the house.
Oh, but isn't it AI-powered?
It is AI-powered, and we collaborate with that team. But I think in terms of the,
his teams own the overall kind of UX and the interactions with customer-facing stuff.
That's a good, that's a good clarification. Well, then what can you tell us about
the AI use in the car, not associated with the assistant? What else is AI doing?
Yeah, yeah. So I think we've seen showed some pretty compelling demos today,
showing AI, you know, interacting with the navigation, messaging, which has been a huge
that customers have been talking about for a long time. And then sort of not just handling the
normal use case, but actually interacting between, you know, so one of the examples he
showed was querying the map for restaurants, sending that context as part of a text message.
So one of these things where you're actually, you're sort of cutting through something that
on other systems would be, they're kind of like a silo, right? You'd have a text messaging app,
and you wouldn't easily be able to get the text out of that app to sort of use it in somewhere
else. Getting map data into a text messaging app. Yeah, yeah, yeah. So yeah, that's this kind of
unified intelligence layer where you can, you can sort of flexibly move inspiration around and
how you have a lot of context when you're doing those types of queries. So I think that's one
thing. The other thing that I think we've seen touched on a little bit was, you know, how we're
using AI to help with service. So we can do sort of more intelligent prognostics and diagnostics
of vehicles. So maybe finding problems before you can even tell they're there.
Give us an example of that, like a battery cell or something?
Yeah, so I think, you know, 12 volt battery is one where, you know, we can actually monitor the
voltage and try to tell if it's reaching end of life before it becomes an issue.
Just replace mine, yes. Yeah. Prognostics is a big, it's a big deal on the commercial side. If
you can, you're a fleet manager, you know when your car might go down, you can do the proactive
maintenance before. Yeah, much better to get the vehicle in before you have a problem, right?
And then after you have a problem. And so the AI would help with that. Yeah. And the way we train
that is, of course, you, you see service cases, you know, that was all fit into the model and the
model has those, you know, those examples that it can pull on. And with this fundamentally
change, and I don't even, this is horrible as a Rivian owner, but I don't even know if there's
like a service schedule if it's in terms of mileage or time. But would this get to the point
where it's like, don't even, there's no set schedule, just we'll let you know when there's
something up, bringing in for service, would it ever get to that? Or are there still like,
I hit the scheduled service intervals? Yeah, I think, you know, in terms of like proactive,
like, you know, service schedules. Yeah, I think we've actually started to roll out those things
for some service items, like the 12 watt battery. Yeah. Okay. Yeah, because I did get an alert,
like change it, you know, yeah, exactly. Yeah. So I think that's, that's that part of one of those,
you know, there's like proactive. Got it. Okay, so let's go back then. I thought we'd have more
on AI, but let's, let's talk about, I want to go back and talk a little bit more about autonomy
and also your background. So you, how long, how long have you been here in Rivian now?
It's three years, I guess you couldn't believe it when someone asked me earlier.
So three years, where should this will go up? Three years in, yeah, three years. Three years.
So this might go in 2026. I don't know, Brent, when are we, when are we running this episode?
Probably later, could be right in the, right in January might be. Okay. So three years,
so call it 2022. And prior to that, you were Waymo. Yes. And prior to that, you were at Zooks.
Okay. So those are two sort of L four commercially, like Robotaxi services, right?
It was very clear when you said L four, that it would be personal L four,
meaning autonomous vehicles that are personally owned, not a service.
Rivian, not nowhere on the roadmap are going to do a Robotaxi or a Waymo type service. You
have all the technology and the big unlock. You know, I remember I've been looking at this stuff
when McKinsey wrote a report, they call it case, right? Connected, autonomous, shared electric.
This is supposed to be the major game changer of EVs is when they're autonomous, they, you,
you go to sleep in your cars, taxi, taxi, make you money. It's not just sitting in your,
in your driveway, losing money. Like if you have all the sensors, if you have,
if you have all the ability, you got a neural network. Will we see a Rivian Robotaxi?
So I think, yeah, first of all, I'd say like, we have all the technology pieces, as you, as you
said. Now, my personal belief is that actually, although Robotaxi will be a significant market,
it's a road sharing, at least in the US, people are not going to be getting rid of their personal
vehicles. People vomit in taxis and worse, all the time. Yeah. And I actually think there's like
just certain geographic realities in the US that make having a taxi service cover, you know,
all of these suburbs in an efficient way, actually very, very challenging. So, you know,
and I look in the back of my car and it's just full of crap from, you know, my family. I have
like car seats in there. I have like stuff for the beach. I have, you know, some hiking boots.
And so it's just like, you need to be willing to give up all that and do everything through a
ride sharing. So I think to me, actually, what will happen is people will demand higher and higher
levels of autonomy from their personal vehicles while they're also using Robotaxi for all the
times when those things are great. And so I actually think that the tide of kind of autonomy
expectations is really going to rise significantly. I think we actually already are seeing this in
terms of when we talk to customers in the sales spaces, you know, what's what are the key buying
purchase decisions. So you're saying that the owners, you don't see people giving up personal
cars to use exclusively Robotaxi services, whether you deliver one or whether it's somebody else.
That may happen a little bit, but I actually think so people have in mind that,
you know, Uber is filled with a lot of personal vehicles. That's actually not
true. And that hasn't been true for many years. The majority of Uber vehicles are fleet vehicles.
And so for me, it's what would be different about an L4 vehicle that would make that equation
different. So the truth is like, you people don't want to be cleaning out their car every night.
You don't want to be dealing with the crazy insurance you'd have to cover just from people
coming in and out of the car all day. You don't want to, you know, just think of tire usage and
all the other things. To me, it's actually like a not, it's an, it doesn't really make sense
economically. And I think, you know, if you look at, you know, an Uber, where they're using all
these fleets, you know, why would the equation be different? Not of us making you literally
hundreds of dollars a night, John? Hundreds? No. Yeah, exactly. That's what's the cut.
But just so everyone knows, like, so at the presentation today, RJ said, like,
L4 personal level for personal meeting, like, you could sit on the couch and doom scroll while
your car drives your kids to soccer practice and then go runs another errand and then comes back
because it doesn't need a driver. Right. That's very compelling. That's like, that's cool. Yeah,
that would be amazing. Yeah, it'd be, again, someone who spends their weekends shuttling kids
around. If you ever sat in a soccer practice with little kids, nothing worse. Yeah. That's,
that's okay. All right. That's, that's interesting. I hadn't thought about it that way. So, okay,
points to that. Well, yeah, because I heard that, I'm like, that would be really, really cool. Now,
yeah, if you wanted to do that, you know, you could, but there's the sort of extra things you
would have to put in the vehicle to handle that. So for example, you need to, you need to deal with
what happens if people have left stuff in the vehicle in the trunk? What if, you know, the doors
are not closed, right? Do you need to auto cinch them? What do you do about teleop? I assume you're
not going to be sitting at home in front of your computer, like helping your vehicle, you know,
get out of a tricky situation. Teleoperation is a requirement to run these things. Yes. They will
always, at least in the foreseeable future. Teleoperation is, it's what a lot of folks are
claiming big company starts with a T, you remember the guy with an E in his name is doing with the
robots, like that. Yeah. That's when they're Warner Brothers or whatever, 100%. I know the
people that are operating. Okay. Let me ask you about your time just briefly at Waymo because,
you know, we've been using the service in LA, it's pretty good. We're looking forward to the
expansion under the highways and all the way into LAX. I think they're test running it now for
Waymo, the SFO for, but employee only. We've also heard some interesting cases of maybe the
cars running over dogs and cats, blowing through. This is the good one. Yeah. The one where there's
a full on police pursuit and they had the guy, the perp down on the sidewalk. And the police
had their guns drawn. All the lights and tyrants are going off on the police car.
Just like right through it. And not to bag on our Waymo friends.
I like Waymo. Waymo's great because, you know, no one talks to you. It's the best.
And actually, sorry, the third one is stopped school buses where the stop sign comes out,
the arm comes out and everybody has to stop. And like, apparently Waymo's are blown right through
it. Do you consider those edge cases and how do you plan for them? Is this
the kind of stuff that keeps you up at night or? Yeah, I think, you know, I would definitely
consider some of those cases edge cases. Now, you know, like as the fleet expands,
you know, you go ever further into the tail. So, you know, the perp on the sidewalk maybe
doesn't become an edge case at some point because you see so many models. Yeah. So,
so I do think, you know, as you scale, you are just, you know, ever more pushing into that long
tail, which is why I think that the building up this, the scalable data flywheel, which can always
kind of cover more and more cases is like so important because otherwise you're always chasing
that next, that next edge case, next edge case, next. It's never ending as these fleets grow.
Whereas if you're building the systems that can scale to handle them, you know, through systems
that are somewhat automatic, that really gives you the sort of superpower to scale up quickly.
And I also wonder too, like, you know, yeah, okay, the Waymo drove through that,
like how many humans have done the exact same thing, like probably like thousand or more than
that. And it's just, it's like, you know, it's like when an EV catches on fire, you know, it's
been like five or six, and it's really well publicized. And we looked it up, 120,000 vehicle
fires in the US a year never get a drop of ink because they're not EVs. I've seen many more
ICE cars on fire.
Yeah, no, I mean, it's not even close. I mean, it's literally like a dozen versus 120,000 or
something like that. And so I wonder, like, you know, with the Waymos, like, if they, you know,
like, I know lots of people that have hit dogs and cats driving cars, you know, and I know lots
of people that have killed people in cars or died in cars and, you know, I mean, I know of them.
And I know, yeah, I know a couple. Yeah, sure.
All right, let me, I'm just, let's, let's, let's jump to, what is it, two years in the future.
And whatever, whichever, assuming all great success, and you got our two's out there,
and they're running on the Gen three, and we have universal hands free.
No, no, no, I have eyes off.
Do you have eyes off? Will the, will you allow me to eyes off, hands off on the freeway
that's marked at 70 miles an hour go go 80?
Can I go 80, can I go 80 85, which is flow of, which is flow of traffic?
So I don't know. And so the reason I'm being a bit evasive is some of this
is actually controlled state by state. And so to get the, the permit to operate an L three system,
there's certain things you have to do. And they're actually differs in California from Nevada,
from other states. So in some states, there probably is no option, we would have to follow
the speed limit. In other states, you could be more probably more flexible.
It was there, I was saying, is there like a waiver? Like, look,
even though you're driving and you have the liability for an accident, I'll take the ticket.
Yeah, it's a, or is that just too much? It's a regulation. That's the thing. So you,
there's no, there's no dodging around it. But even if it flow traffic, like when I can make the
argument that like, going the speed limit, if every other vehicle on the road is going 10 over
the speed limit, it's less safe to, you know, what would happen is we get pulled over, you know,
you get a ticket, it would go to the DMV in Sacramento, I would get a call probably the
head of California DMV and say, Oh, why are you speeding? We're going to revoke your autonomy
license. But again, but every, it's just one of those laws. It's, you know, it's like,
everyone speeds like nobody goes. I encourage you to pick up the phone and call your congressman.
Let me ask you, let me ask you a variation of this question, because
Tesla FSD currently allows you to do that. And I don't understand how, do you have any understanding
how they can, they can allow, they can allow a bad max mode, the vehicle to overspeed and also
jump in and out of an HOV land crossing. But remember that system is still an L2 system.
So that's because I chose to do that. And it's, and your eyes are on.
Interesting. And L2 means that the driver has the legal liability, not the OEM, not Tesla.
So yeah, that's how. Let me, let me ask you my last question, because we're, we're basically
running out of time here. You got your start, I didn't go all the way back, but you spent a lot
of time at Google, you were doing computer vision and machine learning algorithm, image annotation.
You did Google goggles, image search, photo search. Did you think, did you have an interest in cars?
Did you think you would be doing this in what I think it's like 10 years later?
That's a good question. I, yeah, I always thought it was a fascinating problem. Like,
could you teach a vehicle to drive itself? But it just seems so far out of reach at that time.
But then, you know, you start to see, you know, I would say, you know, I did my PhD in, you know,
computer vision and at that time, the stuff barely worked, right? You, you have, we all have stuff
on our phones now, which are just so much more advanced than even what people were just could
dream of at that time. And so I think, you know, as the machine learning computer vision sort of
started to become more mature, that problem felt like it was coming into reach and was like very
exciting because vehicles are really robots, right? They actuate their wheels, they have controls,
steering wheel pedals, they have sensors. And they're, unlike typical robots, which, you know,
at least at that time where they would break down all the time, you know, you'd have like a lab
robot and you maybe get 10 minutes of it doing something and then and you have to, you know,
then it's in the shop for the next few days. Vehicles were very, you know, have been very
reliable because yeah, they've gone through all those iterations. And so that's what made it a
super interesting problem because it's a, it's a robot where the hardware reliability is already
there. It's a relatively simple control problem. It has an amazing set of, you know, use cases
that can really change and improve society, I believe. And so, and then to have these machine
learning techniques come within grasp, it really felt like that was the time.
Did you work with Sebastian Thrun? I didn't know. So he was a, you know, chauffeur and then he left
and then yeah, I went to Zooks. My question is what changed? Because I remember there was like a
BMW technology day. It was right when like Wall Street was obsessed with autonomous cars. You
was going to go autonomous first. It's all Wall Street cares about. Yeah. And then I remember
there was a BMW technology day and they were like, forget about it. We have a thousand engineers
working on this problem. It's never going to happen within the foreseeable future. Like it's
just too complicated. And that was, I feel like in my mind, that was like 2017, 2016, something
like that. But like today, you know, we took a trip, a point to point trip where the guy was
hands off 99% of it. And, you know, admittedly, it's not the finished product. But like what
changed? What was the big like breakthrough? And there's Waymo's driving around. I like taking
Waymo's, you know, yeah, it's a great product. Yeah. What was the change? What happened? So I
think, you know, a few things. So I think there was a huge revolution in ML and AI. So that's one
thing. And that's been, I mean, to consumers, it's maybe only visible the last few years. But
versus being getting better and better and better and better and better. So that's one thing that
Wave has been writing. Is it a hardware advancement, a software both? A bit of both, right? So I
think it's the techniques that were being used. It's the fact that, you know, GPUs came along that
could actually train these things very quickly, much more quickly than you could before. It was
the data as well. So just the massive scaling up in data. You know, when I started my PhD,
you could have a paper that maybe would have a data set of like a hundred images that was considered
a big data set. And now it's like billions, right? And so that's also scaled up. And so as
everything has scaled, these techniques have got better. And then, you know, more recently,
large language models, transformers, everything is sort of just compounding on itself. And then
on the vehicle side, you know, the fidelity of the camera data, you know, having the radars have
gotten much better. Lidars have got incredibly better and so much cheaper in 10 years. It's
same time frame. Did you video which there's a massive dependency on? Absolutely.
Well, so let me let me just do this one last question. Like, what are what are you seeing in
the labs that we're not seeing that's going to like make us real happy in a couple of years? Like,
what's the next like, you know, like besides personal level four, like, you know, what are
not just ravines, but vehicles going to be able to do that they can't do now.
I mean, I think one, yeah, so I, you know, I have, you know, daughters 11 years old and, you know,
at some point, they're going to want to drive. And so something I'm very much looking forward to
is a vehicle that has a safety bubble so good, it can basically not be crashed. So you can, you
know, I still think people will want to drive and they'll need to be trained how to do so. But I
think that the these world models and these AI models could get to the point where you essentially
can't crash your vehicle. You try and turn and it's like, no, no, I can see there's a vehicle there.
You try and speed, it's like, no, no, we're not going to do that. Don't do that in the teenager
mode, right? So I think that's that would be kind of awesome. Well, my last question,
and I want to get you to commit on this is when gen two comes out and it's got all the bells and
whistles. Can we run the Rivian Transamerica Trail fully autonomous? This is on this is not a paved
road. Okay, and it's unmarked. But could you do you have, do you think we could attempt that?
Is there is it within the realm of possibility in your law in your LDM and your large?
That's interesting. It what is it? I don't actually know what it looks like. Is it just
this thing? So we cross the country. Yeah, so we are marked at all. It has that starts in South
North Carolina. Before you were at Rivian, we teamed up 7700 miles, took us 43 days,
all off road the whole way. Yeah. And we try it at least. Yeah. Yeah, to do what, let's
we'll get some data and then let's let's talk. Let's have a yes. Because it's really fun and
we're done. Yeah, great to meet you guys so much. Thank you so much. Cheers.
About this episode
Rivian's recent Autonomy and AI Day revealed exciting advancements in their technology, including the introduction of Universal Hands-Free driving, which expands their highway assist system to over 3.5 million miles. VP of Autonomy and AI, James Philbin, discussed the integration of LiDAR in the upcoming R2 model and the development of a point-to-point driving system. The episode also dives into the implications of AI in vehicle operation, including the Rivian Assistant, which enhances user interaction and diagnostics. The conversation touches on the future of personal Level 4 autonomy and the challenges of implementing such technology.
Original notes
Rivian is making a bold bet on autonomy—and it’s very different from Tesla’s approach. In this episode of MotorTrend’s The Inevitable, Jonny Lieberman and Ed Loh sit down with James Philbin, Rivian’s VP of Autonomy & AI, following Rivian’s Autonomy & AI Day. Midway through the conversation, Rivian CEO RJ Scaringe joins unexpectedly to dive deep into LiDAR, sensor strategy, and why Rivian believes more data—and better data—wins.
Topics include:
• Universal Hands-Free driving across 3.5M+ miles of roads
• Rivian’s Large Driving Model (LDM) and AI-defined vehicles
• Why Rivian chose cameras + radar + LiDAR
• The roadmap from hands-free to eyes-off driving • Personal Level 4 autonomy vs robotaxis
• Ground-truth fleets, edge cases, and safety at scale
• How autonomy could fundamentally change car ownership
This is one of the most technical, candid, and forward-looking autonomy conversations we’ve had—and a rare look inside how a modern automaker is building an advanced driving system from the ground up.
The InEVitable
MotorTrend
0:00
68:50