Rivian’s AI Master Plan: Hands-Free Driving, R2 + LiDAR, and the Road to Personal Level 4 Autonomy

Hi there and welcome to The Inevitable. This is Motor Trends Podcast, our podcast about

the future of the car, the future of the automobile, the future of autonomy and AI at a little

California-based, sort of California-based startup called Rivian. You might have heard

of them.

I hate the fact that their genius CEO is younger than me. It really just is really…

And he looks like Superman. He's taller than you. He's better looking than you.

Yeah, it's just…

Yeah, he's smarter than you.

He's worth a billion dollars or more. It's just endlessly depressing. However, and I

went up there for something they called autonomy and AI day and we learned a lot. We actually,

like, I didn't think we'd learned a lot. We learned a lot. And before we get to our

interview, Ed has a message just for you.

Yes, the Inevitable podcast is brought to you by nobody currently. I'm going to ring

the bell.

Yeah, yeah, do it.

There you go.

So, if you want to sponsor, shoot us a note, Edward.loh.herst.com or slide into our DMs. Indeed,

we are fresh back from Rivian's autonomy and AI day. It was a full, basically, half-day devoted

to everything they're doing within the space. We saw microchips, GPUs, we saw sensors, we

saw where the lidar is going to be positioned on the all-new R2 smaller Rivian SUV.

We took a drive with a robot driving a car.

We went to a hands-free, hands-free drive around Rivian's campus in Palo Alto, California.

But like a big 19-mile kind of drive around.

And then at the end of the day, we had a chance to sit down with James Philbin, who is their

vice president of autonomy and AI at Rivian. This guy, SuperSmart, he was very casual and

modest about saying that he's actually Dr. James Philbin. He's a PhD.

Yeah, who built Waymo's.

Yes, he was previously at Waymo for a couple of years as director of software engineering.

And but prior to that, he was at Zoox for four years, senior director of perception

and computer vision. Prior to that, he spent a number of years at Google. He was a staff

software engineer, core computer vision, machine language algorithms, and image annotation and

understanding. So yeah, so once again, Ed and I are two dummies in a room with someone who's

probably too smart to talk to us. Yeah, very smart, very kind to answer all of our dumb questions.

Philbin worked on such products he would know as Google goggles, Google image search,

and Google photo search. So this is a guy who really knows perception and imaging and is now

applying it to what they're doing, what Rivian is doing with autonomy and AI. It's a fascinating

conversation. And we're interrupted halfway through with a special guest. There's a special

and actually watching and listening because our guests kind of barged in.

Yeah, and did very rudely, gave us some great information, but wasn't wearing a one of the

mics that we had. A lavalier. Yeah, so our producer, we got a directional mic on. Yeah,

I think Brent figured it out. So if the audio levels are too bad, but it's a great conversation,

wide ranging. So let's go right to it. James Philbin, VP of autonomy at Rivian.

James Philbin, Vice President, autonomy and AI at Rivian. Thanks for coming on.

Yeah, thanks for having me. Today is apparently your day. It was it was Rivian's autonomy and AI

day. So why don't we start there? And can you just tell us what was the big, what was the,

what were the headlines from this big day? Because normally it's, it's been what Rivian

autonomy day. And now the AI piece was, is kind of the new, the new hotness. Is that right?

I don't know about that. So I think, you know, that's always been sort of part of

the vision. So I would say, you know, today, we had a few announcements, it's quite a lot to take

in to be there. So I think on the autonomy side, we announced universal hands free, which is our

big scaling up in the hands free system we launched actually earlier this year. So the current system

is enhanced highway assist that covers 135,000 miles of highway, you know, hands free highway

miles. And then universal hands free, which is what's coming out this month to our one gen two

customers scales at up to three and a half million. So it's a massive, massive growth.

And that was a huge map you showed of North America. Like what?

Yeah, it's crazy right.

US and Canada and like everything was green. Everything is right. Except for Wyoming. Yeah.

Universal hands free. Okay. And then

we talked through some of the roadmap for next year. So

big piece is a point to point system that we've been developing based on this very large driving

model. The point to point means that like I want to go to 711 from my house. So I say 711 in the

car without me doing anything drives to 711. That's a full driving task following the routes,

turning, you know, responding to traffic lights, stop signs, other traffic, the whole thing.

So yeah, we, we're not releasing that today. We were, we have been working on it and we

actually gave some demos here in Palo Alto sites. We took a demo ride.

You can tell me how it was. It worked. Here we are. Yeah. So that's been,

yeah, like really incredible progress we've seen there. And a large part of that is really

powered by this sort of data flywheel that we, we launched on actually our one gen two all the

way back last year. Right. So what that has done is essentially create this huge database of driving

scenarios from all of our customers. So we get this sort of incredibly rich kind of long tail

edge case scenarios streaming back into the cloud. That thing gets fed into the model and the model

can then learn those driving behaviors. So some of the drives today, it's kind of like the median

Rivian driver, right? That's kind of what it's learned in terms of how it can respond to things.

And then on the hardware sides, we announced, yeah, two, two important things. So one is

that R2 from next year, later next year, we'll have LiDAR and we showed the integration of

that, which looks amazing. So, and that's really a huge collaboration with the design studio and

all of the, you know, the headline design folks and everything. And then also R2 from next year

will have a new autonomy chip developed in-house. That's our Rivian autonomy processor. And we

went into some of the details of that chip. So we're in December, mid December of 25.

And I don't know if you've given a hard day, but when does the R2 launch?

Can you say even what quarter?

First half of next year.

First half of next year. And I know that LiDAR is an optional thing. So you would buy

a R2 that's equipped with LiDAR and that comes later after the launch. But the chipset, that dual

Rivian chipset that we saw, is that on every R2 or only the ones that are LiDAR equipped?

It will only be on the second phase of R2, which will be later next year. And every

second phase R2 will also have the LiDAR. So there'll be a kind of a flag day where they

switch over and now those R2s will have the new chip and the LiDAR.

The new chip said is Gen 3 autonomy.

Yes.

Gen 3 compute. So the R2 will launch with Gen 2 initially and then it'll move to,

you could, if you want, get the Gen 3.

Exactly. So, you know, it's a very capable system still and it will get all of the features that,

you know, R1, Gen 2 owners will have as well.

And we took a ride, just to be totally clear, we took a ride in a Rivian R1S generation 2.

Yes. Yeah, exactly. Yeah. And the demo rides you saw today, production vehicles, nothing

special about the sensors or the compute in them, exactly what every customer has in their own.

Right. So it will be capable of the hands off, point to point.

Yes, exactly.

Just trying to get the terminology.

Absolutely.

We got a lot of questions about LiDAR and their, but let's just quickly wrap on that.

The other nuts when it's on AI, right?

Yes. And then, you know, Waseem went into some of the details of the new Rivian Assistant,

which will be in every Gen 1 and Gen 2 vehicle and obviously R2.

And that has this sort of amazing integration throughout the stack.

So you can, you know, Waseem showed some demos, you know, integrated with Calendar,

where you can actually change events and then you can kind of have contextual queries.

So you can text someone and it will actually include context in that text.

Yeah, the example, it was a cool example was like, hey, we're going to this somewhere in San Francisco,

give me three restaurants nearby or give me a list of restaurants nearby.

And then the voice command was take those top three, text them to somebody else

and ask them which one they want to eat at, send. And that was, yeah, it's cool.

That's like, it's definitely a real world scenario.

So and then you also mentioned what I thought was interesting was that

RJ said it was going, you're moving from a software-defined vehicle to an AI-defined

vehicle. And I bring that up for a couple reasons. A, it's interesting. B was I quoted that,

I put it up on my Instagram post and a lot of people wrote back like, oh, everything's now AI.

I wish they'd stopped doing that. What does revision mean when you say an AI-defined vehicle?

Yes, I think it's more that the autonomy and AI are kind of infusing everything about the vehicle,

right? And so autonomy is obviously handling whole driving function and more and more of it.

And then AI can sort of provide that, the kind of the fabric for the vehicle,

so that different components that used to be kind of separate can actually interact together.

I think that's the way I would see it, sort of like almost like a connective tissue that ties

all of these systems together. And so AI allows you much more flexibility in terms of combining

the output of this system and that system and how it interacts. And then autonomy, of course,

is about handling more and more of the driving function. So we really see that trend happening

where autonomy is getting more and more powerful. And then on the AI side, it's breaking down these

sort of barriers between systems and allowing a much more fluid sort of user experience.

So I think one of the examples today was changing the seat-heat settings. And you didn't have to

say, you know, turn on the heat of a seat, you know, one, two, and three, you said, like,

turn it on for everyone but me, right? So it's a very contextual kind of query. And then the results

are super useful. Let's take it from the top and go back to what you were saying. So your

current system of E8, EHA. So enhanced highway assist.

Enhanced highway assist has 135,000 miles of mapped roadways. And the new one, which you're calling

Universal Hands Free. Universal Hands Free. I heard some guys talking about it. Are you,

what's the internal, is it just UHF or is it RUF? So we've shown it to kind of UHF,

but yeah. Universal Hands Free. I've been updating our team at Motor Trend during the

press conference in our technical editor. Eric Tingwell was like, where did they get the 3.5

million miles from? Where did, is that from the flywheel? From the, or no?

Yeah, no. So actually, so it's a good question. So the system basically works on any roads with

well marked lane lines. And so we were like, actually, how many roads do have well marked

lane lines? So we didn't get a precise answer. We know that it's more than 3.5 million. So that's

why we said more than 3.5 million. And we got that from like some government statistics and

some other things. It may be quite a bit more, but we didn't want to say that without knowing it.

So that's why we ended up with the 3.5 million mile bigger.

So UHF basically works on any, any paved road.

Marked, marked. So there should be, you know, lane lines that you can see.

Lines or line? Does it need both? No, it doesn't need both, but it needs at least one.

So in other words, like a one mile paved driveway to somebody's cabin, it's not going to work on that.

Yeah. And actually, you know, so we can obviously test it in those conditions and the system mostly

actually works fine in those cases, but we want to build that confidence before we, you know,

it won't stop. It won't read like the ending of the asphalt or bitumen to the, to the dirt

shoulder as a line. So we, we actually predict road edge that would be a road edge. And we also

predict something called a virtual lane center. And that's actually how the system works internally.

So it tries to estimate, you know, here is where a driver would actually position.

Now, what we see is that on those types of roads, the uncertainty in that actually gets quite high.

And so right now, you know, we're just being cautious and basically limiting it to cases

where there's a lane line and that tends to like really tighten the some certainties up.

It gives a lot more confidence that the feature is going to work well.

Have you to make it real provincial? Have you checked out Carmel Valley Road,

which is like one and a half lanes wide with no markings.

But yeah, exactly. So there's the sunroasing trucks coming at you.

Yeah. So the sunroasing California like that. And, you know, right now we don't want people

using the system there. But yeah, I think for me, it's like, you know, and I've been testing

this system for, you know, almost six months. And for me, it's just really the road trip

champ is just fantastic at doing these long journeys on half on highway off highway.

It just it's so much more relaxing than having to do those long drives yourself.

And okay, so you you've been doing the testing in the last six months and you've been doing it

for road trips, but also the point to point like starting from from the parking lot here

Rivian to wherever. So so point to point is actually a different system.

So in parallel, we've been working on this, this extension to that model called the large

driving model. And that kind of goes a step further. So what you see on UHF is that we have

essentially the world model that's predicted out of it. And what we also had ego trajectories,

but we had like a basic form of those. And with the large driving model, we've now like really

been developing that ego prediction head. And so what we've done is we've fed all of this Gen

two customer data where we have the sensor, we have, you know, sequences of sensor data.

And we see how the customer was driving during that sequence. And that gives us the ground

truth that we can train this model so that you know, oh, I saw this, and then here's how the

human drove and the model should try and do something similar. So that's basically the

core of that that system. So that's something that we we've been developing, you know, while

we've been productionizing UHF, we've been working on LDM, you've actually seen some

demos of the point to point on LDM today. LDM will start coming into that stack next year,

and we'll be making all of the autonomy features better. So we'll have universal hands free,

and it will start to have some of that LDM kind of magic being brought into it to make it more

capable. And from a user point of view, it'll just kind of magically get better. Exactly. So

but there's cases actually where you kind of like that for it to be kind of

smartly increasing, decreasing the speed. The large driving model is really good at

actually humanistic speed selection. So that's one of the areas we're thinking we'll bring

the large driving model in first into, you know, enhancements to universal hands free.

I mean, one thing I want to point out is that all of these features, they get better with

every release. So you'll review it now. And literally, every OTA gets better, and we add

more capabilities, more features, and we've been doing that the entire year.

Okay. LDM, is that a trademark? Is that a Rivian trademark?

I don't think so. Probably not. Because like LLM, everybody knows LLM has now

become sort of in the parlance of AI. Is LDM like a tweak? That's not an industry,

like you're building a large driving model. That is unique. That is not an LLM. It's not

because my understanding is your LDM is not language based. It's taking in visual

sensor. Yep. Sorry. It's not rules based. Camera, radar, LiDAR.

Camera and radar today, I want to add next year, we'll add LiDAR.

Okay. So for that reason alone, it's not just, it's not like it's reading maps or whatever,

like language data per se. Yeah. So it also has a map input, right? So the LDM demo you took

today, it actually is following a route. So it actually has an input, an SD map.

And that's, it's kind of, you kind of input geometry. So this kind of speaks to the

generality of the approach. We, in the same kind of architecture, we can input camera data,

we can input radar data, we can input this kind of geometry, which is very different from those two.

But the model is able to incorporate all of these different cues and contacts.

So then I'm trying to, I struggle with, I'm trying to learn AI, and I'm pretty good at

solving and understanding what an LLM is. And I've been hearing world model, but is an LDM

a world model then? Is it, which one's bigger, would you say? So yeah, so LDM is, I mean,

we've intentionally named it, of course, to sound a little bit like LLM. And that's,

that's because it shares so many of the characteristics of an LLM. So the large

driving model, it uses transformers a lot. So in the same way that large language models do,

we use actually tokenization, kind of like large language models, but instead of word tokens,

right, you're sampling little bits of trajectory, right? So normally with an LLM, you feed in some

contexts, and then you sample, you know, texts out of it. For us, we feed in the sensor data,

and then we sample trajectories out of it. That's kind of how it works.

Okay, so then is it not, so it's closer to an LLM than a world model?

Well, so the world model is kind of the base model that is, that it's trained on top of,

if that makes sense. So the way it works is we actually pre-train the model so that it can

detect objects and lane lines and traffic lights and other things. And then we fine-tune this

large driving model head on top of that. So like the world model is like the playing field?

Yeah, so you can imagine that these signals are floating around in some kind of abstract space.

So we have, you know, what you'll see with LLM is it actually handles traffic lights pretty well.

And the model in some sense knew about traffic lights because we'd already been training it

for the perception task of traffic lights. Now, there's no rules that we're putting in

to LLM, but it's sort of that feature space exists, and LLM is able to exploit it well.

So it kind of helps the learning process. Back to the presentation. So point to point,

and then after that, there's eyes off. And that's where you can then start texting or

taking a nap or whatever. That's the idea. Yeah, so point to point will be, you know,

on R1 Gen2 and the R2 Phase 1. And then with the LiDAR and in-house chip that comes on the R2 Phase

2, the Gen3 chip, that will, you know, at a later date stretch to kind of eyes off driving.

Okay. And that's really, is that, because I know RJ said there was Level 4. He said it was

Level 4 personal, which we'll get to in a second. But with eyes off, is that like Level 4 in terms

of like, my eyes are off, now it's just doing the driving and I'm... So that's typically considered

Level 3. Level 3? Yeah. Okay. Yeah. So it's where you, you know, you can take your eyes off within

that defined ODD. Now, we may need to bring you back in for some cases, right? So you, for example,

if there's a construction zone coming up, and we can just take that in a far enough in advance,

we may bring you back in, right? Got it. Obviously, as the system gets better, we can handle more of

those cases. Okay. Level 3 is kind of the way you would do that. ODD means operational design

domain. Yes, exactly. It's what they design the system to work in. And if you puncture or if you

leave that domain, then you have to take control. So is the ODD the 3.5 million miles you're talking

about, or is it more restricted in Level 3? But eyes off, it would be more restricted to begin with.

To begin with. Level 4 means you go back to the full... Well, Level 4 means there's driver out,

right? So there's not even a driver in the seat. So there's no kind of backup. So the challenge

with Level 4 is you, you know, handing over to the driver to say handle construction zone,

that's no longer an option, because the driver's not even there, right? Right. So you essentially

have to, within the routes, handle everything. Okay. So that's how far off... How far off is

Level 4 with... Yeah, RJ mentioned it. So how far off is that in years? So I actually think the gap

from to kind of Level 4 from eyes off is probably smaller than the one to eyes off.

In the sense that once your eyes off, in those ODDs, you essentially have the confidence that

you're human level in terms of safety and performance and comfort and everything else,

and you just need to continually expand the ODD until boom, now you can do the L4 task. I mean,

there's a bit more to it than that, but you know, at a high level, that's kind of...

That's the work that goes into it. So I am intentionally trying to pin you down because

you're going to launch the end of 26, you're going to launch the R2 with LiDAR.

Does that mean eyes off comes with LiDAR or that vehicle is capable of eyes off? Eyes off

happens when eyes off is ready. Yeah, so we won't have eyes off at launch. Okay. We will have...

We'll start to roll out point to point later next year, and yeah, eyes off will be kind of

eyes off by 2028 with an adult debut with Cadillac Escalade IQ. Do you think you'll be

before that or after that? 2028? I don't even know what that means when they say 2028.

Yeah, I mean, I think the key for me is like, is it in a useful ODD? Is it in a useful enough

scale that customers actually will use it? Because I think, you know, people, OEMs have

announced eyes off systems in the past in such a limited way that it's essentially not a real

product in my opinion. So I think for us to do it, I actually want it to be genuinely useful.

And I do think that eyes off is a game changer. Think how many hours people spend commuting every

day. And, you know, you may love driving in certain scenarios, but most people don't enjoy

their commute in the morning. And so I think if you can give people that time back in a real way,

I think that's massive, right? And I think, you know, we'll sell an R2 to you and your wife or

and all your friends at OEM because it has this incredible capability.

That gives you great time back. And then you're saying eyes off is level three. And in my

understanding, the way I was taught was level three means that, you know, Rivian's assuming

the liability. That's in other words, like the, or is that that's with eyes off?

Yeah, I mean, that's, that's a, that's more of a legalistic question. I don't actually don't

know the full answer to. I mean, I think you should assume that if

if you're telling someone it's okay to like, you know, no, no, I agree with you.

Then, then, you know, you need to be very confident that the system is going to behave well.

Okay. Okay. Well, that means that assumes

that Rivian, the OEM is going to assume the liability. I would think

that's my understanding of the definition of level three is that, that when you, when the car

says I'm in level three, you're free to watch a video. You're not driving anymore. The car, I

know that level three is just like, hey, take over. There's like, you know, again,

there's a construction site or there's an ambulance or something.

I think the right edge case is of course, right? Because you're still required to be,

you know, attentive. So it's not like you can sit there and get drunk, right? So that,

so, so that's why it's a nuanced part of the announcement on the hardware side

was this that you're putting LiDAR on the car. And we saw, we saw the sensor demo.

You have 10 cameras, you have five radars, you have a LiDAR and the headliner. I want to,

I want to talk about that, but I want to ask this philosophical question, which is,

you know, I've been doing a lot of testing with Tesla full self-driving the latest version 14.

It's really good. And it is, it is shockingly good point to point. It's, and it's seven cameras.

And I look at the fact that you have 10 plus five radar plus LiDAR.

Why do you need, like, do you, that is, Elon's been very vocal about how, hey, humans have two

eyes. These are seven cameras. It's doing better than humans because I got this neural network.

You guys are talking about, very similarly about, you use a neural net, you're having,

you're having all this back and forth with the cloud and reviewing millions of miles of roads

already traveled. Like, why, why do we need, why do we need additional, this additional sensor

array, the complexity and the cost for that, for a radar and letter? Yeah, I think a couple of

things to think about, right? So one is like speed to market, right? So it is true that if you have

multiple modalities and the same amount of data, you have a much more performance system, or you

can get to the same level of performance with less data. So I think that's one thing is, like,

we can get to a product which is extremely good, much faster. I think the second thing is, like,

that the, the cost of these sensors has really come down massively. So I think you could also

argue, like, why not LiDAR? Why, why actually have a camera and not a LiDAR? So I think there's,

like, there's some asymmetry to, to the argument I hear there, which actually doesn't fully make

sense to me. I think it's assuming that LiDAR is this, this $10,000 spinning thing that breaks down

all the time. I mean, that's not true anymore. That was true 10 years ago. And the world has

really moved on. And LiDAR is really now in that price point where it's similar to a radar.

So to me, it's, it's, you know, I would argue, like, why not put it in there,

especially if it's getting you to that product faster. And the final product has more redundancy

and it's going to be safer and allow that kind of eyes off experience.

So, I mean, I, I mean, you could drive by like covering your eye, but you're a better driver

if you have two eyes. I think it's the same, essentially the same argument.

Yeah. I mean, the way I look at it is that you get, you know, the edge cases get safer. It's like,

it's going to just work better in low visibility and weird lighting situations than cameras will.

And this LiDAR data is important for the ground truth. So, you know, all of these OEMs

have LiDAR based ground truth vehicles, right?

Can you talk about, because ground truth, and I'm sorry if this, if everybody knew what ground

truth was, but when, when they dropped, oh, here we go. Here's, here's the truth right now.

Yes, you're rolling and we're going to capture it anyways.

You're going to get a grill rolling coming in. Yes. Yes.

Yeah, absolutely. I'm sizzling.

I'm trying to, I'm trying to, should we mic him up or are you going to just roll with it?

Okay. Well, we were doing, we were doing great. I just asked him why LiDAR?

Yeah, I was asking. So, I think you, you covered it, but I, I struggle with how good

FSD 14 is. We've been driving it and it's got seven cameras. You guys have 10

and five radar and I will LiDAR. That only points forward. I was a little, I was like,

because you got up on stage and said, we're not taxi cab, we don't got the tiara.

Like, and I was like, oh yeah, this thing's only, the slider is only aimed at this direction.

It's not actually giving you the full 360 degree point cloud, but let's go back. Why,

why all the additional sensors? I know the cost has come down, but still, if the, if the cost is,

you know what you said already, but I can, I can, I can answer it if you haven't said anything.

I did it already. He did, but I, you know, and Ed was like, I think he was suddenly happy.

Yeah. Well, let me, let me, let me give it a try. How are you?

Well, first of all, there's a few things to talk about. So, before we jump into

why LiDAR, I think it's really important to recognize how the model's being built and

how different this is than historically, how LiDARs were used. And so historically,

you talk about LiDARs having turned 60 degree hue and generating this, this three-dimensional

map is one club in a, you know, like think of it as like an 81.0 model. The way that the

systems were designed is you'd had a perception platform, which depending on whether you're

going for level two or level four, it would be cameras and radar or cameras and radar plus

LiDAR in the case of a level four. And it would identify and classify all the objects around you.

And then it would handle all those objects or beyond classifying that associated actors to

those objects. And then it would hand those to a planner and the planner would be rules-based

environment. So, you know, obviously written by humanists that were trying to codify how

what the rules of the road were to say that I drive, is that you drive. And then the system would

make a whole host of decisions around what to do in terms of operation and vehicle.

What the whole world has shifted to and what the, what's clearly going to win in terms of the

approach is what we now often call end-to-end. But what does that actually mean? It's a model

that's trained. It's a, it's a large, very large model that's trained using all the data of the

vehicles that are being driven. Of course, human operating and your training at end-to-end and

running off of this planner that's a rules-based human described environment. In fact, the models,

they have to do a bunch of pre-processing, some are post-processing, but the model's being trained

around behavior. And it's, and it's creating, this is hard for a brain to imagine, but it's

creating a neural net. Like it's creating an understanding of how the world works. And so,

the more information you provide it, the more nuance it can pick up. And so, this is the reason

why you often will use ground-tree fleets to, to add additional nuance to train your cameras. This

is why if you go stand in the corner, you'll see vehicles from, from Tesla that are running with

LiDARs on them. They're not using the LiDARs as production sensors are using as a ground-tree fleet.

And so, our vehicle is a camera-heavy system. So it's primarily camera where I use radars for

additional object detection. The radars also help the model train better. So not only are they there

to identify these corner cases of poor visibility, which are important when you look at the long tail

of safety. Like today, we're driving on a sunny day. So it's, this is not a hard day to drive on.

So you can easily do it today with just cameras. But they cover that long tail of corner cases,

but they also help provide context for the cameras when you're training your camera platform. Like,

is there an object there and it helps you identify, create this, identify what are the

important areas for the, for the model to focus on. In a similar way, LiDARs, which have a non-overlapping

set of strengths relative to a camera or to a radar, of course LiDAR doesn't work well

if you've got things that are blocking its line of sight. I mean, I should stand in front of you,

see, obviously, you block it. But it works really well at providing a three-dimensional image of the

world, which of course, cameras are not three-dimensional devices. This is a 2D set of pixels.

And so you can create deeper understandings of the world that you wouldn't otherwise have,

or you wouldn't as easily have, that you can get with a LiDAR. And so for us, the forward-facing

LiDAR is there because it gets very long range. So it helps cover some of these corner cases for

haspene. I think nitrogen, feeders. Yeah, and it's very helpful for training the model.

And this whole, I think, concern around whether the sensors are confusing each other,

competing. The moment you have more than one sensor, whether it's a camera, a LiDAR or LiDAR,

you have to deal with the fact that you're going to have different cameras or different sensors have

different vantage points, views, or perspectives, and therefore have noise. And in a late fusion

process that's an issue, but in an early fusion process, we're feeding it into it like into a

neural net. It just is a, it doesn't really apply in any of the same ways that we would

classically think about it. And so all that being said, we absolutely, like, and we shouldn't

try to witness, we absolutely are taking more perception heavy approach than Tesla. We have

more, as you said, seven versus 10 cameras, LiAR versus no LiDAR, or Marcus LiAR, Tesla no LiAR.

But it allows us, we believe, to train the model much faster with fewer vehicles

and then to have a higher ceiling on the performance capability

with better corner case coverage. And so that's not in any way of you saying something negative.

I thought the FSD 14 is very, very, very good. It's super impressive what they built architecturally

in terms of building a neural net based approach of those end to end model approaches be very much

aligned to that. In fact, they're very similar in that way. The difference is if you saw the

chart genes had on the front app before you go to the transformer, raise the coding,

we just have more stuff feeding into it. Right. And you have to do that because

they've had a longer head start on the training of their model. They have six billion miles.

There's an important distinction here, which I do want to make. I think we often confuse,

we often overlay time as being equally valuable. So if you're working on a 1.0 architecture,

like a few walls based approach, it's not really that applicable to something that's

AI central approach. And so you actually have like, and this is true of Tesla. So if you look at

autopilot versus FSD, they're completely different model architectures, right? That autopilot is

their historic platform. And FSD is their end to end AI central platform. They're not, they're not

FSD. All the pallets, it's asymptotic to what it is, asymptotic to what it is. And so in our case,

we took the decision in the 2021 to get in 2022 to shift all effort on a complete queen

ship approach, which is our Gen 2 platform, which is an AI central approach. But it's going to get

better with better sensors and better compute. But the model doesn't go away. I'm going to try

to use the analogy if I learn to drive without my glasses on, I would be a less capable driver.

If you suddenly gave me dropped glasses, I wouldn't forget all my skills, right? I would become

better at understanding complex situations and seeing nuance that's hard to see with poor vision.

I meant if you could put radar on my head so I could see if you're fog or seeing the perfect

dark of a night, I'd be even better. And if you have LiDAR that allowed me to train my own

perception models in ways that I can't today, I'd be even better. And then if you said I want to

increase the compute capability of your brain by 20x, something I would do things that are

hard to imagine, but I'd start to notice patterns that I don't see today. You know,

if there's certain things happening in my peripheral vision, maybe that may mean there's

something they need to change or do different. So much more head room, much more potential.

I think that's a fight. Yeah, so yeah, thank you for one question.

Before it leaves us, I don't want to ask this RJ, but is the news here, and this is where I

started before you walked in. Ground truth is sort of a revelation to me. Can you first of all explain

the concept of car manufacturers and ground truth, what it is for our totally ignorant

audience? We're ignorant. Every OAM needs to produce what we call a world model, right? So

I mean, all that is essentially where are the objects around the vehicle? And even in end

to end approaches, they still need that perception signal. It's used for active safety. It's typically

actually used as some of the supervision, even in a more end to end approach. And so typically the

way you do that, because you don't get the depth from a camera bow system, or it's still not clear

often in the camera plus radar based system, just alone, you overlay it with LiDAR and then you can

sort of see the edges of things. So you basically, traditionally, you'd have people like literally

draw in boxes. There's a car there, there's a human there. And so that's what these fleets

traditionally have done, right? So they drive around, they collect this data, humans are annotating

them. You then use those labels to train systems that just use the production senses to produce

those objects. So that's kind of what the ground truth. That's ground truth. That's the traditional

ground truth. Traditional ground truth in every car manufacturer that is attempting anything ADAS

related, anything with cameras. Will have these fleets. Or do they contract with other companies

that do ground truth? I think there are certain, yeah, we have an internal channel where people

post photos of like all these different vehicles. And I remember seeing this, I mean, I remember

seeing this as a kid, my dad had worked for this robotics company, and they were doing

something for the Air Force, and they had like early radar, but yeah, we just put boxes around

like dots on things. Yeah, exactly, right, targets basically. And then I saw like Ford in like 2008

or something, they were tech day at Ford, and it was like slightly better looking. And that's when

Ford had like, what's it called, you know, when you're backing up and it slams on the brakes,

and they were showing, yeah, there's a name for it. It doesn't matter. They were showing boxes

drawn around things. Right. But hang on. So ground truth teams started essentially

early 2000s, kind of the, is it the same time as Google putting cameras on cars and driving around

town for mapping? Does it come from that? And basically, I'm trying to make this real for

the people at home, right? When you see a car on the freeway, and it's got four or five spinning

arrays. So if we really go into the history of it, I mean, I think it was actually Daimler,

and a German professor called Dickman's, very in the actually 80s. And they built some of the

very, very first autonomous vehicles, super early tech, you know, they were, you know,

barely even figuring out, you know, how to build any of this stuff. And he drove vans from, I

think, you know, they had one Berlin to Munich. This was back in the 80s. So that was really the

start of this. Even back then, there was a desire to, you know, we need to know where objects are,

because that's, you know, the most basic thing about driving is you need to know where the lanes

are into where the objects are. How do we know that? So, yeah, so this idea of needing ground

truth. So then is the news, the news portion that when RJ got up and spoke about ground truth and

using the fleet with LIDAR to help, is that the industry first? Are you the first manufacturer

that will be using consumer vehicles to help inform? Was that sort of the... So in the west,

yes, that'll be true. We'll have the largest, you know, LIDAR-equipped consumer fleets

in North America, for sure. Okay. And meaning that in China, this has happened already?

Yes. Okay. Interesting. Okay. And in China, you know, these three modalities, that's not even

really a controversy anymore. Yeah. When you go there, you see all sorts of difference. We talked

about it, like, because we talk to these Chinese manufacturers and they love, it's hilarious. They

send us the spec sheets and they are bragging about the processor inside. They love telling you

they got NVIDIA or they got Snapdragon and they'll tell you how many radars they got,

how many LIDARs. In the halftime, you're like, what do you need? What do you got for them?

You know, we're future-proofing. Right. This is ridiculous.

So what it means is that we get, you know, we get this incredible coverage from that fleet.

And now whenever we see an edge case that, you know, and we can go into some of the ways we

actually would find those, but now we can also, you know, label the perception data in that case.

Right. So it's very, very powerful. So it means that all of that data, we can immediately,

you know, have full label coverage for it. Okay.

So it's not only the driving scenario. We can also do, you know, the perception piece and

everything else. And it just is, it gets a much more powerful data loop that you can build when

you have that ability. So you'll be the, you'll be arguably the first manufacturer. I think you

will be the first manufacturer if this comes out per your plan to deploy LIDAR in wide use. You

were not supposed to be, because Volvo was supposed to have a bunch of cars running around

already. Why has LIDAR been such a challenge for North American or Western?

Yeah, for the industry.

For the Western automobile. Like, what's, what's...

What's tricky about LIDAR?

It's hilarious that Volvo released this vehicle and they put a big blister on the top and they said,

it's there. It's not active yet. We'll turn it on at some point.

And then China has been using it for years.

Nope, we're not doing it. We pulled out with the supplier. We've broken up with that arrangement.

What, why? Like, why is LIDAR still such a... You say it's settled law essentially in China.

Why is the West still having problems with like rolling out LIDAR in vehicles?

I don't know.

If you don't know.

Honestly, I don't because it's a puzzle to me because LIDAR has been used in the L4 domain for

so long, right? So it's, it's not a new sensor in that sense. So I think what is new is the

resolution of it and obviously the cost point of it. And now you can actually bring it to

when you say L4 real quick, you mean like Waymo.

Yeah, the rubber types.

We're also talking about the giant spinning, the spinning things that everyone's like,

there's no way those are coming to a consumer like, like personally on vehicles because...

Yes, because apparently putting spinning things on vehicles is impossible.

But anyway, you try it.

Well, I can imagine my five year old going up and going, what's this? I've watched this,

you know, and that's...

Yeah, yeah. But, but yeah, these, these, the lightest we're talking about, you know,

they're different. They're obviously, you know, have gone through several generations and they're

really, you know, very robust and, you know, capable for their consumer vehicles and all

that. And, you know, the stuff, you'll see in the real world.

All right. I'm sort of sold, I guess. I, you know, I was, it was curious, I'm curious as to

in the time that you've been developing this, which I think RG reference 2021.

So that's the same time as FSD has been launched and we were been super vocal,

super critical about like V12 and how terrible and dangerous it's been in some situations.

But it's gotten really good lately.

What Ed's trying to say is, does your ability to solve edge cases with LiDAR and radar justify

the $500 extra hardware costs that R2 would have that a Model Y won't have?

To me, I think if it accelerates our progress towards getting to those state-of-the-art features,

then I think it justifies it. And I think, you know, if you look at it as a proportion of the

bomb, the overall bomb for R2, it's all still very small, you know, this one.

Bomb is bill of materials. Bill of materials. I'm learning these phrases.

So, you know, for all of the benefits you get and then the headroom you get,

you know, it does, it does feel like it's, you know, the time is right now for putting this into

and, you know, if it had been a few years ago, the answer would probably have been different

because the price has come down so much more in that time.

And I also argue like, you know, you know, these edge cases often involve like hitting people.

So if you can like prevent that, like that, how do you put a price?

Yeah, there's way too many, you know, accidents in the US and

yeah, better sensing and better active safety can really move the needle.

I was, again, a pro at this Instagram post and people are like, I would never like

little robot driving around and I'm like, you know, there's 43,000 people die every day,

every year in the US in non-autonomous vehicles, fully human driven.

So like, you know, if we can improve on that in any way, that's a good thing.

I don't know if this is in your, you're not, you're not a car designer, but the last question

just on LiDAR before, I want to jump into AI with the time we have remaining, but

we saw the sensor array, we saw where you put it and for those of you listening at home,

it is above the mounting point of the rear view mirror and where the two,

Rivian has 10 cameras, there are two forward facing, it's above that array.

It's in the roof.

It's in the right above the windshield, the border of the windshield and it's,

it's recessed slightly. It's an A surface, meaning it's the top, it's the top surface,

but it is not, I thought it was going to be, I thought when they showed it, it was actually

covered, like it would get a nice sweep of the windshield wiper, but it does not to clear

foreign debris. Like, can you just talk about, and again, RG got up on stage, mentioned the,

the blister, the taxi cab, like signal bump, which is sort of what Volvo has, and then the

TRO, which is like the crown of, of, of sensor arrays at a lot of the, the level four, Google

Waymo, if you're not familiar, right? You guys have put this thing now in this spot above the

windshield and it's essentially the 180 degree. I mean, it's 120 degrees. 120 degrees. So it's,

it's not 360. It is a forward view. I get it. That's the direction of travel. Most people

go in, but like, why, what were some of the other concerns? What about things like salt spray?

I'm off-roading. I get mud on it. Getting mud on it or bugs, bugs building up in this high

velocity, high, you know. Yeah. So, yeah. So one of the benefits about putting it up there is

actually you, you do minimize those types of things. So arrow because of arrow and because

you just, you just hire off the ground. So, you know, stuff that's like splitting up, it tends to,

tends to not go that high. And so we, we did a lot of studies on this, actually, you know,

looking at like fascia positions behind the windshield, you know, up in the roofline,

and that, that definitely seems to be kind of the sweet spot. You're also, you kind of want the

sensor higher because then you're above stuff and you can see further, right? If it's always

occluded by the vehicle in front, it's actually not that useful. Okay. I see. You got to get it over.

Okay. Yeah. And then what about, what about maintenance? Will there be an alert in the

car to tell the owner that you need to clean it? They need to wipe it. Yeah, exactly. Yeah. So

actually we, you know, something we built in Gen 2 already is basically like blockage detection,

we call it. So it's essentially every sensor has, you know, another neural network that runs and

actually is telling you, okay, is it blocked? Is it a smear? Is there a bug on the thing?

And, you know, we put that together. And actually a lot of those kind of minor things, we still

enable all the features, but when you get a more significant blockage, then we will tell you,

okay, you need to clean your sensor. And they were mentioning, it meant just for the radars,

but there's some kind of heating. So if something's like ISIS frozen on it,

yeah, does the LiDAR have a heating element? The LiDAR, I actually don't know the answer to that.

We can get you down. All right, let's change gears in the time we have left and talk a little

bit about this, the AI, the Rivian assistant. What is, is that Rivian unified intelligence? Is

that what it's based on? RUI? Did I hear that? Yes, I think so. So I think this is more of a

SIEM's domain. I can talk at a very high level, but I think...

You're the VP of autonomy and AI. Oh, the assistant is not...

Yeah, the assistant is on the SIEM side of the house.

Oh, but isn't it AI-powered?

It is AI-powered, and we collaborate with that team. But I think in terms of the,

his teams own the overall kind of UX and the interactions with customer-facing stuff.

That's a good, that's a good clarification. Well, then what can you tell us about

the AI use in the car, not associated with the assistant? What else is AI doing?

Yeah, yeah. So I think we've seen showed some pretty compelling demos today,

showing AI, you know, interacting with the navigation, messaging, which has been a huge

that customers have been talking about for a long time. And then sort of not just handling the

normal use case, but actually interacting between, you know, so one of the examples he

showed was querying the map for restaurants, sending that context as part of a text message.

So one of these things where you're actually, you're sort of cutting through something that

on other systems would be, they're kind of like a silo, right? You'd have a text messaging app,

and you wouldn't easily be able to get the text out of that app to sort of use it in somewhere

else. Getting map data into a text messaging app. Yeah, yeah, yeah. So yeah, that's this kind of

unified intelligence layer where you can, you can sort of flexibly move inspiration around and

how you have a lot of context when you're doing those types of queries. So I think that's one

thing. The other thing that I think we've seen touched on a little bit was, you know, how we're

using AI to help with service. So we can do sort of more intelligent prognostics and diagnostics

of vehicles. So maybe finding problems before you can even tell they're there.

Give us an example of that, like a battery cell or something?

Yeah, so I think, you know, 12 volt battery is one where, you know, we can actually monitor the

voltage and try to tell if it's reaching end of life before it becomes an issue.

Just replace mine, yes. Yeah. Prognostics is a big, it's a big deal on the commercial side. If

you can, you're a fleet manager, you know when your car might go down, you can do the proactive

maintenance before. Yeah, much better to get the vehicle in before you have a problem, right?

And then after you have a problem. And so the AI would help with that. Yeah. And the way we train

that is, of course, you, you see service cases, you know, that was all fit into the model and the

model has those, you know, those examples that it can pull on. And with this fundamentally

change, and I don't even, this is horrible as a Rivian owner, but I don't even know if there's

like a service schedule if it's in terms of mileage or time. But would this get to the point

where it's like, don't even, there's no set schedule, just we'll let you know when there's

something up, bringing in for service, would it ever get to that? Or are there still like,

I hit the scheduled service intervals? Yeah, I think, you know, in terms of like proactive,

like, you know, service schedules. Yeah, I think we've actually started to roll out those things

for some service items, like the 12 watt battery. Yeah. Okay. Yeah, because I did get an alert,

like change it, you know, yeah, exactly. Yeah. So I think that's, that's that part of one of those,

you know, there's like proactive. Got it. Okay, so let's go back then. I thought we'd have more

on AI, but let's, let's talk about, I want to go back and talk a little bit more about autonomy

and also your background. So you, how long, how long have you been here in Rivian now?

It's three years, I guess you couldn't believe it when someone asked me earlier.

So three years, where should this will go up? Three years in, yeah, three years. Three years.

So this might go in 2026. I don't know, Brent, when are we, when are we running this episode?

Probably later, could be right in the, right in January might be. Okay. So three years,

so call it 2022. And prior to that, you were Waymo. Yes. And prior to that, you were at Zooks.

Okay. So those are two sort of L four commercially, like Robotaxi services, right?

It was very clear when you said L four, that it would be personal L four,

meaning autonomous vehicles that are personally owned, not a service.

Rivian, not nowhere on the roadmap are going to do a Robotaxi or a Waymo type service. You

have all the technology and the big unlock. You know, I remember I've been looking at this stuff

when McKinsey wrote a report, they call it case, right? Connected, autonomous, shared electric.

This is supposed to be the major game changer of EVs is when they're autonomous, they, you,

you go to sleep in your cars, taxi, taxi, make you money. It's not just sitting in your,

in your driveway, losing money. Like if you have all the sensors, if you have,

if you have all the ability, you got a neural network. Will we see a Rivian Robotaxi?

So I think, yeah, first of all, I'd say like, we have all the technology pieces, as you, as you

said. Now, my personal belief is that actually, although Robotaxi will be a significant market,

it's a road sharing, at least in the US, people are not going to be getting rid of their personal

vehicles. People vomit in taxis and worse, all the time. Yeah. And I actually think there's like

just certain geographic realities in the US that make having a taxi service cover, you know,

all of these suburbs in an efficient way, actually very, very challenging. So, you know,

and I look in the back of my car and it's just full of crap from, you know, my family. I have

like car seats in there. I have like stuff for the beach. I have, you know, some hiking boots.

And so it's just like, you need to be willing to give up all that and do everything through a

ride sharing. So I think to me, actually, what will happen is people will demand higher and higher

levels of autonomy from their personal vehicles while they're also using Robotaxi for all the

times when those things are great. And so I actually think that the tide of kind of autonomy

expectations is really going to rise significantly. I think we actually already are seeing this in

terms of when we talk to customers in the sales spaces, you know, what's what are the key buying

purchase decisions. So you're saying that the owners, you don't see people giving up personal

cars to use exclusively Robotaxi services, whether you deliver one or whether it's somebody else.

That may happen a little bit, but I actually think so people have in mind that,

you know, Uber is filled with a lot of personal vehicles. That's actually not

true. And that hasn't been true for many years. The majority of Uber vehicles are fleet vehicles.

And so for me, it's what would be different about an L4 vehicle that would make that equation

different. So the truth is like, you people don't want to be cleaning out their car every night.

You don't want to be dealing with the crazy insurance you'd have to cover just from people

coming in and out of the car all day. You don't want to, you know, just think of tire usage and

all the other things. To me, it's actually like a not, it's an, it doesn't really make sense

economically. And I think, you know, if you look at, you know, an Uber, where they're using all

these fleets, you know, why would the equation be different? Not of us making you literally

hundreds of dollars a night, John? Hundreds? No. Yeah, exactly. That's what's the cut.

But just so everyone knows, like, so at the presentation today, RJ said, like,

L4 personal level for personal meeting, like, you could sit on the couch and doom scroll while

your car drives your kids to soccer practice and then go runs another errand and then comes back

because it doesn't need a driver. Right. That's very compelling. That's like, that's cool. Yeah,

that would be amazing. Yeah, it'd be, again, someone who spends their weekends shuttling kids

around. If you ever sat in a soccer practice with little kids, nothing worse. Yeah. That's,

that's okay. All right. That's, that's interesting. I hadn't thought about it that way. So, okay,

points to that. Well, yeah, because I heard that, I'm like, that would be really, really cool. Now,

yeah, if you wanted to do that, you know, you could, but there's the sort of extra things you

would have to put in the vehicle to handle that. So for example, you need to, you need to deal with

what happens if people have left stuff in the vehicle in the trunk? What if, you know, the doors

are not closed, right? Do you need to auto cinch them? What do you do about teleop? I assume you're

not going to be sitting at home in front of your computer, like helping your vehicle, you know,

get out of a tricky situation. Teleoperation is a requirement to run these things. Yes. They will

always, at least in the foreseeable future. Teleoperation is, it's what a lot of folks are

claiming big company starts with a T, you remember the guy with an E in his name is doing with the

robots, like that. Yeah. That's when they're Warner Brothers or whatever, 100%. I know the

people that are operating. Okay. Let me ask you about your time just briefly at Waymo because,

you know, we've been using the service in LA, it's pretty good. We're looking forward to the

expansion under the highways and all the way into LAX. I think they're test running it now for

Waymo, the SFO for, but employee only. We've also heard some interesting cases of maybe the

cars running over dogs and cats, blowing through. This is the good one. Yeah. The one where there's

a full on police pursuit and they had the guy, the perp down on the sidewalk. And the police

had their guns drawn. All the lights and tyrants are going off on the police car.

Just like right through it. And not to bag on our Waymo friends.

I like Waymo. Waymo's great because, you know, no one talks to you. It's the best.

And actually, sorry, the third one is stopped school buses where the stop sign comes out,

the arm comes out and everybody has to stop. And like, apparently Waymo's are blown right through

it. Do you consider those edge cases and how do you plan for them? Is this

the kind of stuff that keeps you up at night or? Yeah, I think, you know, I would definitely

consider some of those cases edge cases. Now, you know, like as the fleet expands,

you know, you go ever further into the tail. So, you know, the perp on the sidewalk maybe

doesn't become an edge case at some point because you see so many models. Yeah. So,

so I do think, you know, as you scale, you are just, you know, ever more pushing into that long

tail, which is why I think that the building up this, the scalable data flywheel, which can always

kind of cover more and more cases is like so important because otherwise you're always chasing

that next, that next edge case, next edge case, next. It's never ending as these fleets grow.

Whereas if you're building the systems that can scale to handle them, you know, through systems

that are somewhat automatic, that really gives you the sort of superpower to scale up quickly.

And I also wonder too, like, you know, yeah, okay, the Waymo drove through that,

like how many humans have done the exact same thing, like probably like thousand or more than

that. And it's just, it's like, you know, it's like when an EV catches on fire, you know, it's

been like five or six, and it's really well publicized. And we looked it up, 120,000 vehicle

fires in the US a year never get a drop of ink because they're not EVs. I've seen many more

ICE cars on fire.

Yeah, no, I mean, it's not even close. I mean, it's literally like a dozen versus 120,000 or

something like that. And so I wonder, like, you know, with the Waymos, like, if they, you know,

like, I know lots of people that have hit dogs and cats driving cars, you know, and I know lots

of people that have killed people in cars or died in cars and, you know, I mean, I know of them.

And I know, yeah, I know a couple. Yeah, sure.

All right, let me, I'm just, let's, let's, let's jump to, what is it, two years in the future.

And whatever, whichever, assuming all great success, and you got our two's out there,

and they're running on the Gen three, and we have universal hands free.

No, no, no, I have eyes off.

Do you have eyes off? Will the, will you allow me to eyes off, hands off on the freeway

that's marked at 70 miles an hour go go 80?

Can I go 80, can I go 80 85, which is flow of, which is flow of traffic?

So I don't know. And so the reason I'm being a bit evasive is some of this

is actually controlled state by state. And so to get the, the permit to operate an L three system,

there's certain things you have to do. And they're actually differs in California from Nevada,

from other states. So in some states, there probably is no option, we would have to follow

the speed limit. In other states, you could be more probably more flexible.

It was there, I was saying, is there like a waiver? Like, look,

even though you're driving and you have the liability for an accident, I'll take the ticket.

Yeah, it's a, or is that just too much? It's a regulation. That's the thing. So you,

there's no, there's no dodging around it. But even if it flow traffic, like when I can make the

argument that like, going the speed limit, if every other vehicle on the road is going 10 over

the speed limit, it's less safe to, you know, what would happen is we get pulled over, you know,

you get a ticket, it would go to the DMV in Sacramento, I would get a call probably the

head of California DMV and say, Oh, why are you speeding? We're going to revoke your autonomy

license. But again, but every, it's just one of those laws. It's, you know, it's like,

everyone speeds like nobody goes. I encourage you to pick up the phone and call your congressman.

Let me ask you, let me ask you a variation of this question, because

Tesla FSD currently allows you to do that. And I don't understand how, do you have any understanding

how they can, they can allow, they can allow a bad max mode, the vehicle to overspeed and also

jump in and out of an HOV land crossing. But remember that system is still an L2 system.

So that's because I chose to do that. And it's, and your eyes are on.

Interesting. And L2 means that the driver has the legal liability, not the OEM, not Tesla.

So yeah, that's how. Let me, let me ask you my last question, because we're, we're basically

running out of time here. You got your start, I didn't go all the way back, but you spent a lot

of time at Google, you were doing computer vision and machine learning algorithm, image annotation.

You did Google goggles, image search, photo search. Did you think, did you have an interest in cars?

Did you think you would be doing this in what I think it's like 10 years later?

That's a good question. I, yeah, I always thought it was a fascinating problem. Like,

could you teach a vehicle to drive itself? But it just seems so far out of reach at that time.

But then, you know, you start to see, you know, I would say, you know, I did my PhD in, you know,

computer vision and at that time, the stuff barely worked, right? You, you have, we all have stuff

on our phones now, which are just so much more advanced than even what people were just could

dream of at that time. And so I think, you know, as the machine learning computer vision sort of

started to become more mature, that problem felt like it was coming into reach and was like very

exciting because vehicles are really robots, right? They actuate their wheels, they have controls,

steering wheel pedals, they have sensors. And they're, unlike typical robots, which, you know,

at least at that time where they would break down all the time, you know, you'd have like a lab

robot and you maybe get 10 minutes of it doing something and then and you have to, you know,

then it's in the shop for the next few days. Vehicles were very, you know, have been very

reliable because yeah, they've gone through all those iterations. And so that's what made it a

super interesting problem because it's a, it's a robot where the hardware reliability is already

there. It's a relatively simple control problem. It has an amazing set of, you know, use cases

that can really change and improve society, I believe. And so, and then to have these machine

learning techniques come within grasp, it really felt like that was the time.

Did you work with Sebastian Thrun? I didn't know. So he was a, you know, chauffeur and then he left

and then yeah, I went to Zooks. My question is what changed? Because I remember there was like a

BMW technology day. It was right when like Wall Street was obsessed with autonomous cars. You

was going to go autonomous first. It's all Wall Street cares about. Yeah. And then I remember

there was a BMW technology day and they were like, forget about it. We have a thousand engineers

working on this problem. It's never going to happen within the foreseeable future. Like it's

just too complicated. And that was, I feel like in my mind, that was like 2017, 2016, something

like that. But like today, you know, we took a trip, a point to point trip where the guy was

hands off 99% of it. And, you know, admittedly, it's not the finished product. But like what

changed? What was the big like breakthrough? And there's Waymo's driving around. I like taking

Waymo's, you know, yeah, it's a great product. Yeah. What was the change? What happened? So I

think, you know, a few things. So I think there was a huge revolution in ML and AI. So that's one

thing. And that's been, I mean, to consumers, it's maybe only visible the last few years. But

versus being getting better and better and better and better and better. So that's one thing that

Wave has been writing. Is it a hardware advancement, a software both? A bit of both, right? So I

think it's the techniques that were being used. It's the fact that, you know, GPUs came along that

could actually train these things very quickly, much more quickly than you could before. It was

the data as well. So just the massive scaling up in data. You know, when I started my PhD,

you could have a paper that maybe would have a data set of like a hundred images that was considered

a big data set. And now it's like billions, right? And so that's also scaled up. And so as

everything has scaled, these techniques have got better. And then, you know, more recently,

large language models, transformers, everything is sort of just compounding on itself. And then

on the vehicle side, you know, the fidelity of the camera data, you know, having the radars have

gotten much better. Lidars have got incredibly better and so much cheaper in 10 years. It's

same time frame. Did you video which there's a massive dependency on? Absolutely.

Well, so let me let me just do this one last question. Like, what are what are you seeing in

the labs that we're not seeing that's going to like make us real happy in a couple of years? Like,

what's the next like, you know, like besides personal level four, like, you know, what are

not just ravines, but vehicles going to be able to do that they can't do now.

I mean, I think one, yeah, so I, you know, I have, you know, daughters 11 years old and, you know,

at some point, they're going to want to drive. And so something I'm very much looking forward to

is a vehicle that has a safety bubble so good, it can basically not be crashed. So you can, you

know, I still think people will want to drive and they'll need to be trained how to do so. But I

think that the these world models and these AI models could get to the point where you essentially

can't crash your vehicle. You try and turn and it's like, no, no, I can see there's a vehicle there.

You try and speed, it's like, no, no, we're not going to do that. Don't do that in the teenager

mode, right? So I think that's that would be kind of awesome. Well, my last question,

and I want to get you to commit on this is when gen two comes out and it's got all the bells and

whistles. Can we run the Rivian Transamerica Trail fully autonomous? This is on this is not a paved

road. Okay, and it's unmarked. But could you do you have, do you think we could attempt that?

Is there is it within the realm of possibility in your law in your LDM and your large?

That's interesting. It what is it? I don't actually know what it looks like. Is it just

this thing? So we cross the country. Yeah, so we are marked at all. It has that starts in South

North Carolina. Before you were at Rivian, we teamed up 7700 miles, took us 43 days,

all off road the whole way. Yeah. And we try it at least. Yeah. Yeah, to do what, let's

we'll get some data and then let's let's talk. Let's have a yes. Because it's really fun and

we're done. Yeah, great to meet you guys so much. Thank you so much. Cheers.

Rivian’s AI Master Plan: Hands-Free Driving, R2 + LiDAR, and the Road to Personal Level 4 Autonomy

About this episode

GPUs

microchips

sensors

Rivian R2

lidar

autonomy

AI

Rivian R1S

Proton Gen2

Cadillac Escalade

Tesla Full Self-Driving

ground truth

world model

Ford

Daimler

autonomous vehicles

Volvo

L4

Waymo

Tesla Model Y

sensor arrays

level four

prognostics

12 volt battery

proactive maintenance

CASE

EVs

EV

ICE

L2

autonomous cars

BMW

Transamerica Trail

Request an Explanation

Want to learn more?

Help improve this episode

More from The InEVitable