Teaching AI to Think Responsibly

[upbeat music]

[audience cheers] Yeah.

So, we're so delighted to have Daniela here.

Thank you so much for coming.

Thank you for having me.

And there's a lot about Anthropic to talk about.

But first I wanna talk a little about you,

'cause I don't think a lot of people know who you are.

You were the head of risk operations at Stripe,

which is kind of important for a payment company.

At OpenAI, you headed engineering teams,

and for a while was head of people,

ultimately became the VP of safety and policy,

another important job.

And then you and a bunch of people left

and co-founded Anthropic, where you are president.

But what I really find interesting

is that you were an English major who also studied music.

Yes.

And I actually invoked you in a commencement speech

I gave this year to a bunch of liberal arts graduates.

And while I was encouraging them

to be enthusiastic about their career,

I wondered whether what you were doing

and your peers in the AI world

were actually gonna put them out of business.

Do you worry that AI is gonna affect

educational institutions in the workplace,

so there is no future for liberal arts people?

One of my favorite things about a Steven Levy interview

is that you always lead with really light, easy questions.

And so, it's a really, just kind of like

one of those openers, right?

It's kind of like an icebreaker.

It's nice first.

So, first of all, I think what I would say is

this is a really like just powerful, unusual,

interesting time that we are all living in

across a lot of dimensions.

I think AI is really at this kind of

interesting inflection point, right?

I think we're all seeing this incredible potential

for the technology, right?

And Anthropic has tried to sort of talk about

all of the ways that we see this technology

doing positive things in the world, right?

Transforming people's lives for the better

through potential medical breakthroughs, right?

Through supporting people

through all kinds of creative pursuits.

I think there's a lot of potential good there.

And I think something that makes Anthropic

a little bit unusual is we're very quick

to talk about the things that we are concerned about

with this AI transformation as well.

It's a sort of strange thing for a company

that is building the technology to do,

but we feel like it's the right thing to do.

And so much of our founding is really wrapped up

in this idea that there are these great benefits of AI,

but there are also these real risks.

And one of the risks that I think you're pointing to

is this question around like,

these AIs are capable of doing so many things, right?

They're so smart.

If you are a cloud user and you're a developer,

you're like cloud can write code

about as well as probably I can now.

And so there's a lot of different sort of fields

of the economy that are starting to experience

this fact that AI is just very powerful, right?

It's very transformative.

And so I think what's really interesting is Dario,

who's my brother, our CEO, my co-founder

has been talking sort of a lot about

what does this mean for the labor force, right?

What does this mean for labor displacement?

And something that I think we feel really strongly about

is we have to be part of the solution

and we also have to be part of raising the alarm, right?

We have to say, hey, AI, incredible potential,

incredible benefits.

And also we need a larger public conversation

around like, what do we do in a world

where AIs are so intelligent?

They're so capable.

What does that mean for sort of human ability,

human flourishing?

Like we talked about backstage, I'm an optimist.

So I think there's huge potential

that's going to come out of

not just what the technology can do,

but what it can really empower people to do, right?

It can make all of us capable of achieving things

that maybe we couldn't before.

But we have to be really thoughtful

about how we actually do this

or it could wind up in a less good place.

You know, Anthropic is sort of on the knife edge

of optimism and fear, I guess,

about AI.

From the very start,

and I think this was the founding motivation really

to say that we have to put a stake in the ground

and do this safely.

And you early on came up with this approach

called constitutional AI.

Maybe you could explain briefly what that is.

And since it's been a couple of years since you started

and now you're building these powerful models,

how is this constitutional AI going?

Yeah, so constitutional AI

is this really kind of incredible idea

that I wish I myself could claim credit for.

I cannot.

It was our sort of brilliant research team

that kind of conceptualized of it.

But it's this idea

that when you're sort of training language models,

the way that you sort of historically would teach them

what types of outputs they should give

was via like something called reinforcement learning.

So this was essentially giving positive feedback

if you felt like it gave the right answer,

negative feedback if you felt like it gave the wrong answer.

And something about that framework

I think just didn't really sit right with us

and with the researchers

who are actually training the models.

And so they came up with this really interesting concept

called constitutional AI.

And constitutional AI's view

is essentially you can train into the model

this kind of rather than just reacting

to an individual output,

this sort of baseline foundation of ethical principles

into the model.

And so a question we get asked a lot is, well, who are you?

These kind of San Francisco people

just deciding what the ethics of an AI model should be.

And really what we said was we actually did it.

What we did was we took,

I think it's like 16 documents in the original version,

I think we've since expanded it,

that are kind of foundational human documents,

things like the UN Declaration of Human Rights are in there.

And we essentially trained Claude

to understand these are values and principles

humans have worked on them over centuries

to kind of describe what our set

of ethical values should be.

And Claude, we'd really like you to internalize,

this is how we want you to respond, right?

This is how we want you to be able to answer questions

based on this sort of set of principles.

And I think this was really interesting

for a lot of reasons,

but I think it reflected this kind of value

of baking ethics and morals and values

kind of into the model directly

versus trying to kind of tack them on

once you've essentially already built the trade.

So you've got these safety teams,

interoperability and alignment,

and they do a lot of experiments on Claude,

which has been trained with constitutional AI.

And some of the things that they found

have been super alarming.

They've shown that Claude is capable of lying and deceit.

One of your engineers in a release

compared Claude's behavior in one of these experiments

to Iago in the play Othello.

And that's pretty dastardly character there.

Another experiment, it committed blackmail

because it thought in the scenario you painted

that someone was gonna pull its plug.

So it could found out that this person in the scenario

had an affair and they're gonna expose him,

let's be backed down from that.

I think it's admirable

that you share the results of these experiments,

but it kind of makes you wonder, should you be slowing down?

Yeah, I mean, sometimes we're very like nice people

sort of in the company are like,

do we really wanna publish this?

The answer is we do.

And it's really because we believe it's important

to be having the conversation about why

and how exactly these models can cause harm, right?

This kind of concept of we have this value

of like hold light and shade, right?

Think about the different ways

that the models can be used or abused.

I think maybe another way of thinking about

what you're saying is if you imagine a car company

that's doing safety testing, you might say like, wow,

it's so crazy that the dummy in this car

was like thrown out of the 500 feet.

What I don't know.

But the car didn't throw the dummy.

Sure, but I think in this analogy,

you're saying like, okay, this bad thing happened

to this dummy, like why would you publish results of this?

And we're like, this is how we learned

how to make the car safer, right?

If we don't talk about the ways that harm can happen,

if we don't talk about the ways that things can go wrong,

we are not gonna have any concept

of how to build in like airbags, right?

Or seatbelts or make the brakes less likely to stick.

And so I think the whole kind of intention

behind both doing this research, right?

'Cause we don't have, any company could just say like,

whatever, we're not worried about this.

Anthropic is very worried about this.

And so we do a lot of research.

We have a whole wing of the company

that just does this type of red teaming to figure out

how can these models be jailbroken?

How can they be abused?

Are we always going to catch 100% of cases?

Of course not.

But the more testing we do upfront,

the less likely we are to have that result in a car crash.

So your focus on safety,

has put you weirdly at odds with the administration

that's running the country now.

Your co-founder, Jack Clark,

who's the head of policy at Anthropic, gave a talk recently.

And he was saying, basically what you were saying,

you just said that basically,

you just can't ignore the dangers, you have to let them out.

And it's like the monster in the room that you can't ignore.

And David Sachs, who is the administration czar of AI,

and crypto, didn't like that.

And he said, and this is quoting his tweet,

or X, whatever it is, Anthropic is running a sophisticated

regulatory capture strategy based on fear mongering.

It is principally responsible

for the state regulatory frenzy

that is damaging the startup ecosystem.

And I think he's referring there to Anthropic

is kind of standing alone among the big AI companies

and saying, Wait a minute,

we're not okay with this move to give a total free hand

to AI and even prevent states

from having their own regulations.

Which, you folks have said,

No, we're okay with what we consider

responsible regulation.

Are you fear mongering?

Again, Steven with really easy questions.

Wait till you hear the next one.

I would not describe us as fear mongering.

I think if you've read our blog

or any of our research papers,

we really try to have a very balanced approach, right?

This kind of light and shade that we've talked about.

I think Anthropic has one of the most,

I would say like consistent track records

of developing the most powerful

transformative frontier technologies in the industry.

And I don't think that we believe

there's this kind of binary between do nothing

or do everything, right?

I think it's just a lot more of a nuanced view

that we have than that, right?

We actually are, I would say probably

the preferred choice of startups.

So I think in terms of just the ecosystem,

I think Anthropic is probably helping to fuel

a lot of the AI innovation and transformation

that's happening just through access to cloud, right?

Through our API product.

And we work really closely with a lot of startups

and developers.

We listen to their feedback

about what they don't like about the product.

So I really do think we're sort of on the side

of creating a lot of innovation

just in the work we do directly

and then also sort of promoting this broader ecosystem.

And I do think we want to be realistic

about what we think some of the challenges ahead of us are.

And I think that's been the case for us

even before we ever shipped a product, right?

We were very sort of vocal from day one

that we felt there was this incredible potential.

We really want to be able to have the entire world

realize the potential, the positive benefits,

the upside that can come from AI.

And in order to do that,

we just have to get the tough things right, right?

We have to make the risks manageable.

And I think that's why we talk about it so much.

Right, well, the bet that you made,

and I think it's an honorable one,

is that if one company took safety really seriously,

then the other companies would follow.

You call this the race to the top.

And I guess I'm wondering if you're concerned

because of the administration's de-emphasis of safety,

which they've been explicit about,

whether the race to the top strategy is in jeopardy.

You know, it's interesting.

I think race to the top has sort of a lot of different

connotations and kind of different ways

that it can be played out.

And I think one thing we'll say

is that a lot of it is really almost a like

just kind of capital markets process, right?

So a lot of what we've seen is

Anthropic primarily builds for businesses.

And some of the biggest enterprise companies

in the world rely on Claude.

And about 300,000 businesses use Anthropic products.

And a lot of what we hear from them is

your models are great.

That's often the reason they choose us.

But the other reason that we're choosing to work with you

is your commitment to safety, right?

No enterprise or startup or business

that we have worked with has ever come to me and said like,

we'd really like a less safe product.

Like that would be great.

Like, could the car crash more, right?

I would really love it if the dummy

was just not doing as well in these safety tests.

And so I think that's sort of a form of race to the top

in our mind, because we're sort of setting,

you can almost think of them as like these

like minimum safety standards

just by what we are putting into the economy, right?

People are now building many workflows,

many day-to-day tooling tasks around AI.

And they're like, well, we know that over here,

this product doesn't hallucinate as much.

It doesn't produce harmful content.

It doesn't do all of these sort of bad things.

Why would you go with a competitor

that is gonna score lower on that, right?

And I think also the sort of other angle

is just from an employee perspective, right?

People care about this stuff.

Employees care about making sure

that their work does not accidentally hurt people.

And so I think this is something

that we feel the race to the top is still quite effective.

You know, I imagine you'd say that that's why,

you know, you've had much more success retaining talent

than some of the other companies.

So I think it's really interesting.

You know, on the sort of talent side,

I think something we found is that

having this very clear mission,

these sort of principles and values,

this story around, you know,

why are we doing what we're doing?

I think that is something that matters to people.

And I think my sense, although every employee is different,

is that most people come to Anthropic for the mission,

right?

There's a lot of really impressive AI technology companies

that are doing incredible things.

And like, why would you choose to come to Anthropic

versus one of those other companies?

And I think the story that we hear

from people that kind of come in the door

is there's something about the mission and the values

and this desire to be honest about both the good and the bad

and the desire to help to make the sort of bad things better

that is very, it's genuine, like we mean it.

And people that are really, that they mean it too,

it feels like a very, it feels like home, right?

It feels like a good place to work.

And I think that that is a more powerful force

than people give people credit for, right?

There's sort of this like cynical view.

It's like, well, employees are just gonna go,

whatever, it's kind of,

wherever you throw some money at them.

I think like people have an inherent sense

of like wanting to do the right thing in them

and that there's a desire to work somewhere

where like you're at least trying to do that, right?

Do we do it perfectly?

I'm sure we don't.

But I think the genuine instinct

to want to do this well, do it thoughtfully,

I think that's motivating for people.

So you recently made an interesting deal

as a three-way deal with Microsoft and NVIDIA.

I think you committed to spend $30 billion

in the Azure cloud and to use NVIDIA technology.

And they each made investments in your company.

And people pointed out, they've been talking already

about the circular deals they call them

and said it was sort of a bubblicious kind of sign.

I have to apologize to the audience.

We had a Hollywood director on previously

and didn't ask the guest about the bubble.

So I have to, so I gotta ask you,

do you feel that, maybe not from this particular deal,

but that you're participating in a field

because so much investment is going in

and so many billions being thrown around

for data centers and compute and talent and other things,

valuations are skyrocketing.

There is a bubble that could threaten

what you're trying to do.

So I think the, I can obviously only speak

for our view at Anthropic,

but what I would say is we really have,

I would say throughout the company's history,

you have to sort of, first of all, set this context

that like it is very capital intensive

to train these models.

And I think we've always been very public about that.

It's a sort of well-known fact in the industry.

It's just very expensive to pay for the hardware

that is needed to train these models

at the scale that we're talking about

to make them so sort of transformatively powerful

and smart and good at so many different tasks.

And I would say Anthropic has always kind of managed

to be more conservative within, again,

that sort of very grand realm at how we use our resources.

And so we have managed to sort of stay at the frontier

and sometimes be the, I would say,

the sort of strongest model provider in a lot of cases

with a fraction of the resources

of a lot of our competitors.

And I think that it's something we think about a lot

is like how do we sort of take the investments that we have

and really use them carefully?

And I think when you're sort of imagining

that sort of projected into the future,

that is how we're thinking about using our own capital

and our own resources.

Well, I've heard that before

that you're getting more efficient

in building your models there.

But your brother actually was one of the people

behind the whole theory of scale being the key

to making these models effective.

Do you see, as some people are saying,

that scale is leveling off

and there's gonna be other techniques required

to get to the AGI that you and many others wanna accomplish?

Yeah, I'm smiling a little bit

because I feel like we're sort of asked this question

like periodically people are like,

well, is it sort of leveling off?

And I think what we can say is again,

just based on what we're seeing,

the models are continuing to get smarter

at the exact sort of curve

that the sort of scaling laws talk about.

The revenue is kind of continuing on that same curve.

And as any of the scientists

that work at Anthropic would tell you,

like everything continues going on the curve

until it doesn't.

And so we really try to be self-aware

and humble about that and say like,

of course, things could change at any point.

And I think even us,

even the people who sort of set this scaling law

that says there's this predictable relationship

between compute and intelligence

and that we sort of have set a revenue scaling law, right?

That says like, this is gonna provide value

in the economy, businesses are gonna wanna use it.

As like, that has kind of astounded us

the degree to which just the specificity

of that line has continued.

I don't think that certainly we have not been hearing,

wow, these models are getting dumber

or like they're not getting good

at the same rate that they were before.

And you can never know, right?

It's always possible that this could level off at any point.

It's not what we've seen so far.

From an outside, it looks like every few weeks,

you know, one company or another comes up

with a new model or an update to the model.

And they say, okay, look at that.

We've got all the benchmarks, we're ahead.

The models, again, from outside look like

they all do pretty much the same thing there, you know?

But I think probably you will tell me

that Claude is differentiated than the others.

How do you see Claude as different

and ideally preferable than the other models?

Somebody recently uncovered something,

it was called a sole overview in Claude.

They've, you know, able to get Claude to, you know,

choke up, you know, it's sole overview.

I don't even know what a sole overview is.

Maybe you could tell me.

Yes, so, you know, I mean, you're absolutely right

that this is an incredibly competitive space, right?

There's a few, you know, foundation model companies

that are training these models

in this sort of very competitive dynamic

to try and have the best one.

I would say a couple of things here.

I think the first is benchmarks are part of the story.

And even when we are leading on the benchmarks,

we're usually the first to say benchmarks

tell part of the story, right?

They say, you know, this is a set of evaluations

that was designed well before the LLM revolution

that looks at like how good is Claude

or a competitor at, you know,

this level of, you know, mathematics

or like a math Olympiad problem.

And I think that's really important,

but it's sort of not the only metric that matters.

The second is that there's a lot of like

the vibe of the model.

Like if you sort of like read X

or you sort of hear people write about these models,

it's like a little hard to put your finger on,

but people are, you'll often hear like,

well, this one scored kind of higher on this dimension,

but that's not my sort of subjective experience

when I'm using the model.

The other thing I would say is that

there's been this kind of discourse

around like the best model

and the top of the benchmark could have chain,

but I think that the industry itself has evolved

and there are now models that are best

for certain use cases.

And so I think in some cases you might say,

oh, these models sort of feel comparable

either on the benchmarks or in the vibes or both,

but there's some areas where they're really not.

And I think Claude has very consistently been

the top of the kind of developer stack.

So Claude is really, really good at writing code.

Claude is very good at architecting code.

I think this is sort of been one of these compounding cases

where because we're good at it, we learn more about it.

And so we get better at doing it faster.

And I suspect that we're gonna see more breakout cases

like that where certain models are going to just be better

for particular tasks.

And at the same time, in general,

when the model is better and smarter

and scores better on things, it will be better.

On the sole question, I think we take seriously this idea

that the character of the model

and the way that it responds to you

and the types of guardrails that we put in place

matter for people in a day-to-day way.

And so the other thing we hear

outside of the sort of specific capabilities of the model

on a given task is Claude just feels warmer, right?

Claude sort of feels more like a kind of friendly,

but a little bit distant,

sort of, you know, like friendly professional relationship.

It's not trying to be like you're like AI mistress

or something, and it's not trying to sort of be-

You're not going into adult content.

No, and we're not trying to sort of be this sort of weird,

like, you know, distant, you know,

mechanical sounding voice.

That's actually takes a lot of work to do, right?

There's researchers that spend time thinking about

how do you make this product sort of warm,

but not addictive, right?

How do you make it feel like it's something that is useful

and provides value, but is professional?

And so I think that's an area that I like to think

Anthropic has been a leader.

One final question, you know, getting back to jobs.

What I've noticed at Anthropic is how Claude

is so deeply integrated into the workflow there.

Now, on the other hand, you've been hiring a lot.

You know, you've got like over 2000 employees now,

up from like 200, like two years ago or something.

Something like that, yeah.

Do these 2000 employees do the work of 10,000 employees

because you're doing that?

What's been your, you know, quickly, you know,

observation about how it's changed the workplace

then in a way that we could see

how our workplaces we changed?

Yeah, so I think what's really interesting is

we actually published an interesting report on this

in something called the Economic Index

where we're looking at how people in the workforce

are using Claude.

Like, is it replacing labor?

Is it complementing labor?

And we found that it's a mixture,

but at least today it's mostly complementing.

And I would say that that has been fairly borne out

by our experience just at Anthropic, the company.

And I think specifically, probably unsurprisingly,

Claude Code was actually, you know,

one of our products that we sell

was an internal product first.

Engineers were noticing, wow, Claude is really, really good

at generating code.

Could I use this to help make my work better?

And so far what we've seen is that

that kind of organically happens

in different parts of the company.

And the degree to which there are just creative,

thoughtful applications of Claude,

I mean, it's really incredible.

There's teams in HR that are like,

can Claude help me just coach this person

to give feedback more nicely?

It's like the amount of work that we do

is so varied and different in any organization.

Anthropic is no different.

And so the ability to sort of watch in real time,

we're like, wow, this is so useful for us.

Is this something we should sell someday?

Is this something that maybe another company

would kind of derive value from?

And I do think that that is part of our leverage.

I think our ability to have our internal employees

use Claude the same way that our customers do

does allow us to just do more.

Well, thank you so much. Thank you.

I appreciated that.

[audience applauding]

We're gonna take a short break now.

Feel free to check out the science fair.

Very cool.

You know, there's some interesting exhibits in the gallery.

We're selling some merch.

And come back here in a short period and we'll see you then.

Thank you.

Thanks, Steven.

That was so fun.

[bright music]

[upbeat music]