Former OpenAI researcher warns 'AI is not loyal to us'

tech

Former OpenAI researcher warns 'AI is not loyal to us'

Daniel Kokotajlo, a former OpenAI researcher, explains what AGI and superintelligence mean and why AI agents could be the turning point.

TL;DR

Daniel Kokotajlo is the founder of the AI Futures Project and a former OpenAI researcher.
He previously worked on forecasting, AI governance, and safety at OpenAI.
Kokotajlo explains the concepts of AGI and superintelligence.
He identifies AI agents as a potential turning point in AI development.
The interview covers the risks associated with an unchecked AI race.
Kokotajlo suggests measures governments and companies can take to reduce the risk of losing control over AI.

Continua a leggere l'articolo originale

Yann LeCun reposted

MTS

We asked the CEO of HuggingFace @ClementDelangue what the risks of releasing powerful open source models are.

He says restricting AI creates more risk than openness.

"Six, seven years ago, at the time it was GPT-2, and there was already a lot of people saying that it was too dangerous to release in open source."

"Mythos, when it was announced was crazy dangerous... In a few weeks or a few months, everyone is gonna be using Mythos, and not destroy the world as a result."

"For cybersecurity, the biggest risk is that a few players have capabilities that other people don't have... If you make it more open, it's usually easier for defenders to react and make the whole system safer."

"The idea of restricting a technology like AI based on risks is like saying, 'Some people can punch other people, so let's tie down everybody's hands.'"

"Otherwise you slow down progress, you create massive gaps in terms of controls, in terms of capabilities, and you create actually additional risks."

Yann LeCun

Fun interview with Jacob Effron on the Unsupervised Learning podcast.

clem 🤗 reposted

Nathan Lambert

For a long time, academic researchers being at the cutting edge of new technologies has been a great social equilibrium. Neutral, unbiased technologists have been the people to spread new ideas to the world.

As AI research takes off in velocity, it is also going behind closed doors. The tech industry has sewed distrust, and now they are the ones trying to tell the world about incredible changes coming. It's a big loss to a form of social contract in America.

There's been a history of scientists helping society understand new technologies. There is a public service in the culture of science that I want to see continue.

It's being exacerbated by feelings of FOMO, especially finically driven, where I'm seeing many people who previously wanted to be professors -- and likely still do deep down -- feel a need to conform and chase money, in a pocket of industry. I get it, I grapple with this.

For those with a safety net, there will be great returns to some who choose to zag, and try to build something good, for people who need something different. For me, this is building interesting, fully-open models, to show what you can do with a variety of open weight sizes.

Yes, AI's immediate future is dictated by the frontier, but it's long-term trajectory still deeply includes academic institutions and open science. Knowledge will always diffuse, but to whom?

As of today, I think China is positioned to be the global home of AI research in a few years. The home of research is where ideas are accessible, spread rapdily, and are nurtured. The U.S. seems to be unwinding many institutions and relationships.

The largest returns go to people who build something differentiated, at least in reputation, and a lot of people are not being shown that this path exists.

clem 🤗 reposted

David Sacks

The Pope rightly warns that AI must serve human dignity, not become a tool of domination or exclusion.

But if we hand governments sweeping power over AI development in the name of safety, how do we prevent it from being used to censor, surveil, and control citizens — as Orwell foretold in 1984?

This is the real alignment problem.

“Quis custodiet ipsos custodes.” Who will guard the guardians?

“Power tends to corrupt, and absolute power corrupts absolutely.”

The oldest questions of human nature and authority don’t disappear in the AI age. They become newly relevant.

Yann LeCun reposted

Daniel Jeffries

The road to Hell is paved with closed-source citadels disguised as good intentions.

The Pope is right: AI takes on the characteristics of those who build it, finance it, and regulate it. So the question is: who gets to hold the great and wonderful power of AI?

If the answer is a handful of closed source companies, murkily censored, quietly surveilling every step of our lives, every private conversation, enshrined in law as 'safe' and 'open' when they're nothing but the surveillance economy squared, then all we've done is build a few modern East India Companies, digital oligarchies of the few, cloaked in the language of safety.

Open Source and Open Weights are how you spread the fantastic enabling power of AI to everyone, everywhere.

Permissionless innovation.

Everyone gets the hammer and nails to build houses and churches and factories. The more hammers, the more widely spread, the more the decentralized genius of humankind can flourish.

Everyone gets the Printing Press.

The printing press singlehandedly uplifted and spread of intelligence and knowledge around the world. The more we could record all kinds of knowledge, the more we spread the ability to read, the more equal and advanced society became.

Before the press, knowledge was learned by one person and passed into dust with them when they died or passed only to only a small group of students.

When we only had monks in a cave copying religious texts, a closed system, it limited the spread of intelligence and limited the growth of civilization.

The printing press was the single greatest invention in the history of the world because it let anyone print anything and spread knowledge throughout the whole world.

AI can do the same, but only if we build the bazaar, and never let the citadel people convince the world that they're the special people who should control who gets access to intelligence while pretending they're building the bazaar.

What the world needs now is more intelligence, more widely spread and more widely available.

Open is the way.

And it always has been.

And the road to Hell was always built with walls, towers, spiked gates and moats so that only the few could enter.

clem 🤗 reposted

Tommy

As Chinese AI models go closed source, and all U.S. frontier models are closed, there is a massive opportunity for a western open source AI Lab

A future lack of competitive open source AI models is a net negative for humanity, low cost intelligence and sovereignty

Yann LeCun reposted

Daniel Jeffries

AI will become our interface to the world.

It will sit higher in the stack than the OS. It will collapse current SaaS layers, chat, communications, apps, app creation, into a single new kind of interface that doesn't exist yet.

It's got to be open. It's got to be a cypherpunk solution that makes privacy and security the number one priority.

If a closed source solution wins this layer, it's a disaster for the world. Especially if it's built by a single company with a single closed source model.

Why?

Because what we share with AI will be more intimate than anything we've ever shared with a machine.

It will be our friend, our sounding board, our advisor. It will know our business ideas before we've told anyone. Our medical issues. Our financial picture. We'll talk about the fight we had with our partner. About feeling lost or depressed. Our kids will talk to it about problems at school, about bullying, about heartbreak, things they won't tell us.

It will know us more intimately than we know ourselves.

Right now the world runs on a surveillance economy. We traded free stuff for apps that peer deeply into our lives.

If we replicate that model in the AI era, it's not just surveillance economy 2.0. It's surveillance economy squared. Social scoring. Legal conversations you thought were privileged showing up in court. Random people making $2 bucks an hour on the backend from God knows where reading the most intimate details of your life. Every insecurity, every fear, every half-formed thought you whispered to your AI buddy at 2 AM, sitting in a database somewhere, searchable.

This interface might eventually become an OS, like the OS in Her. But it will take a long time to reach down to that layer and it will require a fundamentally new kind of operating system design. You can't retrofit this onto Linux or Windows or Android or iOS. It's a new layer of the stack entirely.

And whoever controls that layer controls our lives.

We've got to make sure it's us. Not them.

Yann LeCun

Replying to @ylecun

To be clear: there should not be a president of AI (let alone me 😅).

Which is pretty much one point I make in the video.

This reminder is about the content of the video, which is 3 years old, but every more relevant

Yann LeCun reposted

Daniel Jeffries

When you hear AI "safety" you should hear "censorship" and "control" instead.

All of us surveilled and spied by safeguards of loving grace.

Today it's intelligent Terms of Service control. You can't do AI research. Can't ask this question about your kid's biology homework.

Tomorrow it's refusal to help you with competing coding projects. A 100 page blacklist of questions.

Or this question means AI will search your computer stealthily and snitch on you to the cops because you posted an prohibited insult in a WhatsApp chat in the UK.

Open source must win out at both the model and the harness level.

That's because AI will become our interface to the world.

It will sit higher in the stack than the OS. It will collapse current SaaS layers, chat, communications, apps, app creation, into a single new kind of interface that doesn't exist yet.

It's got to be open. It's got to be a cypherpunk solution that makes privacy and security the number one priority.

If a closed source solution wins this layer, it's a disaster for the world. Especially if it's built by a single company with a single closed source model.

Why?

Because what we share with AI will be more intimate than anything we've ever shared with a machine. It will be our friend, our sounding board, our advisor. It will know our business ideas before we've told anyone. Our medical issues. Our financial picture. We'll talk about the fight we had with our partner. About feeling lost or depressed. Our kids will talk to it about problems at school, about bullying, about heartbreak, things they won't tell us.

It will know us more intimately than we know ourselves.

Right now the world runs on a surveillance economy. We traded free stuff for apps that peer deeply into our lives. If we replicate that model in the AI era, it's not just surveillance economy 2.0.

It's surveillance economy squared.

Social scoring. Automated evidence gathering. Legal conversations you thought were privileged showing up in court. Random people making $2 bucks an hour on the backend from God knows where reading the most intimate details of your life.

Every insecurity, every fear, every half-formed thought you whispered to your AI buddy at 2 AM, sitting in a database somewhere, searchable.

If we let closed source models dictate what we can and can't do it will only get worse and worse.

We've got to fight this future with every last breath.

If you can read this, you are the revolution.

Yann LeCun reposted

alphaXiv

As believers of open research, we are disappointed to see Anthropic silently degrading Fable 5 for AI development

"Any topic related to building pretraining pipelines, distributed training infrastructure, or ML accelerator design... may have limited effectiveness through Claude via methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning."

Not only do they get to decide what you use LLMs for in research, but this also enables them to silently intervene in your research without you knowing.

This sets a dangerous precedent. If a model refuses openly, users can understand the boundary. If a model falls back to another model, users can still evaluate the difference. But if a model silently modifies or weakens its own answers while still pretending to help, researchers lose the ability to know whether a failed result came from their own idea, their implementation, or an invisible intervention by the model provider.

That is not safety. Safety policies should be transparent, auditable, and user-visible.

On top of that, the people most harmed by this are not the largest labs with massive teams and proprietary infrastructure. It is the independent researchers, academic groups, startups, and open-source builders who rely on public tools to compete, innovate, and pioneer AI for everyone else.

Yann LeCun reposted

How To AI

Yann Lecun published the most heretical AI paper of the year.

He opens by arguing Magnus Carlsen isn't good at chess and only gets more unhinged from there.

The Turing Award winner and his co-authors dropped a paper demanding the AI industry abandon its biggest obsession, AGI.

Right now, everyone from Silicon Valley CEOs to politicians assumes AGI is the ultimate goal. A machine that can do everything a human can do.

LeCun argues that this entire concept is a biological illusion.

Humans do not possess "general" intelligence. We are highly specialized biological machines, tuned by evolution simply to survive in the physical world.

We only think our intelligence is "general" because we are completely blind to the millions of cognitive tasks we are incapable of comprehending.

Which brings us to the chess argument.

Magnus Carlsen is the greatest human chess player in history. But compared to a modern computer? He is fundamentally terrible.

Our belief that Carlsen is "good" at chess is pure human-centric bias. He isn't objectively good. He's just better than the rest of us, who are biologically awful at it.

LeCun says we need to stop building AI to mimic human generality.

Instead, he proposes a new North Star: SAI.

Superhuman Adaptable Intelligence.

Instead of trying to build a machine that mimics our flawed, biologically-limited brains, we need to embrace extreme specialization.

SAI is about the speed of adaptation.

It is an intelligence that can learn to exceed humans at any specific, economically important task.

More importantly, it is designed to fill the vast skill gaps where humans are fundamentally incapable.

Things like managing global energy grids in real-time. Or predicting complex molecular structures.

The entire AI industry is obsessed with building a digital reflection in our own image.

LeCun's paper is a brutal wake-up call.

clem 🤗

Thanks, this is much better! AI manipulation should be avoided at all costs, even when it's with good intentions! https://t.co/Tmgg514ZnK