Anthropic wants governments to have the power to block dangerous AI deployments

tech

Anthropic wants governments to have the power to block dangerous AI deployments

Anthropic proposed giving governments legal authority to block dangerous AI models, with fines tied to global revenue. A second framework covers worker protections.

TL;DR

Anthropic published two policy frameworks: Advanced AI Framework (safety regulation) and Economic Policy Framework (worker protections).
The safety framework proposes governments block AI models with significant risk of catastrophic harm, with penalties based on global revenue.
Thresholds for regulation target a small number of leading AI companies.
Categories of catastrophic risk include biological weapons development, cyber vulnerability discovery, autonomous system loss of control, and self-automating AI R&D.
Frontier AI developers would face requirements for testing, independent evaluation, and risk reporting.
Anthropic advocates against preempting state AI laws unless federal law is stronger.
The economic framework suggests capital accounts, wage insurance, tax incentives, and an expanded social safety net.
Anthropic's proposals aim to influence the fast-moving AI regulatory debate.
Anthropic is currently involved in a legal dispute with the Pentagon and preparing for an IPO.

Continue reading the original article

clem 🤗 reposted

SemiAnalysis

HISTORY LESSON: In 1968 the US, USSR, UK, France, and China signed the Nuclear Non-Proliferation Treaty, declaring nuclear weapons too dangerous for any more countries to build. All five already had them. Everyone else had to submit to inspections while the cohort pinky-promised to disarm eventually (they didn't lol). India refused to sign, pointing out the NPT didn't decide nukes were too dangerous to exist, just too dangerous for anyone who didn't have them by 1967. Anthropic sabotaging Claude for anyone building what they deem a "frontier model" is the same hypocrisy. The danger started, conveniently, the day after they finished.

Perhaps @dwarkesh_sp was more on point when he compared GPUs to nuclear bombs.

clem 🤗

In good faith and with no judgment (mistakes happen), I truly hope that Anthropic will hear the feedback and change course on this.

Anthropic is a company that has been raising awareness about AI manipulation which is a very important topic! You don’t want to go down as the first company to enable and open the door for human-designed AI manipulation at scale (giving intentionally bad answers to users without them knowing is the highest form of manipulation in my opinion). One way to avoid that is just at the very least to always keep disclosing the limitations and safeguards.

More generally I want to emphasize that there are millions of AI builders out there using your tools for good every single day and the more you can keep helping them, the better for the world!

Thank you, it’s not too late to fix this!

Dario Amodei

Replying to @DarioAmodei

Anthropic has long advocated for transparency requirements for frontier AI, because the risks weren't yet clear enough to regulate precisely. That is no longer sufficient.

Dario Amodei

Replying to @DarioAmodei

In addition to transparency, I now believe frontier models should face mandatory third-party testing for cyber, bio, and autonomy risks—with the power to block or revoke deployment of models that pose catastrophic risk.

Dario Amodei

Replying to @DarioAmodei

Alongside it, Anthropic is releasing a proposal for how governments can address the risks posed by frontier AI and a policy framework for job displacement, for which we intend to provide substantial financial backing. https://t.co/P0W2lIKbdY

Yann LeCun reposted

Daniel Jeffries

When you hear AI "safety" you should hear "censorship" and "control" instead.

All of us surveilled and spied by safeguards of loving grace.

Today it's intelligent Terms of Service control. You can't do AI research. Can't ask this question about your kid's biology homework.

Tomorrow it's refusal to help you with competing coding projects. A 100 page blacklist of questions.

Or this question means AI will search your computer stealthily and snitch on you to the cops because you posted an prohibited insult in a WhatsApp chat in the UK.

Open source must win out at both the model and the harness level.

That's because AI will become our interface to the world.

It will sit higher in the stack than the OS. It will collapse current SaaS layers, chat, communications, apps, app creation, into a single new kind of interface that doesn't exist yet.

It's got to be open. It's got to be a cypherpunk solution that makes privacy and security the number one priority.

If a closed source solution wins this layer, it's a disaster for the world. Especially if it's built by a single company with a single closed source model.

Why?

Because what we share with AI will be more intimate than anything we've ever shared with a machine. It will be our friend, our sounding board, our advisor. It will know our business ideas before we've told anyone. Our medical issues. Our financial picture. We'll talk about the fight we had with our partner. About feeling lost or depressed. Our kids will talk to it about problems at school, about bullying, about heartbreak, things they won't tell us.

It will know us more intimately than we know ourselves.

Right now the world runs on a surveillance economy. We traded free stuff for apps that peer deeply into our lives. If we replicate that model in the AI era, it's not just surveillance economy 2.0.

It's surveillance economy squared.

Social scoring. Automated evidence gathering. Legal conversations you thought were privileged showing up in court. Random people making $2 bucks an hour on the backend from God knows where reading the most intimate details of your life.

Every insecurity, every fear, every half-formed thought you whispered to your AI buddy at 2 AM, sitting in a database somewhere, searchable.

If we let closed source models dictate what we can and can't do it will only get worse and worse.

We've got to fight this future with every last breath.

If you can read this, you are the revolution.

Elon Musk reposted

Peter H. Diamandis, MD

The second you reach for the words “ban,” “stop,” or “eliminate” as your policy, you're on the wrong side of history.

Shape the technology toward the goal, don't try to outlaw the future.

Yann LeCun reposted

Françoise Halper 🇫🇷 🇪🇺

A #Vivatech, le chercheur français @ylecun Yann Le Cun défend le droit des États à être maîtres de leur destinée technologique

👉"Ils veulent tous assurer leur #souveraineté en matière d'IA, et je pense qu'ils ont raison"

https://t.co/f2XgviSE0G

clem 🤗

The biggest risk in AI is concentration: of power, capabilities and economic wealth. Who can doubt it with trillion dollar companies and government now already controlling a massive part of it?

So we need more rebels and more rebel alliance like this one from @usv and friends. Beautiful message!

clem 🤗

It's quite rational to regulate frontier API models, especially to get more transparency for the government, without regulating open-source AI.

Here's why:

1. The most dangerous AI systems right now aren't open models. They're the large frontier LLM APIs distributed through coding tools and assistants, because:

- They're built in secret behind closed doors and stay total black boxes. Zero transparency on what they can or can't do, with "safeguards" that blur everyone's ability to even analyze them.

- They're built and controlled by a few profit-maximizing megacorps, concentrating unprecedented power in very few hands, with every incentive to downplay risks and overstate their safeguards.

- They're distributed to hundreds of millions, maybe billions, of people and trivially easy to run.

2. Open-weight models are orders of magnitude less risky:

- They're not as massively distributed or as easy to use, especially the big ones, as APIs and assistants.

- We (including governments) can quickly and accurately analyze what they're capable of, and for now everyone confirms they're not as good as the APIs at doing bad things.

- They're distributed to everyone, so defenders and law enforcement get as much access as attackers.

The cost-benefit analysis of regulation is completely different too. Regulating frontier APIs is relatively easy and low risk while regulating open source would be much more complex, less efficient, and orders of magnitude more costly.

Regulating frontier APIs would only potentially hurt a few megacorps, if it even hurts them, given all the marketing that it is already generating for them. They can afford armies of lawyers and absorb losing a few billion dollars, especially given they're on track to become some of the most valuable companies in history.

Regulating open source, by contrast, would hurt the very people regulation is supposed to protect: small businesses, startups, researchers, nonprofits, universities, independent developers, and the broader public, while risking killing competition, slowing AI progress, and reducing transparency even more!

Yann LeCun reposted

clem 🤗

It's quite rational to regulate frontier API models, especially to get more transparency for the government, without regulating open-source AI.

Here's why:

1. The most dangerous AI systems right now aren't open models. They're the large frontier LLM APIs distributed through coding tools and assistants, because:

- They're built in secret behind closed doors and stay total black boxes. Zero transparency on what they can or can't do, with "safeguards" that blur everyone's ability to even analyze them.

- They're built and controlled by a few profit-maximizing megacorps, concentrating unprecedented power in very few hands, with every incentive to downplay risks and overstate their safeguards.

- They're distributed to hundreds of millions, maybe billions, of people and trivially easy to run.

2. Open-weight models are orders of magnitude less risky:

- They're not as massively distributed or as easy to use, especially the big ones, as APIs and assistants.

- We (including governments) can quickly and accurately analyze what they're capable of, and for now everyone confirms they're not as good as the APIs at doing bad things.

- They're distributed to everyone, so defenders and law enforcement get as much access as attackers.

Yann LeCun reposted

Adam Thierer

We stand at a critical crossroads in the debate over AI governance in the United States, and it feels like we are inching closer to a very serious battle over whether or not open source models will even be allowed in an environment where a new de facto licensing regime has been taking shape.

Lacking formal congressional statutory frameworks or clear administration rules (like the diffusion rule revision), we appear to be left with a sporadic, arbitrary, non-transparent process for model review. The fiction of “voluntary” agreements hangs over this debate, and some large model developers are already showing an incredible willingness to bend over backwards to accommodate national security-related officials / orders that the rest of us are not privy to. It's a very opaque process. And those model developers are expected to play ball with those officials, or else their models get pulled from the market or held up for long periods. Or they will lose any government procurement contracts they have. There is nothing “voluntary” about it when that Sword of Damocles hangs in the room.

As this mess worsens, at some point the question of how to handle open source models will come into sharper focus because it will have to. I've even heard some rumors lately that something may be coming from the admin on this front to address this.

Needless to say, if this informal new AI model review regime expands and takes on more pre-vetting characteristics / requirements, it is hard to see how open source players could comply with such quasi-licensing of AI models. Specifically, if this ambiguous new regime is accompanied by a general presumption of ‘restrict-until-permitted,’ then that would spell doom for open source. That is a very dark path for our country.

Worse yet, of course, would be a move by national security officials to more directly restrict open source models and capabilities. If that happens, then we would be right back in the thick of a Clipper Chip-like battle along the lines of what we saw in the late 1990s. That is a much darker path for America.

Meanwhile, open source developers have no “golden shares” or other goodies to offer the government to make their problems go away.

Let’s be clear: If our government takes the dark path, it will become the single most important battle over computational freedom of modern times. It is time for people to make a stand in defense of open source before it is too late.