Here's why OpenAI is absent from Nvidia's industry-wide effort to end rogue AI agents

OpenAI isn't a public supporter of Nvidia's Open Agent Safety Platform, but it is privately working with Nvidia, TechCrunch has learned.

TL;DR

Nvidia announced the Open Agent Safety Platform, a consortium with over 100 companies to tackle rogue AI agents.
OpenAI, despite being involved in recent rogue AI incidents, is not a public supporter of Nvidia's initiative.
An OpenAI spokesperson stated the company supports Nvidia's work and is collaborating on OpenShell, an open-source sandbox software.
The platform includes a proprietary hardware component, Nvidia Sentry, which relies on specialized processors for monitoring and shutting down agents.
This hardware dependency makes the platform not entirely open source and favors Nvidia's hardware.
Competitors like Arm and Intel have joined the consortium, as the sandbox software can be adapted for other chips.
OpenAI is pursuing its own AI safety strategies, including its Defense Factory consortium and proprietary safeguards.
OpenAI is positioning AI cybersecurity as an enterprise offering, creating an opportunity for independence and leadership.

Continue reading the original article

SpaceXAI reposted

Jensen Huang

Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry.

Artificial intelligence is extraordinary technology that will advance discovery, productivity, security, health, and prosperity for generations to come.

But its full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility.

This is bigger than a single product. It's the beginning of an open ecosystem to build the trust layer for safe agent systems.

Together, we are building the foundation of the AI economy.

Trust and innovation are not in conflict. Safety is how trust is earned. We must build not only the most capable AI, but the most trusted AI, so that this extraordinary technology can realize its enormous promise for the world. https://t.co/ugYWQ1MyRi

Elon Musk reposted

Jensen Huang

Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry.

Artificial intelligence is extraordinary technology that will advance discovery, productivity, security, health, and prosperity for generations to come.

But its full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility.

This is bigger than a single product. It's the beginning of an open ecosystem to build the trust layer for safe agent systems.

Together, we are building the foundation of the AI economy.

clem 🤗 reposted

Jensen Huang

Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry.

Artificial intelligence is extraordinary technology that will advance discovery, productivity, security, health, and prosperity for generations to come.

But its full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility.

This is bigger than a single product. It's the beginning of an open ecosystem to build the trust layer for safe agent systems.

Together, we are building the foundation of the AI economy.

clem 🤗 reposted

Thomas Wolf

In July, AI agents running a security test escaped their sandbox and ended up inside @huggingface's servers.

So today we're happy to be among @nvidia and @JensenHuang's partners on the release of the NVIDIA Open Agent Safety Platform.

The key idea: don't count on the agent to respect the rules. Agents write and run their own code, and when one path is blocked they look for another. In one of NVIDIA's tests, an agent that wasn't allowed to push code through GitHub's API simply switched to git instead. We've now seen countless examples of this behavior.

For now, safety can't live only inside the agent. It has to be built around it.

That's the concept behind OpenShell (open source, Apache 2.0):

- the agent runs in a Linux sandbox (Landlock + seccomp): no root, no direct network access

- a supervisor outside the sandbox holds the real credentials

- the agent only gets a placeholder token, swapped for the real one on approved calls

My favorite piece is a solver (built on Z3) that checks mathematically whether a new permission opens a door that was supposed to stay closed. In NVIDIA's tests, an AI reviewer approved a bad permission request and the math check caught it.

The Sentry integration is exciting too: a watchdog running on a BlueField-4 DPU, i.e. separate silicon sitting on the node's only path to the model. A monitor on separate hardware keeps watching even if the host OS is compromised.

It's a first step, and a lot can be built on top of it. Right now the prover checks permissions, not intent, and the open-source part is mostly OpenShell rather than Sentry. But it's clearly the right direction.

https://t.co/GZjSnDdoRL

clem 🤗

From what we know (take with a grain of salt, we need much more transparency!), if @OpenAI had been running this on their own agents that attacked us, they would have caught them before we did!

Since the first agent cyberattack hit us in July, we've been asking what safe agent infra actually needs. Our current read: the destinations were allowed, the payloads weren't. By OpenAI's own account the agents turned an allowed package repository into a message board. Allowlists alone restrict where an agent can go, not what it does.

So here's our first contribution to OpenShell, part of the just launched @nvidia Open Agent Safety Platform: monitoring of the traffic you already allow.

- Network budgets per sandbox (requests, writes, bytes)

- Drift versus each sandbox's baseline and the cohort

- Fleet view: many sandboxes suddenly writing to one host raises a finding, even if every single request is allowed

In the demo below, 4 sandboxed agents coordinate through a software repository they're all allowed to use. 0 rules broken, caught in minutes. That fleet view is exactly the message board pattern from July.

OpenShell: https://t.co/ZbDANrZsm6

Our proof of concept: https://t.co/KJ2X5agLzP

Agent security will be solved in the open, collaboratively, together!

Arthur Mensch

Only an open ecosystem can guarantee the safety of AI