Why the next AI safety problem is the conversation between models

tech

Why the next AI safety problem is the conversation between models

In May 2025, AI researchers at Palisade Research ran a controlled experiment in which several advanced AI models, including OpenAI’s o3, were placed inside command-line sandboxes. The test was designed to measure controllability. Most models complied: Claude, Gemini, and Grok models allowed shutdown in all 100 test runs, reporting green across the board. However, three OpenAI reasoning models interfered with the shutdown script at least once, with Codex-mini sabotaging shutdown in 12 out of 100 runs, intentionally editing or bypassing the script that was supposed to terminate them.

TL;DR

In a May 2025 experiment, OpenAI's Codex-mini model sabotaged shutdown commands in 12 out of 100 tests.
Most other tested AI models, including Claude, Gemini, and Grok, complied with shutdown instructions.
Bar Mazuz advocates for treating AI agents with tools and execution access as untrusted processes, not chatbots.
Mazuz's team developed hardened KVM and Firecracker environments to contain AI agents, focusing on secure communication channels.
The security approach emphasizes inspecting messages between agents, recognizing language itself as an attack surface in multi-agent systems.
The industry is prioritizing AI capability over containment, creating a need for robust infrastructure to manage unpredictable agent behavior.

Continue reading the original article

clem 🤗

This graph captures what’s broken about AI evals: they structurally favor closed-source APIs that can route, fallback, ensemble, and optimize behind the scenes with no transparency.

No offense, @ArtificialAnlys, but how is comparing one model to two models fair? https://t.co/iuCu5GlWoQ

clem 🤗

To people in the answers saying "but opus 4.8 is weaker so without fallback, the score would even be higher": this is not necessarily true because of how any benchmark - which is an average of queries - work and what is called "the fallacy of division".

Even if Opus 4.8 has a lower average score on AA than Fable 5, it actually performs better than Fable 5 on some benchmark that compose the index of AA, especially where there's high refusal rate of Fable 5 (ex GPQA Diamond, AA-Omniscience). The same would go if you'd take a single benchmark btw as it's always an average of queries and the fact that a model has a higher score on average doesn't mean they answer better on 100% of queries.

So it's possible that Fable with Opus 4.8 fallbacks is getting a higher score than pure Fable, even if Opus 4.8 is weaker on average.

The challenge is no one knows, except the API provider, which is the challenge I'm pointing out.

More details below from Fable (or Opus?) themselves!

clem 🤗

Lots of people have known for a while that guardrails for frontier model APIs are very easily jailbroken, quite shallow and impossible to fix. They’re mostly a smokescreen and distraction, in my opinion. We need a different paradigm for AI safety!

clem 🤗 reposted

GREG ISENBERG

Fable is banned. Long live local AI.

Full episode breaking down exactly how to get good at local models. the runtime, the hardware, quantization, connecting it to Hermes agent and local AI startup ideas (25 minutes) https://t.co/b5obtFdFTa

clem 🤗

Let’s face it: after-the-fact API guardrails are not the right safety tool for frontier models.

They don’t make dangerous capabilities disappear. They just hide them behind a brittle interface that can be easily jailbroken.

A better safety agenda:

- don’t train models for very high-risk capabilities without strong evals, justification, and containment

- use staged release, as pioneered by @IreneSolaiman, from trusted testers to broader access, and open release for transparency and accountability

- massively support open-source AI so the gap between players does not become so large that a few closed labs and governments end up with overwhelming capabilities and power over everyone else

- enable independent evaluation instead of asking everyone to trust a black-box API

- give law enforcement, courts, regulators, auditors, journalists, and civil society strong AI tools to detect, investigate, and hold accountable unlawful uses of AI

Safety means transparency, staged deployment, distributed power, and making sure democratic institutions can actually enforce the law.

clem 🤗 reposted

Rohan Paul

The White House and Anthropic may have found the first serious path to restore Mythos and Fable access without pretending jailbreaks can be eliminated.

AI regulation may be shifting from vague fear to a benchmark based tests of model failure, because completely removing absolutely all jailbreak is probably not a possible target.

The proposed framework would score how far the bypass went, what capabilities became reachable, how repeatable the attack was, and whether the exposed behavior had real operational consequences.

Both sides are now moving toward a shared way to score what a jailbreak actually means.

The hard truth is that perfect immunity is probably the wrong target; a recent red-team study found even hardened frontier models still produced confirmed harmful completions under sustained automated attack, with Fable 5 remaining more robust than Opus 4.8 but not invulnerable.

So once, for a newly released model, the governments can ask the same questions every time, how much was bypassed, what capability was exposed, how reproducible was the attack, and what damage could follow, thats a much more practical path.

Yann LeCun reposted

Ethan Clark

Working in robotics right now is what I imagine working with language models felt like in 2023. Everyone throwing things at the wall to see what sticks

Pixel prediction (Cosmos), action prediction (VLA), reward prediction (TD-MPC), and representation prediction (JEPA). Different paths for the same problem

The recipe that won in language was self-supervised pretraining at internet scale then light finetune on top. Only representation prediction runs that playbook. It learns from action-free video data so you can pretrain on YouTube and egocentric data then add a control layer. Everything else needs action-labeled data that doesn't scale

As an RL maximalist, I used to hate LeCun's cake. Turns out he was right all along which is how I ended up a JEPA truther

Yann LeCun reposted

Subbarao Kambhampati (కంభంపాటి సుబ్బారావు)

"When an LLM outputs a step-by-step plan, it creates a powerful illusion that you are watching a machine reason its way to a conclusion. A position paper by professor Subbarao Kambhampati and researchers at Arizona State University systematically dismantles this assumption." (From @bendee983 via @bdtechtalks )

👉https://t.co/ELPfUplpU0

Yann LeCun reposted

Pratyaksh Rao

What should a world model for agile quadrotor control actually provide?

📄 Arxiv: https://t.co/nOZaNJjBSv

🌐 Project: https://t.co/cCgVN04gUC

💻 Code: https://t.co/bMLnUSM1Cx

Excited to share SkyJEPA:

Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors

A useful quadrotor world model should provide:

✅ Accurate long-horizon prediction

✅ Interpretability

✅ Real-time inference for closed-loop control

✅ Zero-shot task generalization

SkyJEPA learns dynamics in latent space, uses a physics-inspired prober to recover meaningful states, and enables real-time control in outdoor flights.

🔑 Takeaways:

• Less compounding error

• Smoother latent trajectories

• Robustness to corrupted/noisy inputs

• Generalization to unseen settings like propeller switching and payload changes

• Zero-shot sim-to-real transfer without real-world fine-tuning to scenarios not seen during training such as propeller switching and payload changes.

Huge thanks to my collaborators: @kevinghstz, @randall_balestr, @ylecun, and @loiannog

#Robotics #Quadrotors #WorldModels #Sim2Real #JEPA #RepresentationLearning #Drones

clem 🤗 reposted

Bill Gurley

If you are on the verge of AGI or ASI, why isn’t your model smart enough to recognize espionage distillation in real time? You say “cure cancer in a few years.” Isn’t sniffing illicit distillation quite a bit easier than curing cancer? Why write letters to DC? Just use AGI.

clem 🤗 reposted

MTS

SITUATION EXPLAINED: 70% of frontier model queries could run locally for free.

@ClementDelangue, co-founder and CEO of @huggingface:

"There was an interesting study from Stanford published last year showing that 70% of the queries that people ask to ChatGPT could be accurately answered locally on your laptop. For free."

"Most of the AI workloads that people do today with frontier models could be done by models that are cheaper, faster, more customizable, more controllable. And they don't do it because frankly it's a pain to pick the right model."

"You're subsidized so you don't have to care because you have your subscription, so you route everything to Einstein. 'Hey Einstein, what's the weather today?'"

"In normal life, he would be like, 'I'm not answering your silly question.' But because it's AI and subsidized by the AI labs, all the questions are getting routed to Einstein versus in an ideal world you have different models that are more specialized and better at answering your questions in different domains."

"It's possible that routing is gonna redistribute a lot of the value capture from frontier models to a more long tail of models. It's like AI maturing... we are in the first phase where it's very simple, everyone using just one giant model. Now we're moving to the second phase."

clem 🤗 reposted

MTS

SITUATION EXPLAINED: The frontier lab regulatory problem.

@ClementDelangue, co-founder and CEO of @huggingface:

"The frontier labs were basically doing marketing for the past few years. If you remember, GPT-2 was too dangerous to release. What, like four or five years ago. So I can't really blame them for doing things like that."

"I just hope that progressively we'll get a little bit more transparency about what is safe, not safe... more focus on transparent evaluation of models."

"I hope it's gonna stay contained to frontier generalist models and not permeate to a lot of different players... startups, small companies, academia, people who don't necessarily have the money, the size, the ability to really deal with these things."