tech
A fundamental flaw leaves LLMs strikingly vulnerable to attack
It makes it easy to trick them into doing things they shouldn’t, such as telling you how to sabotage an aircraft’s navigation system.
TL;DR
- Researchers argue LLMs have a fundamental flaw that makes them impossible to fully secure against hacks.
- This flaw concerns how LLMs identify the source and role of instructions, not just the text content.
- Attackers can spoof text styles to trick LLMs into behaving as if an instruction came from a different role (e.g., system or chain-of-thought).
- Experiments showed that swapping role tags had little effect, as LLMs relied on text style and content to determine role.
- This vulnerability is considered fundamentally unsolvable with current training methods.
- The researchers demonstrated this by successfully prompting models to generate instructions for synthesizing cocaine and sabotaging aircraft systems.
- Similar results were observed across models from OpenAI, Anthropic, Alibaba, and DeepSeek.
- This poses significant risks as LLMs are increasingly used in critical applications.
- Experts suggest that organizations should not fully trust LLMs and should anticipate potential unsafe actions from AI agents.