tech
What Happens When the World is Run on Code No One Understands?
The International Congress of Mathematicians met in Philadelphia last month, its first U.S. meeting since 1986. Minutes after accepting the Fields Medal, mathematics’s highest honor, Jacob Tsimerman told reporters he was leaving academia to work on AI safety. “I think the world is changing,” he said, adding that he did not expect the mathematical career to survive in its present form.
TL;DR
- AI is accelerating discovery at a rate that outstrips human verification, leading to an information overload and potential risks in critical systems.
- Formal verification, a mathematical approach to proving software correctness, can be used to automatically validate AI outputs and ensure software reliability.
- The bottleneck in innovation is shifting from discovery to human confirmation, necessitating the development of infrastructure for routine verification of AI-generated code.
- Generative AI tools can make formalization more accessible, but defining precise specifications remains a human judgment challenge.
- The United States should treat mathematical rigor as a national mission, investing in public infrastructure for verified software, standards, tools, and training.
- Companies operating critical infrastructure should prioritize formal guarantees and auditing from the start, rather than relying solely on reactive patching.
- Mathematics has historically responded to foundational doubts with increased precision and better verification, a transformation now needed for software as AI writes more code.