tech
A scorecard for the AI age
The question I hear from CFOs everywhere is simple: how do we get more value from our AI spend?
TL;DR
- Measure AI success by 'work accomplished' rather than adoption metrics like seats or licenses.
- The key economic question is whether AI's value creation outpaces its production cost.
- Focus on 'Useful Intelligence per Dollar,' which involves four critical questions: Is AI completing important work? What is the cost per successful task? Can people depend on the results? Does AI value increase with usage?
- Evaluate AI by the actual work it completes, such as resolving customer issues or reviewing contracts.
- Calculate the full cost per successful task by considering model price, compute usage, employee time, human review, and retries.
- Dependability is crucial; track outcomes like 'ready to use,' 'needs correction,' and 'needs escalation' to gauge AI's reliability.
- Economics should improve at scale, with completed work growing faster than total costs while maintaining or improving quality.
- Continuous improvement in AI models, inference efficiency, hardware, and software leads to better outcomes and lower costs for customers.