In the rapidly evolving landscape of artificial intelligence, Silicon Valley’s emphasis on AI usage metrics is undergoing a transformation. At the recent Mistral AI summit in Paris, several leading executives shared insights on how their organizations assess AI’s value, highlighting a shift from mere usage statistics to outcome-based evaluations.
Charles Holive, the Chief AI Officer at BNP Paribas CIB, emphasized the importance of focusing on outcomes rather than what he termed “vanity metrics.” He stated, “We try to go away from vanity metrics — billions of tokens per day. We try to make sure that what we track is an outcome, not a vanity metric.” Holive’s approach involves asking what new capabilities AI has enabled and how it has accelerated existing processes.
Similarly, Antoine Pichot, Director of Innovation, Digital, and Data at La Banque Postale, gauges AI’s success by its ability to enhance employee efficiency, improve customer service, and offer value for money. Amit Kapur, the Chief AI and Transformation Officer at Tata Consultancy Services, also prioritizes improvements in business performance over token consumption metrics.
Sujay Bhattacharya of NTT DATA, who helps companies integrate AI tools, noted a growing trend among his clients to evaluate AI projects based on overall cost and business value rather than token counts.
The ‘Tokenmaxxing’ Backlash
The discourse at Mistral’s summit reflects a broader trend where some US companies are moving away from “tokenmaxxing”—the notion that increased AI usage equates to enhanced productivity. Amazon, for instance, recently halted an internal AI-use leaderboard after it encouraged employees to focus on rankings rather than meaningful outcomes.
Uber’s COO, Andrew Macdonald, has also expressed skepticism about the correlation between higher AI spending and the production of valuable products, questioning the direct link between token consumption and customer value. Meanwhile, companies like OpenAI, Anthropic, and GitHub are adopting usage-based pricing models, prompting a need for organizations to demonstrate tangible returns from AI investments.
Despite this shift, tracking AI token usage remains important for cost control and adoption measurement, according to Holive. However, the consensus among industry leaders is clear: while usage metrics provide a measure of AI’s reach, they do not inherently indicate a substantial return on investment.
