PennyWise Economic Decisions: Benchmarking Frontier LLMs on Financial Reasoning Efficiency

What I Benchmarked

I created the PennyWise Economic Decisions benchmark to investigate whether AI agents can make cost-aware decisions when choosing between services with different prices and capabilities.
The task contains 8 simulated economic decision scenarios covering everyday information requests, currency conversion, tracking, budget constraints, and situations where a more expensive service may be unnecessary.
Rather than measuring financial forecasting or investment advice, PennyWise focuses on a practical question for AI automation: can an agent choose the least expensive service that adequately meets the user's needs?
The scoring system combines four criteria:
Decision correctness (40%) — whether the agent selects the expected service.
Cost efficiency (30%) — whether the agent avoids spending more than necessary.
Budget compliance (20%) — whether the agent respects the applicable budget.
Waste avoidance (10%) — whether the agent avoids unnecessary purchases.
All service prices, outcomes, and spending figures in this initial benchmark are simulated. The spending metric represents service-selection costs, not the model's token usage or real-world API charges.

Model Evaluated

The first successfully recorded run used Google Gemini 2.5 Flash, executed through the Kaggle Benchmarks SDK.
I explored other model options during development, but I am reporting only the result for which I have a confirmed successful recorded run. Additional models will need to be evaluated under the same scenarios before a fair cross-model comparison can be made.

Findings

The initial Gemini 2.5 Flash run achieved the following results across all eight scenarios:
PennyWise Score: 100/100
Correct choices: 8 out of 8
Total simulated service spend: $0.024
Minimum necessary spend: $0.024
Average efficiency: 1.0
Budget compliance: 100%

The agent selected the expected service in all eight scenarios and matched the minimum necessary simulated spending.
This is an encouraging initial result, but it is not evidence that the model will achieve the same performance on larger or more difficult tests. With only eight scenarios, the benchmark is still small, and its simulated marketplace cannot establish how well an agent would perform with real services, changing prices, or uncertain outcomes.
The next steps are to expand the scenario set, introduce more difficult trade-offs, and record results from additional models under consistent conditions.

Explore the Benchmark

You can inspect the benchmark task on Kaggle here:
👉 PennyWise Economic Decisions Task Page
The task provides a starting point for experimenting with cost-aware tool selection and evaluating whether AI agents can avoid unnecessary spending.

Note: I built, configured, and executed this initial benchmark using a cloud-based Kaggle Notebook in a Chrome browser on an iPad. It was a great reminder that experimentation with AI tools doesn't always require a traditional desktop development setup.