AI | Monetary Policy & Inflation
AI Reflections: Which LLM Performs Best? Our Benchmark Says Fine-Tuning Wins
- By Dalvir Mandara, Eric Wang and Bilal Hafeez
Not a day goes by without a new LLM model hitting the market, each claiming to outperform the rest on some benchmark leaderboard. But these self-reported scores often deserve scepticism. Much like marking your own homework, these evaluations can be misleading. For starters, many benchmark datasets – or even their answers – might appear in a model’s training data. And then there’s the risk of ‘benchmark gaming,’ where model builders torture prompts or sampling settings until they get the results they want.
The real test? Evaluating models on genuinely out-of-sample, real-world tasks.
At Macro Hive, we are in a unique position to do just that—thanks to our proprietary, paragraph-level labelled dataset of central bank communications. Using this dataset, we benchmarked a range of top LLMs: Claude 3.7, DeepSeek R1, OpenAI’s GPT-4.5, and our own fine-tuned GPT-4o-mini variant.
The results: the Macro Hive fine-tuned GPT-4o-mini model achieved 88% accuracy, outperforming every other model. GPT-4.5 followed at 78%, with DeepSeek R1 at 76% and Claude 3.7 Sonnet (Extended) at 75%.
Background: Central Bank Tone Classification
In financial markets, traders often interpret central bank statements, minutes, and speeches to infer policy stance – classifying them as dovish (accommodative), neutral, or hawkish (tightening). Traditionally, economists do this work. But LLMs are now capable of interpreting such nuanced language too.
At Macro Hive, we have fine-tuned several LLMs to take on this task, using GPT-4o-mini as our base model. The cornerstone of our approach is our unique labelled dataset, built by our economists at both paragraph and sentence levels – granularity that is rare in the field. Crucially, we continuously expand this dataset, allowing us to test models on a truly out-of-sample basis.
Each model was tested in a standardised zero-shot setup, using a single prompt with no examples, to provide a clean comparison of their raw capabilities.
Punch Line: Domain Fine-Tuning Beats Big Models
Our fine-tuned GPT-4o-mini ranked #1 with 88% accuracy. That is impressive, especially given that GPT-4o-mini is not OpenAI’s top-tier model. It shows how domain-specific fine-tuning can beat larger, more expensive alternatives.
Second place: OpenAI’s largest model, GPT-4.5, at 78% accuracy.
Third and fourth: DeepSeek R1 (76%) and Claude 3.7 Sonnet (Extended) (75%).
But here is the kicker: price. GPT-4.5 costs ~$75 per million tokens, Claude 3.7 Sonnet costs ~$3 per million tokens, DeepSeek-R1 costs ~$0.55 per million tokens, while our fine-tuned GPT-4o-mini costs just ~$0.30 per million tokens.
Key Performance Highlights:
- Best-in-Class Accuracy: Macro Hive Fine-Tuned GPT-4o-mini leads with 88% accuracy, the highest among all models tested.
- Cost Efficiency: Macro Hive Fine-Tuned GPT-4o-mini offers exceptional value at ~$0.30 per million tokens, a staggering 250x cheaper than the next most accurate model, OpenAI GPT-4.5 (78% accuracy at ~$75 per million tokens).
- Strong Open-Source Contender: DeepSeek R1 delivers 76% accuracy – beating all OpenAI and Anthropic models except GPT-4.5. This highlights that open-source models can rival or exceed mainstream alternatives on specialised tasks.
- Notable Gain from ‘Extended Thinking’: Claude 3.7 Sonnet (Extended Thinking mode) scored 75%, significantly above its standard mode (67%). Longer reasoning chains in Claude provided a notable improvement in classification accuracy in this task.
- OpenAI Reasoning Models Underperform: OpenAI’s flagship “o1” reasoning model was the worst performer at 54% accuracy (its sibling o3-mini managed only 55%). Even OpenAI’s general-purpose GPT-4o (67%) and GPT-4o-mini (58%) base models outscored the ‘o-series’ reasoning models.
Class-wise Performance Breakdown
Breaking down performance by class reveals each model’s strengths or biases in detecting specific tones. The top-performing model, Macro Hive Fine-Tuned GPT-4o-mini, shows a strong balance — correctly identifying 85% of dovish, 86% of neutral, and 91% of hawkish paragraphs, leading in all three categories. OpenAI’s GPT-4.5 also performs consistently (81% dovish, 76% neutral, 77% hawkish), albeit at a lower level than our fine-tuned model.
Key Performance Highlights:
- Balance Across Classes: Macro Hive Fine-Tuned GPT-4o-mini delivers best-in-class performance across all tone categories – 85% accuracy on dovish, 86% on neutral, and an 91% on hawkish.
- Superior to Larger Models: OpenAI’s GPT-4.5 performs consistently (81% dovish, 76% neutral, 77% hawkish), but still falls short of our fine-tuned GPT-4o-mini’s accuracy – demonstrating the edge provided by domain-specific fine-tuning.
- Avoidable Bias in Base Models: The base GPT-4o-mini model shows stark class imbalance – achieving 100% accuracy on neutral paragraphs but collapsing on dovish (38%) and hawkish (50%). This highlights the importance of fine-tuning for nuanced tone detection.
- Hawkish Specialists: Macro Hive Fine-Tuned GPT-4o-mini (91%) leads on hawkish classification.
- Dovish Leaders: Macro Hive Fine-Tuned GPT-4o-mini and OpenAI GPT-4.5 stand out for dovish tone detection – both outperforming other models in this category.
Conclusion
These results show that carefully labelled domain-specific datasets – combined with thoughtful fine-tuning – can outperform much larger, more expensive models.
Macro Hive’s fine-tuned GPT-4o-mini ranks highest with 88% accuracy and a cost of just ~$0.30 per million tokens, far ahead of GPT-4.5’s 78% at ~$75.
Moreover, fine-tuning does not just boost accuracy – it also improves class balance and removes biases seen in general-purpose models. This makes fine-tuned smaller models the smart, scalable, and cost-effective choice for real-world tasks.
(The commentary contained in the above article does not constitute an offer or a solicitation, or a recommendation to implement or liquidate an investment or to carry out any other transaction. It should not be used as a basis for any investment decision or other decision. Any investment decision should be based on appropriate professional advice specific to your needs. You are not permitted to publish, transmit, or otherwise reproduce this information, in whole or in part, in any format to any third party without the express written consent of Macro Hive. This includes providing or reproducing this information, in whole or in part, as a prompt.)