Home / Blog

· 2 min read · Citrusworks

Pick the cheapest model that passes your evals

In Models and classifiers

Short answer: do not pick a model by reputation. Build the evals first, run several models against them, and choose the cheapest and fastest one that passes. For some tasks that is a frontier model. For many it is a small open model you run yourself.

The usual mistake

Teams start with the most capable model available and never revisit the choice. It works, so nobody questions the bill. A year later a classification task that a small model could do is running on the most expensive model on the market, millions of times a month.

Run it as a comparison

  1. Write the evals for the task. Without them you are comparing opinions.
  2. Run three or four candidates: a frontier model, a mid-size one, an open model.
  3. Record three numbers for each: pass rate, cost per task, latency.
  4. Drop everything that fails the pass rate you need.
  5. From what is left, take the cheapest one that meets your latency.

When a frontier model wins

Long documents, multi-step reasoning, messy inputs, and tasks where a wrong answer is expensive. Paying more per call is fine when the call replaces an hour of expert time.

When an open model wins

High volume, narrow tasks and anything with a hard latency budget. Classification, extraction, routing and tagging are the usual cases. So is any workload where the data cannot leave your cloud.

When to fine-tune

Fine-tune when a small model is fast and cheap enough but not accurate enough, and you have examples to teach it. In one build, a 1.9 billion parameter model fine-tuned on 150,000 examples across 176 categories fixed the hallucinations a smaller model kept producing.

Do not fine-tune first. Try a better prompt, then retrieval, then fine-tuning. Each step costs more to maintain than the one before.

Keep the option open

Models change every few months. If your evals are in place, switching is a one-day test. If they are not, switching is a leap of faith, and you will stay on whatever you started with.

More in Models and classifiers September 9, 2026 · 2 min read

How to build a classifier that holds up in production

Most AI work inside a company is classification. Here is how to choose between a prompt, a fine-tuned model and a classic model, and how to keep the result accurate.

 August 26, 2026 · 2 min read

When to train your own model, and how

Fine-tune your own model when a small model is cheap and fast enough but not accurate enough, and you have labeled examples. Here is the decision and the process.