Home / Blog

· 2 min read · Citrusworks

How to build a classifier that holds up in production

In Models and classifiers

Short answer: a classifier sorts an input into one of a fixed set of labels. Most useful AI inside a company is this: route the ticket, tag the document, flag the risky transaction, pick the category. Start with a prompt, measure it, and move to a fine-tuned model only when volume, speed or accuracy demands it.

Why classification matters more than chat

A chat answer needs a person to read it. A label can drive software directly. "Fraud risk: high" can hold a payout. "Category: plumbing" can pick the right playbook. That is why classifiers are where automation pays first.

Define the labels before anything else

Most classifier failures are label failures.

  • Keep the set finite and written down. Every label gets a one-line definition and two examples.
  • Make labels mutually exclusive, or say plainly that an input can have several.
  • Add an "unsure" label. A model forced to choose will choose wrongly with confidence.

If two of your own people disagree on a label, the model will too.

Three ways to build one

  • A prompt on a general model. Fastest to start. Good for low volume and for finding out whether your labels make sense.
  • A fine-tuned small model. Best for high volume, tight latency and business-specific categories.
  • A classic model on embeddings. Cheap and fast when you have plenty of labeled data and the labels are stable.

Images are the same problem

Image classification follows the same rules with a different input. In one production system, rebuilding the image classifier cut latency by roughly 75 percent, which moved it from a batch job to something a live flow could wait on.

Measure the right thing

Accuracy alone hides the failures that cost money.

  • Per-label precision and recall. A model can be 95 percent accurate and miss every fraud case.
  • Confidence. Send low-confidence cases to a person.
  • Drift. Sample production weekly and re-score. Categories change. So do customers.

Put a threshold on it

Decide in advance what the model does alone and what goes to review. High confidence, low stakes: automate. Low confidence or high stakes: a person decides. Move the line as the scores earn it.

Start small

Pick one decision your team makes a hundred times a day. Write the labels. Label two hundred examples. Run a prompt against them. You will know within a week whether it is worth building properly.

More in Models and classifiers September 23, 2026 · 2 min read

Pick the cheapest model that passes your evals

Frontier, open or fine-tuned? Choose the cheapest, fastest model that passes your evals. Here is how to run that decision.

 August 26, 2026 · 2 min read

When to train your own model, and how

Fine-tune your own model when a small model is cheap and fast enough but not accurate enough, and you have labeled examples. Here is the decision and the process.