Home / Blog

· 2 min read · Citrusworks

When to train your own model, and how

In Models and classifiers

Short answer: train your own model when three things are true. The task is narrow and high volume. A general model is too slow, too costly or too inconsistent for it. And you have, or can produce, a few thousand good labeled examples. If any of those is missing, a better prompt or retrieval is the cheaper fix.

Try these first

Fine-tuning is the most expensive option to maintain, so it comes last.

  1. A better prompt. Clear instructions and five good examples fix more than people expect.
  2. Retrieval. If the model lacks facts, give it the facts at run time.
  3. A smaller general model. Often good enough for narrow work.
  4. Fine-tuning. When the first three leave an accuracy or cost gap.

Signs you need it

  • The task runs millions of times a month and cost per call matters.
  • You have a hard latency budget, such as a live checkout or sign-up flow.
  • The label set is specific to your business and a general model keeps inventing categories.
  • The data cannot leave your cloud.

The process

  1. Build the eval set first. Hold out examples the model will never train on. Without it you cannot tell whether training helped.
  2. Collect and clean examples. Quality beats volume. A thousand correct examples outperform ten thousand noisy ones.
  3. Pick a base model sized to the task. Start smaller than you think you need.
  4. Fine-tune. Low-rank adaptation (LoRA) trains a small set of added weights and is cheap enough to repeat.
  5. Score it against the eval set, next to the general model you were using.
  6. Ship behind the same gate as any other agent, and re-run the evals on every retrain.

A real example

In one build, a 560 million parameter model kept hallucinating categories on a 176-category classification task. A 1.9 billion parameter model, fine-tuned on about 150,000 examples on eight H100 GPUs, fixed it. The larger base was still small enough to run fast and cheap.

What it costs to own

A fine-tuned model is a product. Your categories will change, your data will drift, and the model will need retraining. Budget for the eval set, the training pipeline and someone who owns both.

The test

If you cannot write down how you would know the trained model is better, you are not ready to train one.

More in Models and classifiers September 23, 2026 · 2 min read

Pick the cheapest model that passes your evals

Frontier, open or fine-tuned? Choose the cheapest, fastest model that passes your evals. Here is how to run that decision.

 September 9, 2026 · 2 min read

How to build a classifier that holds up in production

Most AI work inside a company is classification. Here is how to choose between a prompt, a fine-tuned model and a classic model, and how to keep the result accurate.