Describe the failure before choosing the technique
A model missing a current policy, producing inconsistent labels and taking too long to respond presents three different problems. Fine-tuning may help some of them. Treating all three as a need for a custom model obscures the experiment that would actually answer the question.
Start with a task and a failure record. For example: extract line items from invoices, preserve the original currency and abstain when a field is illegible. Keep examples of each failure rather than relying on an impression that the model does not understand the domain.
Revised September 13, 2026. This is an evaluation workflow, not a report of a Quandelia fine-tuning experiment.
Establish the baseline you would actually ship
Use a maintained prompt, the same input preparation and explicit output validation. If the model needs information it does not have, supply that information through a suitable context or retrieval path before concluding that its weights need to change.
Create separate development and held-out evaluation sets. Keep closely related examples together when splitting: near-identical invoices from the same template can make a random split look easier than deployment will be. Decide how to represent new suppliers, ambiguous inputs and missing fields.
Record accuracy by failure type alongside latency, cost and the rate of manual review. An average improvement can conceal a regression on the cases where a mistake is expensive.
Make the experiment answer one question
| Observed failure | First comparison to run |
|---|---|
| Missing or outdated facts | Baseline versus supplied, current source material |
| Inconsistent format or labels | Better instructions and validation versus supervised adaptation |
| High cost on a stable task | Current model versus a smaller candidate at matched acceptance criteria |
| Poor specialist behaviour | Prompted baseline versus adaptation using representative examples |
These comparisons are not a mandatory ladder. They isolate competing explanations. A small experiment is valuable when it can show that the proposed training work addresses the wrong bottleneck.
Parameter-efficient methods such as LoRA adapt a subset of model parameters rather than updating all weights. They can reduce training requirements, but they do not create representative data or an evaluation method. The LoRA paper and PEFT documentation describe the method and its implementation choices.
Separate adaptation from hosting
Fine-tuning and self-hosting are different decisions. An adapted model may be served through a managed service where supported, while an unmodified open model may run on infrastructure the team operates.
The earlier version of this article tied fine-tuning too closely to a large self-hosting commitment. That made the decision framework unnecessarily restrictive. The relevant comparison includes the actual training and serving options available for the candidate model.
Count data preparation, training runs, evaluations, serving and maintenance. Include the cost of an unsuccessful experiment. Keep recurring expenses separate from one-time work so the traffic assumptions remain visible.
Decide what would count as a win
Set acceptance criteria before examining the final held-out result. A candidate might need to reduce a particular error class while keeping review load and tail latency within the current service budget.
If the apparent gain is small, inspect the cases on which the candidates disagree. Repeat the evaluation where output variation matters, and report uncertainty rather than giving a tiny difference more significance than the sample supports. HELM is a useful reference for treating model quality as more than one metric.
A deployment decision should also include a rollback route and the conditions that trigger re-evaluation. New document types, a changed label taxonomy or a different base model can invalidate the original comparison.
Keep the learning even if you do not ship the model
The most useful output of the experiment may be a better dataset, a clearer acceptance rule or evidence that retrieval fixes the problem more cheaply. Those artifacts remain useful when model offerings change.
Fine-tuning earns its place when it improves a defined workload enough to justify its full cost. The decision should be supported by that comparison, rather than by either enthusiasm for custom models or a blanket preference for prompting.