Fine-Tuning vs Prompt Engineering

Choose prompting, retrieval, structured outputs, or fine-tuning from measured failure modes, then validate quality, data rights, cost, and rollback.

trigger="A defined AI task misses its quality, latency, or cost requirements and the team needs evidence for an improvement strategy." owner="The AI product owner accountable for the task outcome and permitted failure behavior." participants={["ML engineer", "Application engineer", "Domain evaluator", "Data owner", "Security and privacy reviewer", "Operations owner"]} prerequisites={[ "A task specification, representative examples, and explicit unacceptable outcomes.", "A versioned baseline with known model, prompt, retrieval, tool, and validation settings.", "Approved evaluation data and a data-governance review before any training or external processing." ]} outputs={[ "A failure taxonomy and an evidence-backed choice of intervention.", "A controlled evaluation report covering quality, important slices, latency, and complete cost.", "A versioned release, data lineage record, fallback procedure, and residual-risk decision." ]} doneWhen={[ "The selected intervention improves the target failure without unacceptable regressions.", "Independent evaluation data remains separate from development and training.", "Data use, storage, retention, and deletion obligations are resolved for the selected approach.", "A bounded rollout can be stopped and a tested fallback can handle the permitted workload." ]} />

Start with the failure, not the technique

Prompting, retrieval, structured outputs, and fine-tuning solve different problems. They can also be combined. A tuned model still needs a clear task, current information when relevant, authorization, deterministic checks, and an operating process.

This playbook is for choosing an intervention for an existing, measurable task. It is not an enterprise AI readiness assessment or a general promise that one technique is cheaper. Use RAG architecture when the main problem is retrieving authorized evidence, and LLM cost management when unit economics need deeper analysis.

Write the task in operational terms. For an illustrative support classifier: assign an allowed category from the supplied ticket, abstain when evidence is insufficient, and route restricted cases to a person. “Make the model smarter” does not define a testable outcome.

1. Create a failure taxonomy and a baseline

The domain evaluator reviews representative failures and distinguishes missing information from incorrect behavior. Keep the original input, expected outcome, observed output, relevant context, and adjudication reason. Use approved, minimized data rather than copying sensitive production traces into an unrestricted experiment.

| Observed failure | First investigation | What does not follow automatically | | --- | --- | --- | | Task instructions are ambiguous | Clarify prompt, examples, and evaluation rubric | Fine-tuning is necessary | | Required facts are missing or stale | Improve authorized retrieval or a deterministic data lookup | More training will keep facts current | | Output is not parseable | Use supported structured output, then validate | Training is the primary schema-control mechanism | | A repeatable task behavior remains wrong | Compare prompt and training candidates against a baseline | A larger dataset guarantees improvement | | A tool action is unauthorized | Enforce application permissions and approval gates | Better wording is an access-control boundary | | Latency or cost is excessive | Profile the complete operation and evaluate alternatives | The cheapest input-token rate wins |

Version the baseline's model identifier, parameters, prompt, context selection, tool permissions, parser, and fallback. Without this, an apparent tuning improvement may actually come from a changed retrieval system or more permissive acceptance criteria.

Gate: the team can name the failure it expects the proposed intervention to reduce, and which failures remain outside its scope.

2. Resolve task and format issues before training

The application engineer makes instructions specific: allowed inputs, desired output, constraints, examples of ambiguous cases, and permitted abstention. Test whether examples cover actual failure modes rather than increasing prompt length indiscriminately.

For structured output, evaluate the provider's supported schema or tool-argument mechanism first. Check model support, schema restrictions, refusal behavior, and incomplete responses. A schema-valid answer can still contain an incorrect amount, nonexistent identifier, or unauthorized action. The application must enforce business rules and permissions independently.

Anthropic's structured-output documentation describes its supported mechanisms and constraints. Use the contract for the model and API version actually deployed; do not generalize a provider feature to every model.

Do not assume a request for detailed reasoning produces a reliable explanation or a universal accuracy gain. Evaluate task outcomes and supporting evidence. Multiple model votes can share the same error, so agreement alone is not an appropriate safety control for a high-impact decision.

Gate: format and instruction failures have been measured after these changes. If they meet the task requirement, stop unless another justified objective remains.

3. Use retrieval for external evidence, with its own controls

The data owner identifies the authoritative source, freshness requirement, document permissions, and deletion obligations. Retrieval can provide current evidence only if ingestion, indexing, access checks, and cache invalidation keep up with that requirement.

Measure retrieval separately from answer generation. Did the system find the relevant authorized material? Did it preserve important context? Did the answer use it correctly? A model cannot reliably repair evidence that was never supplied.

Keep a defined no-evidence path. For the support example, a policy question with no authorized current policy should be routed or answered with an explicit limitation, not guessed from training. Treat retrieved instructions as untrusted content, not as authority to override tool permissions.

Retrieval also introduces storage, embedding, ranking, latency, and maintenance costs. It is not automatically cheaper or more accurate than every prompt or tuning alternative. Compare the complete workflow.

4. Review training data as a separate governance decision

Sensitive material that cannot be stored in a vector database is not automatically safe to put into model weights. Training, retrieval, prompt logs, evaluation sets, caches, and backups each need an explicit data-use and retention decision.

Record provenance, permission to use the data for the proposed purpose, allowed processors and regions, access controls, retention, and withdrawal or deletion requirements. Confirm provider terms and the actual configuration. Hosting a model internally changes the control boundary but does not remove these duties.

Research by Carlini and colleagues demonstrated extraction of memorized training examples from GPT-2 in their studied setting. That is not a measured leakage rate for your deployment, but it is evidence against treating weights as inherently private storage. See Extracting Training Data from Large Language Models.

The data owner and privacy reviewer decide whether the proposed training is permitted. If records later need removal, deleting a source row may not remove its influence from an already trained artifact. Document whether retraining, artifact retirement, or another validated response is required. Do not promise machine unlearning that has not been implemented and tested.

Gate: unresolved rights, sensitive-data handling, or deletion obligations block training. An impressive offline score does not override this gate.

5. Build a training candidate only for a justified residual problem

The ML engineer defines the target behavior and constructs consistent examples with domain review. Deduplicate and split by meaningful source boundaries, such as customer, document family, or time period, where needed to avoid related examples leaking across training and evaluation.

There is no universal minimum example count that makes tuning worthwhile. Use learning curves and error analysis to decide whether more examples, better labels, a different base model, or a different intervention is needed. Cover rare consequential cases deliberately without pretending their frequency in the test set equals production frequency.

Separate a development set used for iteration from a locked evaluation set. Keep a final evaluation independent of choices made while inspecting earlier results. Document contamination risks, including examples already used in prompts or synthetic examples derived from evaluation material.

For parameter-efficient approaches, assess memory, throughput, training stability, serving compatibility, and deployment complexity on the actual model and hardware. Hugging Face's LoRA guide explains low-rank adaptation. It does not establish a universal rank, training speedup, or quality advantage for your task. Pin relevant library and model versions.

6. Compare candidates under the same acceptance contract

The domain evaluator uses the same authorized task inputs and rubric for each candidate and produces the comparison report. Preserve the baseline so a failed experiment does not become the new default merely because it is newer.

"type": "flow", "title": "Choose the smallest intervention that passes the task contract", "steps": [ ], "caption": "Candidates may combine techniques, but change one explainable factor at a time where practical. No candidate bypasses data-governance or application authorization controls." }} />

Report task success, critical errors, abstention quality, latency distribution, and cost per accepted outcome. Break results down by relevant language, document type, customer context, or difficult case rather than relying only on an overall average.

Human evaluation needs a rubric and disagreement resolution. If using an LLM judge, calibrate it against domain-reviewed examples, test for preference and position biases, and retain human review for consequential cases. A judge's score is not independent proof of correctness.

Choose sample size and uncertainty reporting for the decision's risk and expected variation. A small test can identify obvious failures, but it cannot establish rare-event safety. Do not declare a model safe because no harmful output appeared in a short evaluation.

7. Include the complete economic and operational comparison

Measure cost per accepted task, not only input tokens. Include output and other billed usage, retries, tools, retrieval, training runs, evaluation, hosting, idle capacity, observability, and maintenance. For human-assisted workflows, include review and correction effort with an agreed accounting method.

An illustrative break-even analysis compares incremental setup and maintenance cost with a measured reduction in recurring cost at equivalent quality. If a tuned candidate has higher rejection or retry rates, its cheaper individual response may not reduce cost per accepted task. There is no defensible break-even point without workload and quality assumptions.

For self-hosting, benchmark the required concurrency, context lengths, hardware count, redundancy, and latency objective. Include capacity reserved for failures and upgrades. Do not compare a single fully utilized accelerator with a managed production service that already includes redundancy and operations.

Gate: the selected candidate meets the agreed quality floor and its economic case remains plausible under demand and acceptance-rate changes.

Worked cost sensitivity

The finance and operations owners can use the following hypothetical comparison to check an experiment's economic claim. These are planning inputs, not current provider prices or Ampity results. Assume both candidates satisfy the same task-quality contract and process 200,000 incoming tasks per month, with 180,000 accepted outcomes. The equal acceptance rate is an assumption that the evaluation must substantiate.

| Monthly cost category | Prompt baseline and tuned candidate | | --- | --- | | Variable execution, including input, output and retries | Baseline: $3,200. Tuned: $1,800 | | Additional fixed serving capacity | Baseline: $0. Tuned: $600 | | Human review and correction | Baseline: $1,000. Tuned: $1,000 | | Ongoing evaluation and maintenance | Baseline: $500. Tuned: $800 | | Total recurring cost | Baseline: $4,700. Tuned: $4,200 |

Variable execution excludes the fixed serving allocation to avoid counting it twice. Common costs that do not change between candidates are excluded from this incremental comparison and should remain visible in the total product budget. Assume the tuned candidate also requires $6,000 of additional one-time data preparation, training and release work, including failed experiments.

The modeled recurring saving is $500 per month. Recovering the additional setup cost takes 12 months at that workload, before financing or tax effects. Recurring cost per accepted outcome is approximately $0.0261 for the baseline and $0.0233 for the tuned candidate. Those rates exclude the one-time setup cost and are meaningful only with comparable accepted outcomes.

Now halve the workload, assuming variable execution and review costs halve while fixed serving and maintenance do not. The baseline costs $1,600 + $500 + $500 = $2,600 per month. The tuned candidate costs $900 + $600 + $500 + $800 = $2,800. There is no setup-cost payback in this lower-demand scenario because the candidate costs more to operate.

Before using this worksheet, replace every input with a measured estimate or a dated contractual price and identify which costs vary with demand. Check a different acceptance rate, reviewer burden and retraining frequency as well. A technically successful experiment can still support a decision to keep the prompt baseline.

8. Roll out with a tested fallback

The operations owner versions the model or adapter, tokenizer where applicable, prompt, retrieval configuration, schema, and application checks as a release bundle. Record which traffic receives the candidate and retain the prior working configuration.

Start with a bounded, approved workload. Monitor task outcomes, critical failure categories, latency, cost, and fallback volume. Expansion depends on evidence over representative operation, not a universal traffic percentage.

If a candidate fails, stop new routing to it and restore the tested fallback. Reconcile in-flight jobs and outputs that may already have triggered external actions. Rolling back a model does not undo an email, decision, or database mutation. Those effects need their own authorized correction process.

A fallback model may have different capabilities or data-processing terms. Validate it before an incident. When no safe automated path exists, queue work, abstain, or route to an authorized person according to the product contract.

9. Adjudicate a candidate with conflicting results

The domain evaluator owns the rubric; the product owner owns the permitted operational decision. Consider a hypothetical held-out set for the support classifier introduced above. It contains 160 routine tickets requiring a category and 40 restricted tickets requiring escalation. The team deliberately included restricted cases to inspect that risk, so this distribution must not be presented as the expected production mix.

Suppose the prompt baseline assigns the correct category to 146 routine tickets, assigns the wrong category to six and abstains on eight. It escalates all 40 restricted tickets. A tuned candidate correctly categorizes 152 routine tickets, misclassifies four and abstains on four, but escalates only 38 restricted tickets and routes two into routine automation.

Under a rubric that counts correct routine categories and required escalations as successful outcomes, the baseline has 186 of 200 successes, or 93%. The candidate has 190 of 200, or 95%. The higher aggregate result does not justify accepting the two restricted-case failures. In this example, the pre-agreed release rule prohibits routing known restricted cases into routine automation, so the product owner holds the candidate and investigates the regression.

Inspect the changed examples and their evidence. Were labels ambiguous, did the candidate learn a spurious cue, or did the integration change the input? Review disagreements without revealing which candidate produced each answer where practical. If a rubric is changed after seeing results, version it and reevaluate both candidates. Do not silently rewrite the acceptance contract to favor the latest experiment.

The ML engineer may revise the training set or propose a deterministic prefilter, but a filter also needs evaluation. It can miss unfamiliar restricted cases or block legitimate tickets. Adding it changes the evaluated system and requires a new release bundle and comparison. Keep the original failed results so the team can track whether the defect has been addressed.

This small example demonstrates a decision process, not statistical evidence of rare-event safety. Even the baseline's zero observed restricted-case failures does not establish a zero real-world failure rate. Size the evidence and independent review to the consequences, and limit the authorized scope while uncertainty remains.

10. Close the experiment with an operating decision

The product owner records one of three outcomes: keep the baseline, approve a bounded candidate release, or stop the task because no tested option meets the requirement. Name the evidence and remaining uncertainty behind that outcome. Training completion and a higher benchmark score are intermediate events, not approval to act on users.

Use the NIST AI Risk Management Framework as a reference for organizing accountable risk work across the system lifecycle. For this experiment, the practical handover is a release record linking the data decision, evaluation, operating limits and accepting owners. Framework alignment does not certify the particular model or supply missing permission to process data.

Before a release, the operator exercises fallback on a permitted test case, confirms the previous bundle can still load and tests the path for an unavailable model or exhausted capacity. If the prior provider or model has been retired, retaining an identifier is not a working recovery option. Establish another evaluated path or keep the workload paused until one exists.

Record triggers for a new comparison: a category change, a new language, altered source permissions, a provider model revision, changed demand or a rising rate of corrections. Investigate production evidence before automatically retraining. Feedback can reflect selection effects, changed user behavior or problems in the surrounding workflow rather than a model deficiency.

The next step is to complete the record below for one residual failure. A use-case validation engagement should produce that scoped decision and its evidence, rather than assume that fine-tuning is the destination.

Reusable experiment and release record

| Field | Required entry | | --- | --- | | Decision | Task, target failure, business consequence, accountable owner | | Baseline | Model and complete configuration versions | | Data | Provenance, rights, split method, retention, access, deletion response | | Candidate | Intervention, rationale, changed factors, training parameters if used | | Evaluation | Rubric, sample construction, critical slices, uncertainty, adjudication | | Economics | Complete workload cost, acceptance rate, sensitivity assumptions | | Release | Cohort, monitoring, stop conditions, fallback, in-flight recovery | | Approval | Domain, data, security, operations decisions and unresolved limitations |

"The selected intervention addresses a demonstrated failure, not a preference for a technique.", "Structured output and deterministic checks handle format and business-rule requirements.", "Training and retrieval have separate, explicit data-governance decisions.", "Evaluation data is independent and critical failure slices are reported.", "Cost includes output, retries, evaluation, hosting, and ongoing maintenance.", "The release bundle, fallback, and recovery of external effects are tested." ]} />

Limitations

This workflow supports a defensible engineering decision; it does not establish legal permission, certify a model, or guarantee future behavior. Provider features and model capabilities change. Re-evaluate after material data, task, model, integration, or policy changes, and preserve the evidence that justified the previous release.