Diagnose system issues (prompt failure, hallucinations, model mismatch)

Key points

1. 1

   Start from what changed and what the symptom is. A failure that begins right after a document refresh points at retrieval or indexing; after a prompt edit, at the prompt or output contract; after a model swap, at model mismatch with the prompt's assumptions; with no change at all, at drifting inputs.

2. 2

   Confident but wrong answers after a document refresh, with model and latency unchanged, mean retrieval is feeding poor context (broken re-index, mismatched embeddings, stale or irrelevant chunks). This is the exam guide's own sample question.

3. 3

   To separate retrieval from generation, replay the failing request from its trace and check whether the correct passage was among the retrieved chunks. Missing passage: retrieval fault. Passage present but ignored or contradicted: prompt or model fault.

4. 4

   Retrieval evidence: gold passage absent from top-k, failures clustered on documents ingested after a chunking or embedding change, tables or clauses split across chunk boundaries.

5. 5

   Format failures (unparseable JSON, wrong types, prose around the payload) are prompt and output-contract problems. Fix them deterministically with structured outputs or an explicit schema and examples, not with temperature changes or a bigger model.

6. 6

   Hallucination remedies from the docs: allow "I don't know", ground long-document tasks in direct quotes extracted first, require citations and retract unsupported claims, restrict the model to provided documents, verify with chain-of-thought, compare best-of-N outputs, and refine iteratively. None eliminates hallucination entirely; validate critical outputs.

7. 7

   Model mismatch shows as systematic reasoning errors on a task that exceeds the model tier, once retrieval and prompt are verified sound. Move only the hard step to a stronger model or enable extended thinking, confirm on the eval, and keep the cheaper model for simpler steps.

8. 8

   After a model migration, behaviour that was never stated explicitly (length, tone, format) can change while accuracy holds. Make the requirements explicit, add checks to the eval, and validate the migration instead of rolling back or truncating output.

9. 9

   Input drift: when nothing in the system changed but quality dropped, the inputs changed. New topics not covered by the index cause retrieval misses that the model papers over; a static eval set built before the change cannot see it. Fix coverage, add a "not in my sources" behaviour, and refresh the eval set from recent traffic.

10. 10

    Non-answers to recognize as distractors: switch to a bigger model before isolating the layer, add "be accurate" or "MUST" to the prompt, set temperature to 0 (a repeatable wrong answer is still wrong), increase top-k when the right chunk is already present.

11. 11

    Pin model versions, log prompt versions and index versions with every request, and keep traces with retrieved chunks and tool results; diagnosis is a lookup when the evidence exists and guesswork when it does not.

Read the source

• Reduce hallucinations
• Increase output consistency
• Structured outputs
• Model migration guide