Manage stakeholder feedback loops and expectation alignment (including SLAs)
Align expectations with what a probabilistic system can deliver: measurable acceptance criteria, a scoped pilot with exit criteria, SLA wording that separates availability from answer quality, and feedback loops with a fixed cadence.
Key points
- 1
LLM output is probabilistic. No prompt, temperature setting or model tier guarantees zero errors, so never answer "can you guarantee it will never be wrong?" with yes. Replace the guarantee with measurable acceptance criteria, a review strategy for high-risk content and a pilot that measures the real error rate.
- 2
Refusing the project outright because errors are possible is also wrong; the current human process has an error rate too, and a pilot can establish whether the measured rate with review is acceptable.
- 3
SLA wording: availability and latency are continuously observable and can be committed as uptime-style SLOs (99.5% availability, p95 under 3 seconds), with carve-outs for upstream dependencies such as the model API. Answer quality can only be measured on a sample by an agreed method, so state it as a measured target on a versioned, jointly agreed evaluation set with the sample size, grading method and review cadence written down.
- 4
Do not commit to a quality figure the evidence does not support. A 96.2% result on 400 cases with a ±2-point confidence interval does not support a 97% guarantee; gaming the evaluation set with easy cases to reach the number will be exposed by the customer's own measurement.
- 5
Pair quality shortfalls with a remediation process (root-cause review, fix window, re-measurement) rather than uptime-style financial penalties; quality on probabilistic output improves through diagnosis and iteration.
- 6
A pilot has a charter: scope, cohort, success criteria, exit criteria and go-live gates, agreed in advance. When a sponsor forwards the demo link to the whole department or adds scope mid-pilot, re-anchor on the charter, restore the cohort, and log the new requests for the next iteration rather than welcoming the expansion, declaring launch, or stopping the pilot.
- 7
A demo is not a launch and sponsor enthusiasm is not an exit criterion.
- 8
When a pilot misses its criterion (91% against 95%), keep the trade-off visible: present the gap with the error breakdown and its business impact, offer options that trade scope, time or review effort against risk (launch only the segments already above threshold, extend the pilot, launch with a higher review rate), recommend one, and leave the decision with the sponsor. Do not hide the gap, report it as "on track", or lower the criterion after the fact.
- 9
Feedback loops need a cadence and structure: a weekly review of a stratified sample of interactions with the business owners, an error taxonomy, in-product flags routed to a triage queue and reported by category, and an evaluation set that grows with the failures found.
- 10
Weak feedback signals to avoid: an end-of-pilot survey (too late to act on), the model rating its own answers or self-reported confidence, anecdotal impressions from one senior stakeholder, and changing the prompt after every single complaint without measurement.
- 11
Decision rights stay with the sponsor; the architect's obligation is to make the risk and the options explicit, not to veto or to capitulate silently.
Read the source
Test yourself on Manage stakeholder feedback loops and expectation alignment (including SLAs)
Ten questions, with the answer and explanation after each one.