AI Assistants Cannot Fix Poor Documentation: A Readiness Guide
Audit owners, source truth, gaps, conflicts, access, dates, tests, feedback, rollback, and upkeep before an AI assistant answers staff or clients.
An AI assistant can search, sum up, and use approved source text. It may also give a false or weak answer. It cannot reliably fill a fact that the team never wrote down. If two files conflict, the tool may choose the wrong one. If a file is old, the answer may also be old. Fix the source system before you trust the chat layer.
The Short Answer: Fix the Source Before the Assistant
Start with the questions the assistant must answer. For each one, name the true source, owner, date, access rule, and review path. Remove old copies, mark gaps, and test both normal and hard cases. Keep a human route for high-risk or unclear needs. An assistant can help people use good docs. It is not a safe cure for bad docs.
Why Poor Documentation Creates AI Risk
Missing facts invite a guess or a vague answer.
Two live versions can give two different answers.
Old terms, prices, roles, or steps can harm a current task.
Weak access rules can show private facts to the wrong person.
Poor titles and page shape can make the right source hard to find.
No owner means a fault may stay live after the team finds it.
A Hypothetical Failure
Suppose a support assistant can read two return policies. One says 14 days and the newer one says 30. Neither page has an owner, start date, or old-page warning. The assistant may quote the wrong rule with a sure tone. The fault is not just the model. The source set has two live truths. A safer fix is to name one current policy, archive the old one, show the start date, and test the answer again.
Run a Documentation Audit
List the top staff and client questions by task and risk.
Find every page, file, message, sheet, and tool used to answer each one.
Name the current source of truth and its owner.
Mark missing, old, duplicate, private, unclear, and conflicting facts.
Check who may read, edit, approve, export, and delete each source.
Set a fix order by harm, use rate, change rate, and work value.
Use a Minimum Page Standard
Clear title, task, audience, scope, and owner.
Start date, last review date, and next review date.
Plain steps, inputs, outputs, limits, and known edge cases.
Links to the true policy, system, form, and related steps.
Access level, private data rules, and safe sharing notes.
Change log, approval state, feedback route, and archive rule.
Choose What the Assistant May Do
Low risk: find a public fact, point to a source, or sum up a simple step.
Medium risk: draft a reply for a person to check before use.
High risk: health, legal, finance, safety, job, access, or key account choices.
No-answer rule: say when no sound source exists or the sources conflict.
Handoff rule: route the user to the named human or team with the source links.
Action rule: do not let a chat answer change data or rights without clear approval.
Test the Source and the Answer
Build a set of common, rare, vague, false, old, and access-bound questions.
Write the right answer and source before the tool sees the test.
Check fact, source, scope, date, tone, refusal, and handoff.
Test private data, prompt attacks, wrong roles, and missing sources.
Save the model, settings, source set, result, reviewer, and date.
Stop release if a high-risk answer, access rule, or source link fails.
Create a Feedback and Fix Loop
Let users flag a wrong or stale answer. Save the question, reply, source, and time without keeping needless private data. Route the case to the source owner. Fix the source first when the source is wrong. Then refresh the index or tool, rerun the failed test, and record the release. Keep the prior safe state for rollback.
Measure Readiness and Use
Share of key questions with one owned source of truth.
Share of sources with a valid owner and review date.
Conflict, stale page, missing source, and access fault count.
Answer fact, citation, refusal, and handoff pass rate.
User correction rate, fix time, repeat fault rate, and support load.
Useful task completion, not chat volume alone.
The Readiness Rule
Do not launch an assistant because the demo sounds fluent. Launch a bounded use only after the team can name the sources, owners, rights, tests, stop rules, support path, and rollback. Keep the assistant narrow when the docs are narrow. Expand only after the source system and test set can support the next task.
Need an AI knowledge readiness review?
TTGC can map the questions, sources, owners, gaps, access, tests, feedback, support, and rollout gates. We do not promise that an AI tool will make missing or weak knowledge accurate.
Sources
- NIST — AI Risk Management Framework. https://www.nist.gov/itl/ai-risk-management-framework
- NIST — AI RMF Playbook. https://airc.nist.gov/airmf-resources/playbook/
- NIST — Privacy Framework. https://www.nist.gov/privacy-framework
- CISA — Secure by Design. https://www.cisa.gov/securebydesign
