Why is ordinary document search not enough?
Employees usually want an answer for a specific situation, not a file. They need to know which process applies, whether it is current and what it means for them. A keyword may find several documents that conflict or concern different products.
AI can connect relevant passages and explain their meaning. It cannot take responsibility for the truth of company policies. Without an assigned owner and a current version, the model merely presents disorganised material more elegantly.
Before starting, select common questions and their correct sources. This creates both a test set and an initial list of missing knowledge. There is no need to begin by migrating every historical document.
What does Upside's knowledge-search case show?
For Upside, Glean reports 92% adoption, savings of more than 2,000 hours per month and over 1.5 hours per employee per week. These vendor-published metrics concern one organisation and may partly overlap. They cannot be added together or treated as a promise of equivalent results elsewhere. Glean and Upside case study.
“People were just constantly reinventing the wheel.” — Elizabeth Vaggelatos, Head of Knowledge Management at Upside, in the cited study.
Ask which knowledge people repeatedly search for or recreate. Once specific work situations are identified, results can be measured. The number of indexed documents alone does not show whether employees received the right answer.
How does RAG work, without unnecessary complexity?
For each question, the system first retrieves relevant document passages. It gives them to a model, which composes an answer and shows sources. The model does not need to know all company information beforehand. What matters is receiving the right passages and recognising when they are insufficient.
| Layer | Its job | How to verify it |
|---|---|---|
| Source processing | Preserve text, tables and context | Compare with the original document |
| Permissions | Select accessible documents | Test different users and roles |
| Retrieval | Find the correct evidence | Expected sources for test questions |
| Answer | Explain supported information | Accuracy, completeness and citations |
| Updates | Reflect changes and deletions | Change a policy and repeat the question |
Locate an error in the right layer. If a relevant document was not retrieved, a longer prompt may not help. If the source was right but the answer omitted an important condition, the error arose when processing the retrieved context.
How should knowledge be organised to make it useful?
Every important document should have a title, owner, scope and date or version. For a process, describe its inputs, steps, exceptions and expected outcome. A table without column explanations can be ambiguous to both people and models.
Separate binding rules from examples and historical conversations. A one-off discount in an email is not a new company price list. Old instructions should be archived and labelled so routine queries do not treat them as current.
In the XWiki community on Reddit discusses preparing knowledge before connecting AI. As this is a vendor community, we treat the questions as prompts for investigation, not independent research. Your own inventory should establish who will correct each area when the assistant finds a conflict.
How can answers preserve access permissions?
The same rules must cover retrieval, citations and subsequent actions. Users must not receive private information through summaries of documents they cannot access. Even source links or titles must not reveal sensitive content beyond their permissions.
Atlassian's explanations of Rovo describe how user context and permissions relate. In a specific deployment, verify the connected account and how automations execute. A broadly privileged service account does not replace individual employee access controls.
Test role removal, document moves, account deletion and links to inaccessible sources. Retrieval and caches must respond to changes. Otherwise, an apparently protected source can remain accessible through an old copy in the AI system.
What should the assistant do when it does not know?
State what it found and what is missing. When sources conflict, name the conflict and direct the question to the information owner. Inventing one consistent policy from inconsistent sources can be worse than giving an incomplete answer.
“What is the current rate?” requires product, customer and time-period context. The assistant should ask for clarification if it cannot otherwise choose the right source. Clear follow-up questions are part of good service, not model failure.
The newer Anthropic's 2026 agent evaluation methodology supports checking actual outcomes. For knowledge bases, we therefore assess retrieved sources and permissions as well as answer fluency.
Which metrics should the first pilot track?
Answer accuracy, source accuracy, handling unknown questions, respecting access and time to a usable result. Test common wording and questions using different terms from the documentation.
Illustrative example: out of 100 questions, the assistant answers 80 correctly, correctly acknowledges insufficient evidence for 15, and invents answers for five. Those five errors may matter more than generally fluent responses. Further work should address their causes, not increase the word count.
The pilot should also establish maintenance rules. Who updates sources, how are errors reported, and when are tests repeated? Only then does it make sense to extend the knowledge base into an AI assistant for employees or public support.
Frequently asked questions
Does this mean training our own model?
Usually not. A common approach retrieves relevant material for each question and gives it to the model as context. This is called RAG. Training a custom model is not a prerequisite for a good company knowledge base.
Can the knowledge base include PDFs and scans?
Yes, if the integration reliably processes them. Scans need OCR, and tables need structural checks. These steps depend on the particular solution; they are not automatic features of every chat.
What happens after a document is deleted or changed?
The change must affect the index, caches and availability for future questions. Existing history also needs defined rules. Deleting the file in the source system alone may not be enough.
Research and solution design: Tanduva with AI assistance. External case studies are identified; illustrative examples are not measured results from our clients. Editorial methodology and corrections.