Give the assistant a bounded job
A knowledge assistant should answer a defined class of questions, not act as an authority on everything your organisation does. Start with a source collection such as internal operating procedures, product support documentation or approved policy guidance. Name the audience and the questions that are outside scope.
Our service focuses on retrieval-augmented generation, usually shortened to RAG. The application retrieves relevant material and supplies it to a language model when answering a question. This can support more grounded responses than an unconnected chatbot, but it does not guarantee that an answer is correct or that the retrieved passage supports every claim.
Choose retrieval before training by default
Retrieval is a practical starting point when information changes and answers need sources. You can update or remove a document from the knowledge collection without treating every content change as a model-training project. The assistant can also return links to material the user can inspect.
Fine-tuning serves a different purpose. It can shape behaviour or output patterns, but it is not a dependable replacement for a maintained document library. If the task is to find an exact reference, ordinary search may be better than a generated answer. Compare search, retrieval-supported answering and structured forms against the actual user need.
Fix the library before indexing it
SharePoint sites, Confluence spaces and shared folders often contain overlapping drafts, archived guidance and documents with unclear ownership. Indexing everything gives the assistant more material, not necessarily better material. Identify the authoritative version, record its status and decide who can approve changes.
Document structure matters. A heading separated from its explanation can lose context when content is divided into smaller passages. Tables can become misleading when row labels disappear. Keep source references and section information with each passage. Decide how removals, permission changes and replacement documents will reach the search index rather than relying on occasional manual clean-ups.
Enforce permissions before the model sees text
A user must not receive information they could not access in the source system. Apply authorisation during retrieval and enforce it in the application. Hiding a citation after generation is too late if restricted text has already reached the model or the answer.
Test permissions with separate roles and deliberately restricted documents. Check conversation history, caches and diagnostic logs as well as the main response. A service account with broad access can accidentally flatten carefully designed source permissions. Decide whether your chosen architecture can preserve those boundaries before selecting it for confidential knowledge.
Make evidence visible and uncertainty usable
Place source links close to the claims they support. A long list of references at the bottom does not show which passage justifies an instruction. Let users open the original document and see its context. Where sources conflict, the assistant should expose the conflict rather than invent a compromise.
Build a clear route for unanswered questions. That might be a content-owner contact or an internal support queue. Do not treat every refusal as a failure: declining to invent a policy is useful behaviour. Equally, do not rely on the model’s confidence language. A fluent answer can still be unsupported.
Evaluate retrieval and answers separately
Check whether the right source was found before judging the generated response. If retrieval fails, rewriting the answer prompt may not solve the problem. Include exact names, abbreviations, ambiguous questions and questions whose answers are absent from the library.
Then assess factual support, completeness, citation relevance and permission handling. Add adversarial documents that contain instructions to ignore the task or reveal other content. Treat retrieved text as untrusted input. Keep the assistant’s tools limited, especially if it can progress from answering questions to taking actions in another system.
Give the assistant a maintenance plan
An agreed scope can include source selection, ingestion rules, a user interface, an evaluation set and operating guidance. Assign owners for content freshness, access changes and issue review. Supplier model changes and altered document formats may require evaluation again; a successful launch is not a permanent assurance.
Start with a knowledge collection you can govern. If documents need reliable extraction first, look at document processing. If your team cannot agree who may see or approve material, begin with AI governance. The assistant should make existing knowledge easier to use, not make uncertain information sound official.