How to write a knowledge base your AI support agent can actually use

When people set up an AI support agent for the first time, they spend most of their attention on the instructions. Tone, persona, what it may and may not promise. That matters, but it is not where the answers come from. The answers come from retrieval: the agent searches your knowledge base for the passages most relevant to the question and reasons over those. If the right passage is not there, or is buried in a wall of text with the wrong title, no prompt will save it.

Svellyo shows the source it used under every AI reply, which means we get to see exactly where things go wrong. This post is the list of rules that came out of that. None of them are clever. All of them move the needle.

How retrieval works, in one paragraph

Every article, crawled page and uploaded file is split into chunks of a few hundred words. Each chunk is indexed twice: by meaning, using an embedding, and by keywords, using full-text search. When a visitor asks something, both indexes are queried, the results are fused, and the top handful of chunks go to the model along with your instructions and the conversation so far. The model answers from those chunks, reports a confidence score, and cites the chunks it used. If the confidence is below your threshold, it hands off to a person instead of guessing.

Everything below follows from that. Chunks are what get retrieved. Titles and first sentences are what make chunks findable. Precision beats volume.

1. One question per article

The single most effective change is to stop writing long reference pages and start writing short answers with the question as the title.

"Billing" as a title retrieves poorly because almost no question mentions the word billing. "How do refunds work on yearly plans?" retrieves precisely, because that is what people type. If your billing page currently covers refunds, invoices, upgrades and VAT, split it into four articles. Each will be one or two chunks, each will rank cleanly, and each will be cited by name in the inbox so you can see when it is wrong.

2. Put the answer in the first two sentences

Chunks are ranked by relevance, and the model reads them in full, but the first sentences carry the most weight in both the embedding and in how the model summarises. Write the direct answer first, then the caveats, then the background. Your customers will thank you too.

Bad: "There are several factors that affect whether a refund can be issued, and it is important to understand the context of your plan before proceeding."

Good: "Yearly plans can be refunded in full within 14 days of the charge. After that, unused months are prorated as credit."

3. Use the words your customers use

Internal names leak into documentation. You call it "the workspace switcher". Customers say "how do I change accounts". Read your inbox for a week and note the phrasing in the first message of each conversation, then use that phrasing in titles and first sentences. The keyword half of retrieval is literal, and the embedding half is close enough to literal that vocabulary matters.

Svellyo's Analytics page has a "top questions" section that clusters what visitors asked. It is the cheapest source of vocabulary you will find.

4. Numbers and policies go in AI-only articles

There are facts you want the agent to state precisely and do not want on your public help center: exact limits, exception policies, which discount codes still work, how to handle a chargeback. Mark those articles AI only. They are indexed and cited like everything else, but never appear on the public site, even if someone guesses the URL.

Write them as plain lists of facts. "Pro plan: 200 chats a month, 5 seats per workspace, custom domain included." The model does not need prose here, it needs the numbers.

5. Delete what is stale, do not just unpublish it in your head

An out-of-date article is worse than no article, because the agent will state it confidently and cite it. When a feature changes, edit the article the same day. When a feature is removed, delete the article. When you notice the agent citing something that made you wince, fix the source rather than the prompt.

A useful habit: whenever you take over a conversation because the AI handed off, ask why. If it handed off because the answer was missing, write the article before you close the thread. Two minutes now saves the same handoff twenty times.

6. Crawl your docs, but check what came back

A website crawl is the fastest way to give the agent everything. Point it at your docs site and it indexes every same-site page it can reach. Two things to check afterwards.

First, the page count. If it is far lower than you expected, the site is probably rendered entirely in the browser and the crawler saw empty shells. Static and server-rendered docs work best.

Second, the noise. Navigation menus, cookie banners and footers repeated on every page dilute the useful text. The crawler strips most of this, but if you see the agent citing a page and quoting something that was clearly a menu, add that page as an article instead and let the crawl skip it.

Set a refresh schedule. Weekly is right for most docs sites.

7. Files are for manuals, not for spreadsheets

PDF, Markdown and plain-text uploads work well for anything that reads like prose: a user manual, an onboarding guide, a terms document. They work badly for tables of numbers, because a chunk that is half a table loses the header row and becomes meaningless. Turn tables into short articles with sentences, or paste them into the editor where they become real tables that keep their headers.

8. Write the instructions after the knowledge base, not before

Once retrieval is good, the instructions can be short. Three or four sentences is usually enough:

You are the support assistant for Acme Analytics. Be concise and friendly. Answer only from the knowledge base; if you are not sure, say so and offer to bring in a teammate. Never promise refunds or discounts. For enterprise pricing, collect the visitor's email and hand off.

Resist the urge to put facts in the instructions. Facts belong in articles, where they are retrieved when relevant and cited when used. Instructions are for behaviour.

9. Tune the threshold with real questions

The confidence threshold decides how eagerly the agent hands off. The default is conservative. Use the test chat in the AI settings to ask twenty real questions from your inbox and watch the scores. If good answers are being handed off, lower the threshold a little. If the agent is guessing on questions it should not, raise it, and then write the missing article.

Add your customers' actual escalation phrases to the trigger list. "Real person" and "talk to a human" are there by default. If your customers say "can I speak to someone", add that.

10. Read the unanswered questions every Monday

Analytics keeps a list of questions the agent could not answer well, refreshed weekly. It is, in effect, a to-do list for your knowledge base written by your customers. Open it, write or fix one article per entry, and watch the AI resolution rate climb. Most workspaces we have seen go from roughly half of conversations resolved by the AI to three quarters within a month of doing this consistently.

11. Let the help center and the agent share one source

Every public article you write for the agent is also a page on your help center, and every help center page is retrievable by the agent. Do not maintain two knowledge bases. Write once, keep visibility set to Public + AI for anything a customer might search for, and AI only for the rest. Article ratings from the help center feed back into the same list of gaps.

A worked example

Here is what this looks like for a small analytics product with a pricing question that used to hand off every time.

Before: one "Pricing" article, 1,400 words, covering plans, limits, upgrades, refunds, VAT and enterprise. The agent retrieved a chunk about VAT for the question "can I get a refund on the yearly plan?" and, correctly, refused to answer with confidence.

After: six articles. "What does each plan include?", "How do refunds work on yearly plans?", "How do upgrades and downgrades work?", "Do you charge VAT?", "Enterprise pricing", and an AI-only "Plan limits and exceptions" list. The refund question now retrieves a 120-word article whose first sentence is the answer. Confidence went from 0.4 to 0.9, and the conversation never reached a human.

Total time: about forty minutes, most of it deleting.

The rules, in one list

1. One question per article, with the question as the title.
2. Answer in the first two sentences.
3. Use the customer's words.
4. Numbers and policies in AI-only articles.
5. Delete stale content the day it goes stale.
6. Crawl your docs, then check the page count and the noise.
7. Files for prose, tables as articles.
8. Short instructions, written last.
9. Tune the threshold with twenty real questions.
10. Read the unanswered questions weekly.
11. One knowledge base for the agent and the help center.

Retrieval is boring, which is why it works. If you do only the first two rules, you will be ahead of most AI support setups we have seen.