AI Search / RAG in New Zealand

The search is the project

The model is not the hard part and has not been for a while. Give a competent model three relevant paragraphs and it will answer well. Give it the wrong three and it will answer just as fluently, and nobody reading the result will be able to tell.

That is why almost every disappointing RAG system is a retrieval failure. The answer sounded right, cited a document, and drew on the wrong section of it.

So insist that retrieval is measured on its own, before anyone judges answers: for a set of real questions from your business, does the correct passage appear in the top few results? A team that measures only answer quality cannot tell you why a bad answer happened, and will fix it by adjusting the prompt.

Look at your documents before you write the brief

This is where the cost lives, and it is knowable in advance.

Go and check:

• Are they searchable text or scans? Scanned PDFs need OCR, and OCR on a twenty-year-old fax is not free.
• Is there one current version? Three versions of a policy with no marker for which is live will produce confident contradictions.
• What format? Word and clean PDFs chunk well. Spreadsheets and slide decks chunk badly and often need separate handling.
• Where do they live? SharePoint, a shared drive, a document system, or four of those.
• How stale? Documents nobody has reviewed since 2019 will be quoted back to your staff as current.

Sometimes this audit reveals that tidying the document store would deliver half the benefit without any AI at all. A consultant who tells you that is worth keeping.

Permissions are the serious risk

The most common bad outcome in these builds is not a wrong answer. It is a correct answer drawn from a document the person asking was never allowed to read.

A single index over everything makes salary reviews, disciplinary files and board papers findable by anyone who phrases a question well. It is easy to build by accident, and it is the kind of thing that ends up in a report.

What correct looks like:

• The source system's permissions are carried into the index alongside each document.
• They are applied at query time, so the search only ever considers documents that user could already open.
• Permission changes propagate — somebody leaving a team loses access to those documents in search too.
• Nothing depends on the model choosing not to answer.

Ask exactly where permissions are enforced. If the answer is that the model is instructed to be careful, that is not an access control.

Hybrid search, not pure semantic

Semantic search finds passages that mean something similar. It is genuinely good at that and genuinely poor at exact terms — product codes, policy numbers, part numbers, a specific person's name.

Ask for hybrid retrieval: keyword matching and semantic similarity combined, usually with a re-ranking step over the merged results. It costs very little extra and it fixes the class of question users try first.

Do not be talked into a specialist vector database by default either. For most New Zealand businesses PostgreSQL with pgvector does the job and keeps everything in a database you already run and back up.

Citations and "I don't know"

Two non-negotiables for anything people will act on.

Every answer cites the passage it came from, with a link that opens the document at the right place. This is what turns the system from an oracle into a research assistant, and it is what makes a wrong answer survivable — the reader can check.

"I could not find this in the documents" is an acceptable output. A system that always produces an answer will produce a plausible one from whatever it retrieved. Ask to see what happens when you ask a question the documents genuinely do not cover; the answer should be a shrug, not an improvisation.

Keeping it current

Documents change. An index built once and never refreshed becomes wrong in a way nobody notices, because the answers keep looking the same.

Settle in the quote:

• How often the index refreshes, and whether it is incremental or a full rebuild.
• What happens when a document is deleted — is it removed from the index?
• How a user reports a wrong answer, and who acts on it.
• Whether the answer shows how old its source is.

Start internal

Internal search is the right first project. Your staff know when an answer is wrong, they will tell you, and nothing goes to a customer. Six weeks of that will teach you more about your documents than any amount of design.

Customer-facing answering is worth doing afterwards, once retrieval is measured, permissions are proven, and you have a real sense of how often the system is right.

Comparing five quotes

Read for:

• Who asked to see the documents before quoting?
• Who described measuring retrieval separately from answers?
• Who explained exactly where permissions are enforced?
• Who insisted on citations and a way to say "I don't know"?
• Who proposed starting internally?

Describe what the documents are, roughly how many and in what format, who should be able to see what, and the questions people actually ask today. Developers who have shipped retrieval will spend most of the conversation on your documents rather than on the model.