AI Agent in New Zealand

Start with the work, not the technology

The briefs that succeed describe a job somebody currently does. The ones that struggle describe a capability somebody read about.

Good candidates for an agent look like this:

• Someone reads a piece of incoming text — an email, a form, a document — and decides where it goes.
• The decision needs judgement, which is why the rules-based version was never built.
• It happens often enough that the time adds up.
• Being wrong occasionally is survivable, because a person sees the result or it can be undone.

That last point is the one to be honest about. Work where a single mistake is expensive and invisible is the wrong first project, whatever the volume.

Decide what it may do without asking

This is the whole design, and it is a business decision rather than a technical one.

There is a ladder, and most projects should start lower than they intend to finish:

1. Reads and suggests. A person acts. Almost no risk, and usually most of the time saving.
2. Prepares and a person approves. The draft reply, the coded invoice, the triaged ticket — one click to accept.
3. Acts within limits. Under a value, within a category, with anything unusual escalated.
4. Acts freely with oversight. Rare, and it should be earned with months of measured performance.

Run at level two for a month before moving up. You get the time saving immediately, and you accumulate a record of what the agent would have done — which is the only honest basis for trusting it further.

Guardrails go in the code, not the prompt

An instruction in a prompt is a strong suggestion. Under unusual input, or under deliberate pressure from someone who wants a refund, a model can be talked past it. A permission check cannot be talked past anything.

What real guardrails look like:

• The agent runs as a user account with specific, limited rights. It cannot reach what that account cannot reach.
• Value and volume limits are checked in code before an action runs — not described in the prompt.
• Irreversible actions — payments, deletions, anything a customer sees — require a person.
• Every action is logged with what it did and why, in a form somebody non-technical can read.
• There is a switch that stops it now, and somebody knows where it is.

Ask each developer where each limit lives. "We tell it not to" is not a guardrail, and the answer is very quick to give.

Ask to see the evaluation set

This is the question that separates people who have shipped an agent from people who have demoed one.

An evaluation set is fifty to two hundred real examples from your business with the correct outcome recorded. The agent is run against them, scored, and scored again after every change.

Without one, nobody can tell you whether last Tuesday's prompt change made things better or quietly worse — and prompt changes routinely do both at once, improving one category while breaking another. Teams without evaluations manage their agent by anecdote and are always surprised.

Building the set is a real cost, and mostly your team's time rather than the developer's, because only they know the right answers. It is the best money in the project.

Being unsure is a feature

An agent that always produces an answer will confidently produce a wrong one. What you want is an agent that recognises the cases it should not handle and passes them on.

Design that in explicitly:

• What does "I am not sure" look like, and what happens next?
• Who receives an escalation, and how quickly?
• Is the customer told a person is now involved?
• Are escalations reviewed, so the pattern feeds back into the design?

A well-built agent that handles 70% of cases and reliably escalates the rest is worth far more than one that attempts everything at 85%.

What it costs to run

Model usage is usually smaller than people expect — often a few hundred dollars a month for a New Zealand business, occasionally into the low thousands at real volume. Ask for a cost per task and what happens if volume triples.

The costs people forget:

• The person reviewing escalations, which is real time even when it is less time.
• Maintaining the evaluation set as the business changes.
• Re-testing when a provider ships a new model — which they do regularly, and behaviour shifts.
• The integrations, which need the same care as any other integration.

Settle where the data goes

Unless you host the model yourself, your data goes to a provider. The mainstream ones do not train on business API traffic by default and offer residency options, but this should be a decision rather than a discovery.

If you hold health information, financial records, or anything covered by a customer contract or a government agreement, agree the answer before the build and get it in writing. The Privacy Commissioner has published expectations worth reading, and if you sell to government there is guidance you will be measured against.

Comparing five quotes

Read for:

• Who asked what the agent may do without a human, before quoting?
• Who proposed an evaluation set, and included building it?
• Who described guardrails as code rather than as instructions?
• Who told you where the data goes without being asked?
• Who suggested starting one rung lower than you asked for?

Describe the job a person does today, how often, what a mistake would cost, and which systems the agent would need to touch. Developers who have put one into production will spend most of the conversation on that third question.