Machine Learning in New Zealand

Start with the decision, not the data

Machine learning earns its cost when it changes something someone does. So the first question is not what you could predict — it is what you would do differently if you knew.

Write down, before speaking to anyone:

1. What decision is being made?
2. Who makes it, and how often?
3. What do they do now, and what would they do differently with a prediction?
4. What does being wrong cost, in each direction? A missed churn risk and a wasted retention call are not equally expensive.
5. How would you know, in six months, whether it helped?

Projects that can answer those five succeed. Projects that produce an accurate model and a dashboard nobody opens are the standard failure, and they are accurate right up until they are cancelled.

Buy the feasibility study first

Nobody can honestly quote a build before looking at your data, and a firm that does is guessing with your money.

A feasibility engagement — typically $15,000 to $35,000 over three or four weeks — should tell you:

• Whether you have enough examples, and enough history.
• Whether the outcome you want to predict was recorded reliably. This is where most projects die.
• What a simple rule achieves, as the baseline.
• What a first model achieves against that baseline.
• An honest recommendation, including "not yet" or "not worth it".

You keep that whether or not you proceed, and a firm willing to report that the signal is not there has just cost itself the build.

Insist on the baseline

This is the number that tells you whether the project was worth buying, and it is frequently absent.

Before any model, measure the obvious rule. "Customers who have not ordered in ninety days will churn." "Invoices over sixty days will be paid late." "Jobs quoted under twenty hours overrun."

Those rules are usually better than people expect. A model that beats one by two percentage points is not worth six figures. One that beats it by twenty plainly is.

Ask for the baseline and the model result side by side, in the same units. A firm that reports only the model's accuracy is showing you a number with nothing to compare it against.

Your outcome data is the constraint

Businesses usually have plenty of input data and almost no reliable record of what happened.

You want to predict which quotes will convert — but nobody recorded why the lost ones were lost. You want to predict which jobs overrun — but the actual hours were entered inconsistently. You want to predict churn — but nobody agreed what churn means, so a customer who orders annually looks lapsed every January.

Ask the feasibility study to examine this specifically. If the answer is that the outcome was never captured properly, the correct project is to start capturing it, and to revisit this in a year. That is a genuinely good outcome and it costs a fraction of a model built on noise.

Getting the prediction where it is needed

A model in a notebook helps nobody. Most of the engineering in a real project is delivery:

• Where does the prediction appear — in the CRM, on a job, in a list someone works through?
• Is it batch overnight, or live when a record is opened? Batch is far cheaper and usually enough.
• What does the person see besides a number — why was this flagged?
• Is the prediction recorded, so you can compare it later with what actually happened?

That last point is what lets you prove the model is still working, and it is the one most often left out.

Plan for it going stale

Models degrade quietly. A new product line, a pricing change, a shift in customer mix, and a model trained on last year is confidently wrong rather than obviously broken.

Ask for:

• Monitoring that compares predictions against what actually happened.
• An alert when accuracy falls below an agreed level.
• A retraining schedule, and who runs it.
• An agreed answer to what happens if it degrades badly — is there a fallback to the simple rule?

Be able to explain a decision

If the prediction affects a person — credit, pricing, prioritisation, eligibility — someone will eventually ask why.

"The model said so" is not an answer, and under the Privacy Act people have rights over decisions made about them from their information. Ask each firm how a prediction is explained, and whether a person can override one. Simpler models are often chosen for exactly this reason, and giving up a little accuracy for an explanation is frequently the right trade in a business setting.

Comparing five quotes

Read for:

• Who insisted on assessing your data before quoting a build?
• Who proposed measuring a simple baseline first?
• Who asked who would act on the prediction?
• Who included drift monitoring and retraining?
• Who was willing to say the data is not ready?

Describe the decision, how often it is made, what data you hold and for how long, and what a wrong answer costs. Firms who do this work will spend most of the first conversation on your outcome data.