Data Integration in New Zealand

Ask what decision the data is for

The single most expensive assumption in this category is that data has to move in real time. It is also the least often examined.

For each flow, ask what decision is being made with the information and how quickly:

• A monthly board report — overnight is fine, and daily is generous.
• A salesperson checking whether an invoice was paid — hourly is fine.
• A customer on a website seeing whether an item is in stock — minutes, cached.
• A warehouse scanner allocating the last unit — genuinely immediate.

Real-time integration means queues, ordering guarantees, replay and a great deal more testing. It typically doubles the cost of a pipeline. Choose it because something needs it, not because it sounds more modern.

Price the tools before building the plumbing

Fivetran, Stitch, Airbyte and Azure Data Factory have ready-made connectors for most of the systems a New Zealand business runs. Where one fits, it beats developer time comfortably.

They stop making sense when:

• A source has no connector — usually your own software, or something local.
• The transformation is unusual enough that you are writing the logic anyway.
• Volumes push the subscription past what building would have cost.
• Data cannot leave your own infrastructure.

Ask each developer whether they priced a tool. The one who says a subscription would do it and loses the job is worth calling next time.

The failure screen is the deliverable

Integrations run at 2am and fail silently. The first symptom is usually a number that does not reconcile, noticed a fortnight later by an accountant.

Ask every developer quoting: what do I look at on Monday to know last night worked?

The answer should be a page a non-technical person can read showing:

• What ran, when, and how long it took.
• How many records moved, per source.
• What failed, and a reason in plain words.
• A re-run button that cannot create duplicates.

Not a log file. Not an email nobody reads. A screen. This is the single most useful question in the whole quoting process and the answers will not be similar.

Decide ownership before you decide technology

Two systems will disagree about the same customer. One has "Smith Builders Ltd", the other has "Smith Builders", and somebody changed the phone number in one of them last week.

Before any pipeline is built, settle for each kind of record:

• Which system is the master?
• How are the two records matched to each other in the first place?
• What happens when both changed since the last run?
• What happens to the ones that cannot be matched — are they dropped, queued, or flagged to a person?

Matching on names produces silent duplicates that take years to unpick. The reliable approach is to store an explicit link between the two records the first time they are matched — by hand if necessary — and never guess again.

Expect a couple of meetings with your own people. A developer cannot make these calls for you.

Build the layer if you know more is coming

Point-to-point integrations are fine for the first two. By the sixth, each has its own credentials, its own retry logic, its own idea of what a customer is, and no two fail the same way. Nobody can produce an inventory of what talks to what.

An integration layer costs more up front and pays back on the third connection. One place for credentials, one queue, one retry policy, one log, one failure screen.

If this is your first integration and you know two more are coming, say so during quoting. It changes what a good developer proposes.

Make every run repeatable

Things fail. The question is whether re-running is safe.

What to ask for:

• Idempotency — running the same batch twice produces the same result, not two copies.
• Restartability — a run that died halfway can resume or be redone cleanly.
• A record of what was processed, so nothing is silently skipped after a failure.
• Backfill — the ability to reprocess a past period when something turns out to have been wrong.

That last one is worth insisting on. Every integration eventually needs a month rebuilt.

Credentials, and where they live

Ask where the logins for each system will be stored. Common answers include a configuration file in the repository, a shared spreadsheet, and one developer's laptop.

They belong in a secrets manager, granted per environment, with a record of who accessed what and a way to rotate one without a redeploy. Ask the question during quoting; the answer is usually quick to fix and expensive to ignore.

Comparing five quotes

Read for:

• Who challenged the real-time assumption?
• Who priced an off-the-shelf tool before proposing to build?
• Who described the failure screen without being asked?
• Who raised record ownership and matching?
• Who included backfill and safe re-runs?

Describe the systems, what needs to flow which way, how fresh it genuinely needs to be, and roughly how many records a day. Developers who do this work will ask what happens when the numbers disagree before they ask about your stack.