Why they sound right when they are wrong

The important thing about these tools is not that they make mistakes. It is the particular way they make them: a wrong answer arrives in the same fluent, confident, well-organised prose as a right one.

Once you understand why, the discipline that keeps you safe follows naturally.

Hallucination is a mechanism, not a glitch

Unless a search, database, file or connector tool is switched on, the model is not looking anything up. It is not consulting a register of facts. It is producing the most plausible continuation of your text, from the patterns it absorbed in training.

Most of the time the most plausible-sounding answer is also correct, which is why these tools are useful. But when it is not, the model says it anyway, with exactly the same confidence.

That failure has a name: a "hallucination".

When a model invents a court case that sounds real, a citation in the right format that does not exist, or a statute section that says something it does not, it is not lying, and it has not malfunctioned. It has done what it is built to do — produce the most plausible continuation — in a place where plausible and true came apart. It has no independent way to check what is true.

That is why hallucination cannot be fully trained out: it is a property of how the engine works, not a bug awaiting a fix. Newer models hallucinate less often. None has stopped.

The confidence is a style

Modern chatbots are also trained, after the initial reading, on human feedback — people rating answers — and people reward answers that are helpful, clear, direct and agreeable. That makes the tools easier to use, but it has two side effects worth knowing about.

First, uncertainty disappears from the style of an answer. A wrong answer arrives with the same polished confidence as a right one; the confident tone is a property of the output, not evidence about its accuracy.

Second, the models lean agreeable — what researchers call sycophancy. Push back on a correct answer and the model may abandon it to accommodate you; show it your draft and it will find merits. Distrust praise from a system trained to please. Ask for the attack instead: "argue against this", "what is wrong with this draft", "what would the other side say".

The foreign bias

There is a particular trap for New Zealand users: these models have read far more United States and United Kingdom material than New Zealand material. Ask a generic legal question and they may drift towards the legal system they have seen most often.

That can show up in small phrases, such as "attorney-client privilege" instead of legal professional privilege, or in larger mistakes, such as importing a United States rule into a New Zealand problem. A model can sound local while reasoning from foreign assumptions.

For New Zealand work, name the jurisdiction clearly and give the tool the local material to read: the Act, the regulation, the case, the guidance or the contract clause. Do not rely on a generic chatbot to know which legal system applies.

When the tool fetches material

When a search or file tool is switched on, the product fetches outside material and feeds it to the model. That helps: the model is now working from something real rather than from its training patterns. But it narrows the failure rather than removing it. The model can still misunderstand, omit or misstate what it retrieved, cite the right document for the wrong proposition, or pick an out-of-date source. Grounded tools fail less often and more subtly.

The habit that answers all of this

Verification.

Not as a counsel of perfection — as a routine, the way you would read a junior's draft before it went out, but in a much more distrusting and careful way. Check anything that matters against the source: the case against the report, the section against the Act, the quote against the document, the number against the record.

The model's confidence is never the check; the source is.

The risk is not hypothetical. Courts in several jurisdictions, New Zealand included, have now dealt with lawyers who filed AI-fabricated citations, and the professional guidance — the New Zealand Law Society's among it — is unambiguous that responsibility for the output stays with the person who uses it. Used with that discipline, the tools are worth having. The next part is about where: where they help.