How to spot when AI is wrong, and the scams to watch for

There are two very different worries hiding inside one question. The first is the AI itself being wrong, telling you something false in the same calm, friendly voice it uses when it is right. The second is people using AI to fool you, with cloned voices, fake invoices and emails that no longer have the tell-tale spelling mistakes. Both are real. Both are manageable once you know what you are looking at. I use these tools every day to write books and music, and I have been caught out by the first kind more than once, so this is the plain version I wish someone had given me early.

Quick answer

AI predicts what sounds right, and what sounds right is usually true, but not always. So use it freely to shape words, plans and ideas, and verify anything it presents as a fact, a date, a price, a law or a medical claim before you rely on it. Separately, AI is now powering very convincing scams. Slow down on anything urgent that involves money or passwords, verify the request through a separate channel you already trust, and agree on a family code word. Real organisations never ask for your password or full card details.

Part 1: when the AI itself is wrong

Hallucinations, explained plainly

When people say an AI “hallucinated”, it sounds dramatic. It is actually quite ordinary. These tools work by predicting the next most likely word, over and over, based on patterns they learned from enormous amounts of text. Most of the time the most likely words happen to be true, because true things are written down more often. But the machine is not checking a fact. It is producing something that sounds correct.

The catch is the tone. When ChatGPT or Gemini or Claude is right, it sounds warm and certain. When it is completely wrong, it sounds exactly the same. There is no nervous wobble in its voice, no “I think”. It will invent a court case, a quote, a statistic or a book that does not exist, and present it with the same confidence as the genuine ones. The mistake I see most often is people trusting the tone instead of the facts.

The one rule worth memorising

Here is the whole thing in a sentence: shape with it, verify its facts.

Use AI freely for the things it is genuinely brilliant at. Drafting an email. Rewording a clumsy paragraph. Brainstorming names. Building a plan, a packing list, a study outline. Explaining a hard idea three different ways until one clicks. None of that needs verifying, because you are the one judging whether it reads well or helps you think.

But the moment it states something as a fact, a date, a price, a law, a dosage, a name, a number, treat that as a claim to be checked, not a truth to be copied.

How to verify, in practice

You do not need to fact-check every word. You need a quick habit for the handful of claims you are about to rely on.

• Check one trusted source. For a fact, open the actual website. A price goes to the seller. A law or government rule goes to the .gov.au site. A health claim goes to your doctor or a service like Healthdirect, not a chatbot.
• Ask the tool for its source, then check the source. Say “where did that come from?” If it gives a link, open it and confirm the link is real and says what the AI claimed. Sometimes the link itself is invented.
• Cross-check across two tools. Ask the same question in ChatGPT and in Gemini. If they disagree, you have found exactly the spot that needs a human and a real source.

What AI simply cannot be trusted for

Some things are not occasional mistakes, they are built-in blind spots:

• The very latest news. Most models were trained up to a certain date and do not know what happened this morning unless they are actively searching the web in front of you.
• Exact figures and dates. Precise numbers are where hallucinations love to hide. A roughly-right summary can carry a confidently wrong figure inside it.
• Maths and counting. These tools are language machines, not calculators. For anything that matters, use a calculator or a spreadsheet.
• Anything about your private records. It has never met you. It does not know your bank balance, your medical history, your appointments or your account. If it answers a question about your personal details, it is guessing, and it does not have the data to put in anyway.

That last point is also a safety line. You are the Captain. AI is the crew. Never put your passwords, bank details, card numbers or ID into a chatbot, no matter how helpful it seems.

Part 2: the human scams AI now powers

The same technology that drafts a friendly email can draft a very convincing lie. Scammers have noticed. In Australia this is no longer rare, and the old advice about “watch for bad spelling” has stopped working, because the spelling is now perfect.

The scale is worth knowing, because it is easy to think this only happens to other people. Australians reported more than 2 billion dollars in scam losses in 2024, according to the National Anti-Scam Centre, and losses rose again through 2025. The biggest single category by money lost is investment scams, followed by payment-redirection and romance scams. These are not fringe crimes. They are an industry, and AI has made the convincing part cheap.

Here is what is actually circulating:

• Voice cloning and family-emergency scams. A few seconds of audio, often lifted from social media, is enough to mimic a voice. You get a call or a voice message that sounds like your son, your mum, a grandchild, upset and needing money fast. The voice is fake. The panic is the weapon.
• Fake invoices and payment-redirect scams. A real-looking invoice arrives, or an email mid-conversation says the bank details “have changed, please pay to this new account”. The money goes straight to the scammer.
• Romance scams. AI writes tireless, charming, fluent messages at scale, and can even generate the photos. The warmth feels genuine. The person on the other end may not exist.
• Deepfake videos and images. A clip of a celebrity, a politician or even a local figure “endorsing” an investment, generated from scratch. If it seems too good and asks for money, it is bait.
• Far more convincing phishing emails. Polished, correctly branded, addressed to you by name, with no clumsy errors to give it away.

Practical defences that actually work

You do not need to be technical. You need a few simple habits, and the discipline to use them when you feel rushed.

• Agree on a family code word. A single word only your family knows. If a panicked call asks for money, ask for the word. A scammer cannot answer.
• Verify any money request through a separate known channel. Hang up and ring the person back on the number you already have saved. For a changed-bank-details email, phone the company on a number from their official website, never the number in the email.
• Slow down when urgency is pushed. Manufactured urgency is the common thread in nearly every scam. “Act now, or else.” Real life rarely needs your bank details in the next ninety seconds. The pressure itself is the warning sign.
• Check the real sender. Look at the actual email address, not just the display name. Hover over links to see where they truly go before clicking. On a phone, press and hold to preview the link.
• Remember what real organisations never do. Your bank, the tax office and legitimate companies will never ask for your password or your full card details by email, text or phone. If someone does, it is a scam, full stop.

None of this requires you to be afraid of technology. It just asks you to keep the controls in your own hands, which is the whole idea anyway.

FAQ

Which AI tool hallucinates the least? All of the major ones still do it, including the newest paid versions of ChatGPT, Gemini and Claude. The tools that search the web while they answer and show you their links are easier to check, because you can open the source yourself. The safest approach is the habit, not the brand: verify any fact you are about to rely on.

If AI gets facts wrong, is it even worth using? Yes, for the right jobs. It is genuinely excellent at shaping language, ideas, drafts and plans, where you are the judge of whether it works. Save the verifying for facts, numbers, dates and anything legal, financial or medical. Used that way, it is a brilliant crew member, just not a reference book.

Someone called sounding exactly like my family member. What do I do? Do not act on it in the moment. Hang up and call that person back on the number you already have. Ask for your family code word if you have one. A cloned voice cannot survive a simple “what was the word?” or a quick callback.

Module 5 of the Cleared For Take-Off course walks through verifying AI and spotting these scams step by step, and you can start with the free Module 1 to get the foundations first.