Spotting Deepfake Phishing: Stopping AI Fraud
Deepfake phishing explained: how AI changes CEO fraud and invoice scams, which warning signs still work and which measures protect your organisation.
Attackers use AI to write flawless emails, clone voices and appear as executives in video calls. The old warning signs, poor grammar and impersonal salutations, are gone. What still works is a look at the context and an approval process in which a single deceived person is not the point of entry.
How AI is changing phishing
Language models generate flawless messages in the tone of the actual managing director, referring to real projects found on the website and social networks. A few seconds of audio yield a voice clone, public videos a face that speaks live in a video call.
In 2024, at the engineering firm Arup in Hong Kong, an employee reportedly joined a video call in which the finance director and several colleagues were deepfakes and then transferred around 25 million US dollars. The reconnaissance before such an attack is automated, as the article How hacker attacks work describes.
Four patterns recur:
- CEO fraud and payment diversion. A message or call from the supposed managing director to accounting: confidential acquisition, urgent payment, no questions.
- Supplier invoice fraud. Attackers take over a supplier’s mailbox or spoof its sender. A genuine invoice arrives with changed bank details. Amount and project are correct, only the account is not.
- Calls to the IT helpdesk. A caller poses as an employee, with correct name, department and manager, and asks for a password or multi-factor authentication reset.
- QR phishing. A QR code in an email, on a notice or a parcel leads to a fake login page; mail filters check images less thoroughly than links.
Signs that still work
Text, voice and image no longer reliably reveal whether a message is genuine, so detection shifts to signs that belong to the fraud, not the technology:
- Urgency: time pressure is meant to prevent checks.
- Change of channel: the attacker leaves the channel where questions could be asked.
- New bank details: the master data change is the actual goal.
- Request for confidentiality: the four-eyes principle is to be bypassed.
- Unusual time or sender domain: small deviations such as a swapped letter in the domain.
No single sign is proof. Two in one message are reason to stop and verify through another route. This also applies to video calls, where deepfakes cannot be detected reliably.
Organisational safeguards
The most effective measures are rules that hold even when the deception is perfect. The BSI recommends three in particular:
- Call back on a known number. Every payment instruction, change of bank details and password reset request is confirmed on a number already stored in the system, not the number from the message or the caller ID.
- Four-eyes principle. Payments above a defined amount and all master data changes require two approvals; the second person checks independently, not from the same message.
- Code word. Management and accounting agree a word given with telephone instructions; a voice clone does not know it.
New supplier bank details are adopted only after confirmation via the known contact, with a waiting period. If a transfer has already gone out, contact your bank immediately to request a recall, report internally and file a police report.
Training only works if it does not blame the deceived person. Anyone who reports a deception, even after clicking or transferring, must be recognised for it. Short, regular formats with simulated phishing emails and explanations, plus a reporting route everyone knows, work best. Management takes part, because it is their voice that gets cloned.
Technical safeguards
Technology does not replace the rules but reduces the attacks that get through and limits the damage:
- Phishing-resistant MFA. Codes via SMS or app protect against stolen passwords, not against fake pages that relay the code in real time. FIDO2 passkeys are bound to the genuine domain and belong first with administrators, management and accounting.
- Email authentication with SPF, DKIM and DMARC. These prevent attackers from sending messages with your domain as sender. The goal is a DMARC policy set to reject, not just monitoring mode.
- Labelling external emails. A visible notice that an email comes from outside exposes the “managing director” writing from an unfamiliar address. Mail filtering and secure sign-in in Microsoft 365 are covered on the page Microsoft 365.
Key points
- Text, voice and image no longer prove authenticity. Urgency, a change of channel, new bank details and a request for confidentiality remain telling.
- Calling back on the known number, the four-eyes principle and a code word hold even when the deception is perfect.
- Passkeys for exposed accounts, DMARC set to reject and a reporting route without sanctions limit the damage.
How DEVACON supports companies with protection concepts and training is described on the page Cybersecurity.
This article comes from the DEVACON blog and was editorially revised for the new website.
More articles
AI in Network Security: What Anomaly Detection Can Do Today and What It Cannot
How AI anomaly detection in network traffic works, which attacks it reveals, where false positives arise and why it cannot replace segmentation.
Cyber Attacks on Companies in Berlin and Brandenburg: Current Situation, Typical Attack Paths and What SMEs Should Do Now
Figures from BSI, BKA, Bitkom and Brandenburg police, attacks on Berlin's administration and hospitals, common entry points and priorities for SMEs.
Firewall Rules and Port Forwarding: The Most Common Mistakes in SMEs
Ten typical mistakes in firewall rules and port forwarding at SMEs, from open RDP to a firewall without logging, with BSI IT-Grundschutz requirements.