In Development

One interesting consequence of humanity’s current technological revolution is deepfakes. Once requiring mastery of intricate photo and video editing software, deepfake creation is now within reach of anyone with a smartphone and internet access.

 

The word “deepfake” comes from a combination of the terms “deep learning” (the study of AI systems whose design is inspired by the structure of the human brain) and “fake” (the fake content produced using such AI systems). The definition we will mostly stick to comes from Australia’s eSafety Commissioner, which defines a deepfake as a digital photo, video, or sound file of a real person that has been created with AI to make an extremely realistic but false depiction of them doing or saying something that they did not actually do or say.

 

It is worth noting that there is no globally consistent definition, with some extending to include document forgery and the generation of realistic-looking images of people or things that don’t actually exist.

 

 

(1) What does deepfake creation readily enable today?

 

According to scam data from the Australian Competition and Consumer Commission, investment scams consistently eclipse all other scam types (e.g. invoice, romance) in terms of the total amount stolen each year by a wide margin, despite the absolute number of occurrences being relatively low. My best guess is that scammers are much more likely to seize “all the money in the bank” if their victims genuinely believe in a scheme promising strong returns.

 

A fake online business or a scammer masquerading as an exiled Nigerian Prince is less likely to extract the entirety of one’s life savings. One common manifestation of investment scams is powered by deepfakes of celebrities. At this point I’ve lost count of the number of times I’ve seen an eerily convincing clip of Elon Musk talking about a new cryptocurrency or investment opportunity.

 

Scammers would also like to extract personal information from victims, known as “phishing”, where celebrity deepfakes are opening up new avenues of attack. If I were a teenage girl, blissfully unaware of the importance of information privacy, and my favourite singer Taylor Swift offered me free stuff in exchange for my email address, home address, a few selfies, the name of the school I attend, and a 30-second video recording of me explaining which Tay Tay song is my favourite and why, that form would be filled out faster than front row seats on the Eras Tour.

 

Part of what makes these types of scams “readily enabled” by deepfake technology is that there are countless images, videos, and audio clips of celebrities online. The immense online presence of famous people provides the perfect dataset for deepfake generation using their likeness. Generally speaking, the more images, audio clips, and videos you have, the more convincing you can make your deepfake.

 

Another example where deepfakes can readily be deployed is in the context of romance scams. Scammers without a PhD in Adobe Photoshop and hundreds of spare hours used to rely on the few images of people they could find online. As such, fake social media profiles were generally kept sparse for a lack of coherent content. Historically, this often made them seem a bit dubious.

 

Now anyone can generate a convincing fake profile using the likeness of a real person (or even an entirely fictitious individual). Such profiles are often indistinguishable from the genuine article, even to a discerning eye. In this way, scammers have unlocked access to a wider range of potential victims. The speed with which content can be generated also allows them to target more people, and this is to say nothing of AI’s ability to autonomously hold a text-based conversation.

 

Outside of scams, deepfakes can be used to great effect in conjunction with illicit document forgery services like OnlyFake, whose creator was recently charged in the US. A fake ID can be used to satisfy KYC requirements, and an accompanying deepfake handles liveness checks (tests used to validate that a customer on the other end of a digital interaction is the real person they claim to be, e.g. when you are required to take a selfie or rotate your face in front of a camera as part of opening a new bank account).

 

All that’s left is for criminals to shop around for the financial institution with the weakest onboarding controls. With few technical bottlenecks limiting the speed of synthetic identity creation, small numbers of bad actors could potentially create complex networks of accounts using stolen or fake identities to launder money or finance terrorism at a scale previously reserved for highly organised groups.

 

 

(2) What could deepfake creation enable with access to the right data?

 

We’ve talked about the use of deepfakes to fool victims into surrendering information and assets. What about fooling the victim’s bank? Here, criminals need to work a bit harder to source the right data first. With access to a sufficient quantity of photos, videos, and audio recordings of a real person, their likeness can be imitated with relative ease.

 

Banks still relying heavily on voice authentication for telephone banking may find these controls are already becoming less effective on their own, as AI models are able to clone voices with more accuracy and less data. ElevenLabs, an AI company specialising in audio and voice cloning, requires users to submit one minute of high-quality audio for what is presumably a convincing voice clone. Other tools purport to require only seconds of someone speaking to produce a clone which may not be identifiably artificial over a crackling phone line.

 

Perhaps the most worrying aspect of the Taylor Swift example is the door it opens for even more pernicious attacks. While minors are less likely to have bank accounts of their own to steal from, their parents almost certainly do. A well-timed audio deepfake could convince a frantic parent to recite credit card details over the phone to their child in perceived distress.

 

There are ways this can be made especially convincing depending on the information phished. A scammer could also ask for information as part of a fake celebrity giveaway such as “what’s a fun nickname you call your parents?”, as if a near-perfect voice clone isn’t convincing enough… Here’s a real-world example where a man was tricked into paying for what he thought were his son’s bail and legal fees.

 

Despite the clear threat to individuals, businesses are certainly not immune. A multinational company was defrauded of USD 25 million after scammers impersonated the chief financial officer of the company using audio and video deepfakes. An employee was tricked into paying the scammers in a fake online conference call. Businesses are simply collections of people and things after all, and sadly, people can be exploited.

 

With the right data (i.e. a sufficient collection of audio, video, and images of a person), deepfake technology has given rise to a new brand of highly personalised fraud and scams, where impersonators are able to craft immaculate digital disguises and assume a raft of online identities that can fool even the most discerning among us.

 

 

(3) What could deepfake technology enable in the future?

 

This part of each blog will be largely speculative and is likely to age poorly.

 

Future systems could allow for even greater levels of personalisation in deepfake-driven deception. As well as the fakes themselves constantly getting better, AI’s ability to understand and exploit the human condition may well improve in parallel. In the same way that Anthropic’s new Mythos model has reportedly found previously unknown exploits in critical software, who’s to say some future system, highly attuned to the proclivities of humankind, won’t be able to exploit us in subtle, unforeseen ways?

 

We’re talking about AI that can be used to analyse information about and profile potential fraud victims to come up with the most effective strategies, used in tandem with an AI that can generate very realistic deepfakes to bring any story to life. Perhaps the vulnerability is a feeling of loneliness, a gambling addiction, a grudge, or something too subtle to capture with an English phrase, but that an emotionally intelligent AI could build the perfect criminal strategy to exploit.

 

Without wanting to leave readers with that feeling you get at the end of a Black Mirror episode, future deepfake technology also has the potential to be used for good. One example that comes to mind is undercover police operations. Highly realistic, real-time deepfakes could potentially be used to catch child predators, although there are clearly ethical concerns with this use case. If the good guys can produce better deepfakes that don’t have the known tells of current methods, it will be increasingly difficult for predators to know they are dealing with a minor and not a police detective.

 

This idea can be extended to any crime or investigation with an online component. Increasingly sophisticated synthetic victims with a realistic online presence can be used to bait scammers. Industry is already taking action, with the Commonwealth Bank of Australia last year launching a fleet of AI bots designed to waste scammers’ time and gather intelligence. In this way, future deepfake technology also has the potential to pave the way for more targeted, automated crime fighting.

 

 

What can be done?

 

Thankfully, many governments, institutions, and identity verification providers are already wise to the threat of deepfakes. The most obvious mitigant is detection, i.e. systems designed to analyse subtle features of media (e.g. the consistency of lighting and shadows in an image) to spot deepfakes. These controls are likely to struggle into the future in the same way that detecting an AI-written essay has proved to be virtually impossible. There are some tells, like the use of em dashes to run with our analogy, but detection systems will always be playing catch-up to evolving methods of generation.

 

Such controls are more effective when used in conjunction with others, such as cryptographic signatures at the point of media capture. This type of control is about proving the authenticity of media, and gives banks, for example, a way to verify the time and device of capture of a video snippet submitted for a liveness check, which is particularly useful when guarding against injection attacks.

 

There are also plenty of highly effective, low-tech mitigants such as the use of safe words among families and coworkers over digital channels, something encouraged by the National Cybersecurity Alliance in the US. A layered approach is necessary to cover this complex and shifting attack surface.

 

There is another key defensive layer that is perhaps less obvious but critically important: Intelligence. The best solutions are those to clearly defined problems. As the pace of technological advancement accelerates, so does the pace at which criminals employ new methods. As criminals rapidly gain more tools and opportunities, the volume of intelligence in the aether increases sharply, along with the need for businesses to gather, analyse, and operationalise it.

 

Indeed, there is also a heightened regulatory imperative, with the importance of intelligence playing a central role in recent reforms to Australia’s anti-money laundering regime and the incoming Scams Prevention Framework. The proper management of intelligence is quickly becoming a cornerstone of a robust approach to financial crime risk management in the age of artificial intelligence, and is another area in which AI can be used to great effect in the fight against crime.

 

Deepfakes are but one small indicator of the incoming financial crime intelligence explosion.