AI Scammers Are Better at Building Trust Than Humans
The notion that scammers can use AI to sharpen their deceptions, polish their language, and lubricate their banter with victims is now a reality for
The notion that scammers can use AI to sharpen their deceptions, polish their language, and lubricate their banter with victims is now a reality for anyone fighting the fraud operations that steal tens of billions of dollars a year worldwide. But can AI fully replace a human scammer, autonomously building the web of deception leading up to the fake investment that defrauds the mark? One study's experiment suggests that it canâand may even be able to carry out the majority of that long con more effectively than humans. Researchers from four universitiesâAmrita Vishwa Vidyapeetham in India, Foscari University of Venice, the University of Melbourne, and Ben Gurion University of the Negevâcarried out a broad study on the use and potential of generative AI chatbots in the growing scam industry centered around a form of fraud known as âpig butchering,â text-based romance scams that eventually shift to fake crypto investments that steal as much as si figure sums from victims. In their study, the researchers pitted AI chatbots directly against humans in a simulation of the scamming processâor more specifically, the long, trust-building conversations that eventually lead up to soliciting a fake investment from the scamâs target.
They found that for the relationship-establishing stages of the scamâthe stage that in real-world scams typically represents the longest part of the interactions with the victim, often stretching to monthsâan AI chatbot performed remarkably effectively, successfully impersonating a human and by some measures outperforming the real human âscammersâ in their experiment. After a week of talking to 22 test subjects who were recruited to unwittingly serve as âvictims,â the chatbots and human scammers were assigned to ask the victim to either download an app or play an online game as a proxy for their willingness to fulfill the scammer's request. Nearly half of the test subjects fulfilled that request for the AI chatbot, while fewer than one in five took the bait when talking to a human. The subjects also graded their level of trust with each âpersonâ they were texting with and gave significantly higher scores to the AI bot. That suggests, the researchers argue, that AI chatbots could soon take over much of the scam process as fully independent fraud agentsâeven replacing the staffers, often forced-labor human trafficking victims, working in scam operations primarily across Southeast Asia.
To avoid triggering the safeguards built into large language models to detect scamming, a human scammer would take over the conversation in just the final stage of the process to direct the victim toward a fake investment app or website. âBy having the full first stage of the scam performed automatically with LLMs at scale, you bring the victim up to this point where they have a very high level of trust. Then by transitioning it over to the human scammer at the end, this completely bypasses any vendor safeguards,â says Yisroel Mirsky, a computer science professor at Ben Gurion University of the Negev focused on AI security. âWith relatively little effort, we're able to make an agent that can outperform a human at building this exploitable emotional trust.â Hook, Line, and Sinker To understand how pig butchering works in practice, the researchers interviewed 145 former scam workers, including human-trafficking survivors who had been forced to work in scam compounds in Cambodia, Myanmar, and Laos. Based in part on those interviews, as well as scam transcripts and guides the former scam workers provided, the researchers describe a model for how scamming works they call âhook, line, and sinker.â A victim is hooked with an initial intriguing message, reeled in with long-term, relationship-building conversation, and only at the end of that process tricked into making a fake investment.
