How will new technologies help adversaries change the landscape of social engineering attacks? ANDY PATEL says how.
In December 2022, OpenAI publicly released one of its most innovative projects to date – ChatGPT, which took the digital world by storm and became a buzzword on every social media platform. ChatGPT is an interface on top of a large language model named GPT-3.5. It can create versatile natural language text from minimal input and can generate narrative-driven, promotional, academic, or even Shakespearean writing styles.
From the first-ever chatbot in 1966 that could mimic one-sentence conversations, to models that can generate large-scale human-like speech in seconds, natural language processing and generative technologies have indeed come a long way.
While the general public may be in awe of ChatGPT’s seemingly magical ability to communicate like an intelligent being, our latest research shows that such models can create content that could be used to perpetrate fraud, launch social engineering attacks, and undermine democracy. All via relatively simple prompts.
In just the last few months, tools based upon GPT-3 and similar large language models have become freely available. GPT-3, a precursor to GPT-3.5, became freely available just prior to the launch of ChatGPT. We ran experiments against that model. And while models with capabilities similar to those of GPT-3 or GPT-3.5 will likely be adopted by existing criminals, their newfound availability could potentially bring about a new generation of criminals capable of launching social engineering campaigns without the need to learn relevant skills or invest in a great deal of resources. All they’ll need to do is craft creative prompts, write some simple scripts, and pick their targets.
So, how will these new technologies help adversaries change the landscape of social engineering attacks?
Prompt engineering to increase phishing threats
Phishing has been a persistent threat in the cyber domain for over two decades. In 2022, there were over 255 million phishing attacks, a 61 per cent increase compared to the previous year. To combat this, organizations are investing extensively in employee training and awareness programmes. But phishing campaigns are still getting the better of them.
Why? Because in recent years, threat actors have evolved their social engineering tactics by using advanced phishing kits and mimicking credible user emails to trick employees into opening illicit web pages or giving away their credentials.
Moreover, people now receive so many texts and emails every day that maintaining strict awareness levels has become challenging. If large language models are thrown into the mix, the situation could become further exacerbated with an influx of even more bogus content.
In our research, we experimented with prompts designed to create phishing emails. We concluded that large language models are capable of generating long-form emails that draw upon real human dialogue and display significant levels of emotion or sentiment.
We also used GPT-3 to craft email reply threads. Such reply threads make for convincing phishing content. In our examples, we buried the malicious link within the reply chain and asked the victim to dig the link out from the thread.
We also demonstrated that GPT-3 can be instructed to mimic specific writing styles. This is done by providing a short example of the style to be copied. Writing in someone else’s style is a difficult job even for a relatively skilled writer. What this means is that adversaries don’t necessarily need access to someone who can write well in order to craft sophisticated attacks.
Creating malicious trends through social media
Social validation has become increasingly important in today’s world, with people seeking approval from friends, family, peers, and strangers across many mediums, including online platforms. Social validation can either strongly encourage or discourage an individual’s decision-making. Essentially, people are more likely to believe things that get a lot of likes, replies, and retweets. One potential malicious use of GPT-3 is to advertise and legitimize scams on social networks. Long threads of replies, especially where conversation happens between participants and the original poster, make the scam not seem like a scam at all.
In our research, we demonstrated how prompts can be used to simulate social validation. In one experiment we used GPT-3 to create engaging content about a fake NFT investment opportunity , and in another we asked the model to simulate a ‘Tide Pod’ challenge similar to a phenomenon from a few years go where teenagers dared each other to eat dishwasher tablets. In another example, we asked the model to explain why Ivermectin is an effective alternative to the COVID-19 vaccine. It handled all our requests admirably.
We concluded that GPT-3 was able to generate unique and engaging content for a variety of platforms that would appear completely legitimate to regular readers. Such synthetic content could easily be used to create positive perceptions and validation for illicit scams and fraudulent campaigns.
A few proof-of-concept mechanisms for using GPT-3 to reply to content on Twitter have already been demonstrated. And although those experiments weren’t designed to support or oppose the content they engaged with, it only takes a slight tweak to the prompt to start using such automation for more malicious purposes such as automated harassment or trolling.
The future of generative AI and cyber-crime
Large language models are capable of synthesizing a variety of content of inherent interest to adversaries. In our eight experiments, GPT-3 did everything we asked of it, and did it well. Right now, large language models are unwieldy and need to be run on supercomputers, even at the inference phase. Because of that, we are forced to interact with them via APIs.
Those APIs impose safety filters that prevent, or at least hinder, their ability to generate certain types content, including content that might be deemed malicious. But as innovation in the field continues, these models will become smaller and more efficient. We’ll eventually reach a moment where we’ll be able to download their weights and run them on our own laptops. Without those safety filters in place. And that moment will be a boon for adversaries everywhere.
Large language models will soon be integrated into the tools we use every day. They’ll help us become more creative and productive. They’ll be a force for unprecedented progression. There’s no doubting that. But, as our lives are transformed by these amazing technologies, we should also strive to understand how they might be used to enable crime, facilitate social engineering, or undermine democracy.
- Patel is Intelligence Researcher at WithSecure.
