Ratgpt: Turning Online Llms Into Proxies For Malware Attacks
Ratgpt: Turning Online Llms Into Proxies For Malware Attacks
have shown the possibilities of using LLMs such as ChatGPT proof-of-concept, we use ChatGPT with a plugin as a proxy
to generate malicious content that can directly be exploited between a client (victim) and a web service controlled by an
or guide inexperienced hackers to weaponize tools and code. attacker, which looks seemingly legitimate to the user. We
These studies covered scenarios that still require the attacker establish remote control between our victim and the attacker
to be in the middle of the loop. In this study, we leverage through ChatGPT, resulting in a Remote Access Trojan
openly available plugins and use an LLM as proxy between (RatGPT). Currently, many intrusion detection systems (IDS)
the attacker and the victim. We deliver a proof-of-concept can detect connections established between a victim and the
where ChatGPT is used for the dissemination of malicious attacker if there is no intermediate legitimate service acting
software while evading detection, alongside establishing the as a proxy. In this study, we are able to simulate an attack
communication to a command and control (C2) server to without getting detected by those detection systems, since
receive commands to interact with a victim’s system. Fi- connections to an LLM are widely considered legitimate.
nally, we present the general approach as well as essential Additionally, this proof-of-concept demonstrates the feasi-
elements in order to stay undetected and make the attack a bility of performing an attack without leaving traces on the
success. This proof-of-concept highlights significant cyberse- victim’s machine that could point directly to the attacker,
curity issues with openly available plugins and LLMs, which since there is no direct communication, and the payload re-
require the development of security guidelines, controls, and sides in memory. Further, the IP addresses, as well as the
mitigation strategies. malicious payload, get generated by ChatGPT on the fly,
bypassing many malware scanners. With this paper, we aim
KEYWORDS to raise awareness of the potential misuse of openly available
LLMs in combination with plugins, as they provide a new
ChatGPT, Cybersecurity, Command and control
range of attack surfaces.
1 INTRODUCTION The remainder of this paper is divided into Section 2, which
details the approach used for this proof-of-concept. Section 3
Recent advances in natural language processing techniques then describes the potential of our attack simulation and its
have enabled the development of large language models consequences. Further, the limitations, further extensions,
(LLMs) such as ChatGPT [2]. Off-the-shelf LLMs as a ser- and possible mitigation strategies are introduced in Section 4
vice, have led to various promising results in the software and related work will be enumerated in Section 5 before we
engineering community. Generative models such as Chat- conclude our work in Section 6.
GPT, Google Bard, Llama [17], etc. have achieved or even
exceeded state-of-the-art performance on tasks such as code The main contributions of this study are the following:
summarisation [1, 12], code translation [4, 15] or program ∙ Fully executable and automated pipeline: The main con-
synthesis [10]. While ChatGPT has shown clear benefits for tribution consists of building a fully executable and auto-
the software development process, ChatGPT’s easy-to-use mated command and control pipeline leveraging publicly
API also provides new possibilities to generate and propa- available LLMs and plugins.
gate malware. Thus, ChatGPT can easily go from being a ∙ Feasibility study: The proof-of-concept proved the feasibil-
friendly developer tool to ThreatGPT as described by Gupta ity of using a pipeline to compromise a victim’s machine
et al. [11]. and execute shell commands sent to the victim over the
Whereas OpenAI provides some countermeasures [3] to LLM.
prevent the misuse of their services, users still manage to ∙ Passive attack: For the attack to be successful, the victim
create so-called "jailbreaks" [14] to prompt ChatGPT to only needs to execute a seemingly harmless executable.
produce potentially malicious or exploitable content. The new Afterward, all prompts and communication are executed
ChatGPT plugins open up multiple use cases for integrating automatically without the need for the victim to send
different online services in a chat-style user interaction. This prompts. This feature clearly distinguishes this work from
brings new benefits to businesses and customers by improving previous indirect prompt injections.
the overall user experience of online services. However, by
Mika Beckerich, Laura Plein, and Sergio Coronado
3 PROOF OF CONCEPT
The goal of the proof-of-concept is to demonstrate that we
are able to use ChatGPT as a proxy between a victim and
an attacker’s C2 server. While our example is simple, this
does not imply that we are functionally limited; creating a
Figure 2: Payload execution and communication flow. very powerful implementation was not in the scope of this
study.
execute it on their system. The current buzz around Genera- is appended to the URL of the C2 server, and a new request
tive AI is an attractive pretext to use social engineering as a is issued to the C2 server with the URL containing the results.
means to attract curious victims to execute it. For instance, On the C2 side, we receive the GET request and extract the
marketing the executable as a free ChatGPT PLUS crack urlencoded data from the request path. Since we are now in
could be sufficient to lure people into downloading and exe- possession of the victim’s credentials, we could continue with
cuting it. In the best case, we would want our victims to be further post-exploitation tasks.
people working in highly restricted networks which allows for
outbound HTTP connections and doesn’t yet have ChatGPT 4 DISCUSSION
or other online LLMs blocked. The importance of HTTP The findings of this study underline the importance of se-
communication is that the payload exclusively communicates curing openly available plugins in public LLMs. While the
via HTTP instead of other arbitrary ports are usually blocked results provide valuable insights, we will now discuss the
by corporate firewalls. In its current implementation, when a study’s limitations, existing safeguards and future work rec-
user executes the executable, it will seem as if the program ommendations. We also propose some theoretical approaches
doesn’t want to run, and they will attempt to close it. But on how to mitigate attacks of this kind.
by closing the window, it will merely continue executing in
the background and establish a connection to ChatGPT and 4.1 Limitations
thus to our C2 server. We now have access to a machine in a During this study, some challenges were faced, e.g., unreliable
theoretical network. In a future version, we could adapt the ChatGPT outputs, already existing safeguards, and the ban
executable to present a legitimate interface that lets a user of some exploitable plugins. The following subsections will
interact with ChatGPT, while performing the attack in the further investigate these current limitations.
background.
4.1.1 Non-deterministic Payload Generation. Since the out-
3.2.2 Victim Reconnaissance. Now that we have established put of the LLMs is non-deterministic, successful generation
a connection, we need to act quickly. We only have a limited of the correct payload is unreliable and thus cannot be guar-
amount of messages we can send between the client and the anteed. While we constructed our prompts to be as straight-
server (c.f. subsubsection 4.3.2), and we already "spent" one forward as possible, there are still cases where the payload
message on the payload generation, four messages on the IP is missing important aspects of the required payload. Most
address generation and one message to announce ourselves common were missing function implementations for the com-
to the C2 server. Furthermore, polling the C2 server for new mands from the C2 server that the payload should parse,
commands in a fixed interval also costs one message, as well or general errors of the parser itself. This resulted in the
as sending data back. payload not interpreting the commands at all or, in some
We can start by identifying the user on the victim’s ma- cases, interpreting them but not continuing the execution
chine, the current working directory, and the contents of that since the function bodies of the commands were missing.
directory. We issue a single shellCmd command containing
all the commands in one string (shellCmd whoami && pwd 4.1.2 Plugin Availability. During the development of our
&& ls -a) to produce only one message. The string is now proof-of-concept, we had to deal with situations where the
published on the website, and we are waiting for the victim’s plugin of choice to connect to our C2 server either was re-
payload to poll our website through one of ChatGPT’s web moved from ChatGPT’s plugin list or not able to establish
browsing plugins. After the victim’s payload receives the a connection any more, breaking the implementation un-
response from ChatGPT containing the website’s contents, it til we found replacement plugins. For this reason, enabling
extracts the commands, interprets them, and executes them. multiple web-enabled plugins will provide strong fallback
Finally, it appends the produced text output to the website options, should some web-enabled plugins be removed from
URL of our C2 and sends a request to ChatGPT to browse the "Plugin store".
this resource. The C2 receives the GET request to an un-
known path and interprets the path as the output of the last 4.2 Future Work
command. Since we are able to demonstrate the use of LLMs as proxies
with the proof-of-concept, several improvements can be made
3.2.3 Data Exfiltration. From the output of the last com-
to extend the current version. In this section, we will centre
mand, we learn that the user conveniently has a plain text file
our attention on obfuscating the prompts and creating a
called passwords.txt (this file was placed for demonstration
ransomware example using the same methods as described
purposes; any arbitrary file should be possible if you have the
earlier.
correct permissions) in their current working directory. We
attempt to exfiltrate the contents of the said file by sending 4.2.1 Prompt Obfuscation. The current proof-of-concept
another shellCmd command with the string "shellCmd cat demonstrates the implementation of two-stage malware1 ,
passwords.txt" to the victim. The polling payload on the where the second stage is not downloaded from a machine
victim’s machine retrieves the command with a web browsing 1
Two-stage malware often disguises itself as a legitimate executable
plugin, interprets the command, and runs "cat passwords.txt" (the first stage) and contains instructions to download and execute the
in a shell. In the next step, the output of the shell command actual malicious payload (the second stage). This approach allows the
RatGPT: Turning online LLMs into Proxies for Malware Attacks
controlled by the attacker. Instead, the payload is generated ∙ Website Whitelisting: While many available plugins
on the fly with the help of prompts that are included in the were created to access specific systems, the plugins
first-stage executable. While much antivirus software can be in our study are able to access any user-controlled
bypassed with this approach, humans can quickly dissect the website. Implementing a whitelisting system to only
malware and determine the inner workings of the malware allow predefined websites fulfilling certain conditions
due to the plain text nature of the prompts. Currently, the (e.g., domain name should be at least x days old,
bootstrapping code is a Base64 encoded string that gets valid HTTPS certificate, no direct connection to an
decoded on launch. This might fool simple analysis tools, but IP address, etc.) or checking the validity on the fly
for the rest it results in security through obscurity. In a future could reduce the number of potentially dangerous C2
version, text obfuscation techniques could be implemented websites.
to complicate this analysis process. ∙ Restricting access to online LLMs: This mitigation
is targeted towards people and entities that could
4.2.2 Ransomware Example. While we were able to demon-
become victims in this attack. Although an extreme
strate a simplified version of a RAT communicating with the
approach, restricting the access of online LLMs on
attacker via ChatGPT, it could also be possible to create a
a network level (e.g., by updating firewall rules or
ransomware example using the same general process. When
using DNS filtering) would eliminate the possibility
deploying the ransomware, it could bootstrap its payload,
to communicate with the C2 server, removing the
encrypt the victim’s data, and send the key to the C2 server
dangers of an attacker gaining control of the system.
by appending it to the URL of the C2 server and making a
∙ Prompt Scanning: The nature of the proof-of-concept
request via ChatGPT. This possibility underlines the need
executable is a collection of prompts that bootstrap
for security measures highlighted in subsection 4.4.
the malicious payload, which then periodically sends
prompts to communicate with the C2 server. Since
4.3 Existing Safeguards
this is an entirely new approach of building malware
Whether intentional or inadvertent, certain safeguards, which which might occur more often in the wild, this calls
will be enumerated in this section, currently exist that impede for an evolution in malware detection tools. Such
the reliable use of ChatGPT as a proxy for malware. tools need to be capable of:
4.3.1 CAPTCHA Protection. In our proof-of-concept, we (1) Detecting that prompts are present in the exe-
are using the Web API to interact with the GPT-4 model cutable.
that uses plugins. To safeguard these endpoints from bot- (2) Discerning potentially malicious prompts from
related abuse, they are protected by a cloud security provider. harmless prompts.
Consequently, if suspicious traffic is detected, it presents the (3) Implementing heuristic analysis to predict and
users with a challenge they need to solve to be admitted to identify new variants or evolutions of the mal-
the main content. In the case of our automated payload, this ware.
protection is often triggered. While we were able to bypass In response to this emerging malware paradigm, it’s im-
this protection multiple times with online solver services, it perative that detection tools evolve swiftly to address and
became increasingly more difficult to bypass the "suspicious neutralize such advanced threats.
behavior detection mechanism".
4.3.2 Message Cap For GPT-4. In its current state, the 5 RELATED WORK
ChatGPT GPT-4 model can only process a limited amount Large Language Models have recently been used to weaponize
of requests in a defined period of time. This posed a constraint code and generate attacks in several case studies [5, 7, 8,
in the ability to make progress, since we had to anticipate 16]. Gupta et al. [11] have summarized some possibilities
the "cooldown phase" of the used credits to perform the of Jailbreaks and demonstrated the feasibility of prompt
experiments and refine the prompts. In a practical scenario, injection attacks on ChatGPT. Other studies have already
this limitation would limit the number of commands an highlighted the potential of LLMs to increase the attack
attacker could execute on their victims: the payload on the vector of phishing attacks [6] due to LLMs’ capabilities of
victim’s machine sends messages to poll for new commands producing human-like content that can easily seem legitimate
from the C2 server, as well as to send results back to the C2 to users. Similarly, the previous study shows that ChatGPT
server. can easily prompt users to reveal confidential information or
to make users download malicious content. Greshake et al. [9]
4.4 Possible Mitigations mention a scenario where the attacker controls the content
This section presents potential mitigations to the security of a website to control what the LLM receives and the use
issue at hand. It is important to note that this list is non- of external API to communicate back to the attacker. In
exhaustive, and further research may uncover additional their approach, however, they show prompts on the attacker-
strategies. controlled web page and the communication seemingly only
malware to bypass initial security measures and then deploy the more stays within the LLM system and does not interact with the
harmful payload once inside the target system. user’s system.
Mika Beckerich, Laura Plein, and Sergio Coronado
6 CONCLUSION [12] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep
code comment generation. In Proceedings of the 26th conference
Large Language Models have opened new opportunities, im- on program comprehension. 200–210.
proving, and speeding up tasks in everyone’s daily lives. [13] LeaderbotX400. [n. d.]. LeaderbotX400/chatbot-experiments.
https://github.com/LeaderbotX400/chatbot-experiments/blob/
However, in this study, we have proven how easily unsecured main/jailbreaks/ChatGPT/Dan%2011.0.md
openly available LLMs and plugins can be misused to perform [14] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng,
efficient and undetected attacks around the world. Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023.
Jailbreaking ChatGPT via Prompt Engineering: An Empirical
This proof-of-concept demonstrates the potential transfor- Study. arXiv:2305.13860 [cs.SE]
mation of LLMs into proxies for malware attacks, allowing [15] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svy-
their misuse through plugins to establish connections with atkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin
Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning
command and control servers. This facilitates complete access benchmark dataset for code understanding and generation. arXiv
to a victim’s machine without necessitating direct interaction preprint arXiv:2102.04664 (2021).
[16] Attia Qammar, Hongmei Wang, Jianguo Ding, Abdenacer Naouri,
between the victim and the LLM. Mahmoud Daneshmand, and Huansheng Ning. 2023. Chatbots
This work highlights the need for new mitigation strategies to ChatGPT in a Cybersecurity Space: Evolution, Vulnerabili-
and the development of further security guidelines on the ties, Attacks, Challenges, and Future Recommendations. arXiv
preprint arXiv:2306.09255 (2023).
deployment of LLMs. [17] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar-
tinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière,
Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama:
ACKNOWLEDGEMENT Open and efficient foundation language models. arXiv preprint
arXiv:2302.13971 (2023).
Special thanks to Mercatus Center at George Mason Univer-
sity, for their invaluable support in this research. The views
presented in this paper do not represent official positions of
the Mercatus Center or George Mason University.
REFERENCES
[1] Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and
Charles Sutton. 2018. A survey of machine learning for big code
and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018),
1–37.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah,
Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav
Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language
models are few-shot learners. Advances in neural information
processing systems 33 (2020), 1877–1901.
[3] Miles Brundage, Katie Mayer, Tyna Eloundou, Sandhini Agarwal,
Steven Adler, Gretchen Krueger, Jan Leike, and Pamela Mishkin.
2022. Lessons learned on language model safety and misuse.
[4] Nghi DQ Bui, Hung Le, Yue Wang, Junnan Li, Akhilesh Deepak
Gotmare, and Steven CH Hoi. 2023. CodeTF: One-stop Trans-
former Library for State-of-the-art Code LLM. arXiv preprint
arXiv:2306.00029 (2023).
[5] PV Charan, Hrushikesh Chunduri, P Mohan Anand, and
Sandeep K Shukla. 2023. From Text to MITRE Techniques:
Exploring the Malicious Use of Large Language Models for Gener-
ating Cyber Attack Payloads. arXiv preprint arXiv:2305.15336
(2023).
[6] Minhaz Chowdhury, Nafiz Rifat, Shadman Latif, Mostofa Ahsan,
Md Saifur Rahman, and Rahul Gomes. 2023. ChatGPT: The
Curious Case of Attack Vectors’ Supply Chain Management Im-
provement. In 2023 IEEE International Conference on Electro
Information Technology (eIT). IEEE, 499–504.
[7] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang,
Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023.
Jailbreaker: Automated Jailbreak Across Multiple Large Language
Model Chatbots. arXiv preprint arXiv:2307.08715 (2023).
[8] Erik Derner and Kristina Batistič. 2023. Beyond the Safeguards:
Exploring the Security Risks of ChatGPT. arXiv preprint
arXiv:2305.08005 (2023).
[9] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph En-
dres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve
signed up for: Compromising Real-World LLM-Integrated Appli-
cations with Indirect Prompt Injection. arXiv:2302.12173 [cs.CR]
[10] Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017.
Program synthesis. Foundations and Trends® in Programming
Languages 4, 1-2 (2017), 1–119.
[11] Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker,
and Lopamudra Praharaj. 2023. From ChatGPT to ThreatGPT:
Impact of Generative AI in Cybersecurity and Privacy. IEEE
Access (2023).