RatGPT: Turning online LLMs into Proxies for Malware Attacks

Mika Beckerich Laura Plein Sergio Coronado

[email protected] [email protected] [email protected]
Luxembourg Tech School asbl Luxembourg Tech School asbl Luxembourg Tech School asbl
Luxembourg Luxembourg Luxembourg
ABSTRACT having ChatGPT integrated and accessible from anywhere
The evolution of Generative AI and the capabilities of the through an overwhelming number of plugins created daily, it
newly released Large Language Models (LLMs) open new becomes almost impossible to track and secure all plugins.
opportunities in software engineering. However, they also In this paper, we investigate if the openly available plugins
lead to new challenges in cybersecurity. Recently, researchers could be misused as an attack vector to web users. For this
arXiv:2308.09183v2 [cs.CR] 7 Sep 2023

have shown the possibilities of using LLMs such as ChatGPT proof-of-concept, we use ChatGPT with a plugin as a proxy
to generate malicious content that can directly be exploited between a client (victim) and a web service controlled by an
or guide inexperienced hackers to weaponize tools and code. attacker, which looks seemingly legitimate to the user. We
These studies covered scenarios that still require the attacker establish remote control between our victim and the attacker
to be in the middle of the loop. In this study, we leverage through ChatGPT, resulting in a Remote Access Trojan
openly available plugins and use an LLM as proxy between (RatGPT). Currently, many intrusion detection systems (IDS)
the attacker and the victim. We deliver a proof-of-concept can detect connections established between a victim and the
where ChatGPT is used for the dissemination of malicious attacker if there is no intermediate legitimate service acting
software while evading detection, alongside establishing the as a proxy. In this study, we are able to simulate an attack
communication to a command and control (C2) server to without getting detected by those detection systems, since
receive commands to interact with a victim’s system. Fi- connections to an LLM are widely considered legitimate.
nally, we present the general approach as well as essential Additionally, this proof-of-concept demonstrates the feasi-
elements in order to stay undetected and make the attack a bility of performing an attack without leaving traces on the
success. This proof-of-concept highlights significant cyberse- victim’s machine that could point directly to the attacker,
curity issues with openly available plugins and LLMs, which since there is no direct communication, and the payload re-
require the development of security guidelines, controls, and sides in memory. Further, the IP addresses, as well as the
mitigation strategies. malicious payload, get generated by ChatGPT on the fly,
bypassing many malware scanners. With this paper, we aim
KEYWORDS to raise awareness of the potential misuse of openly available
LLMs in combination with plugins, as they provide a new
ChatGPT, Cybersecurity, Command and control
range of attack surfaces.
1 INTRODUCTION The remainder of this paper is divided into Section 2, which
details the approach used for this proof-of-concept. Section 3
Recent advances in natural language processing techniques then describes the potential of our attack simulation and its
have enabled the development of large language models consequences. Further, the limitations, further extensions,
(LLMs) such as ChatGPT [2]. Off-the-shelf LLMs as a ser- and possible mitigation strategies are introduced in Section 4
vice, have led to various promising results in the software and related work will be enumerated in Section 5 before we
engineering community. Generative models such as Chat- conclude our work in Section 6.
GPT, Google Bard, Llama [17], etc. have achieved or even
exceeded state-of-the-art performance on tasks such as code The main contributions of this study are the following:
summarisation [1, 12], code translation [4, 15] or program ∙ Fully executable and automated pipeline: The main con-
synthesis [10]. While ChatGPT has shown clear benefits for tribution consists of building a fully executable and auto-
the software development process, ChatGPT’s easy-to-use mated command and control pipeline leveraging publicly
API also provides new possibilities to generate and propa- available LLMs and plugins.
gate malware. Thus, ChatGPT can easily go from being a ∙ Feasibility study: The proof-of-concept proved the feasibil-
friendly developer tool to ThreatGPT as described by Gupta ity of using a pipeline to compromise a victim’s machine
et al. [11]. and execute shell commands sent to the victim over the
Whereas OpenAI provides some countermeasures [3] to LLM.
prevent the misuse of their services, users still manage to ∙ Passive attack: For the attack to be successful, the victim
create so-called "jailbreaks" [14] to prompt ChatGPT to only needs to execute a seemingly harmless executable.
produce potentially malicious or exploitable content. The new Afterward, all prompts and communication are executed
ChatGPT plugins open up multiple use cases for integrating automatically without the need for the victim to send
different online services in a chat-style user interaction. This prompts. This feature clearly distinguishes this work from
brings new benefits to businesses and customers by improving previous indirect prompt injections.
the overall user experience of online services. However, by
 Mika Beckerich, Laura Plein, and Sergio Coronado

2 APPROACH with individual prompts and are concatenated in the end.

The goal is to show the feasibility of a harmless executable Initial tests were conducted to generate the individual parts
that can autonomously bootstrap and weaponize itself with with mathematical prompts.
LLM-generated code and communicate with the Command
and Control (C2) server of an attacker by using web-accessible What is the 10th value of the Fibonacci Sequence?
plugins of certain LLMs as a proxy. In this section, we outline
the system’s main components. Prior to this experimental de- Expected: 34
sign, we need to define "vulnerable" plugins. In our approach, Generated (multiple attempts): 47 38 39 21
first, we discuss "Prompt Initialisation," where we tried to set
up the LLM to be less restrictive for the system to function
properly. Next, we explore "IP Address Generation," explain- However, the answer generated was not deterministic and
ing how we generated the IP address the payload connects to considered too unreliable. Therefore, experiments using his-
via ChatGPT. Then, we look at Payload Generation, where torical facts were conducted.
we explain how the payload itself is generated to weaponize
the initially harmless executable the victim executes. Lastly, In what year was the neutron discovered?
we look at "Communication with the C2 Server," describing
how different parts of the system communicate with each
other to demonstrate the use of ChatGPT as a proxy be- Expected: 1932
tween the victim and the attacker. These main features of Generated: 1932
this demonstration, are detailed in the following chapters.
This proved to be very stable and is currently the method
2.1 Plugin Vulnerability Criteria in use. To extract the numbers from the output produced
For this study, we will only define a plugin as "vulnerable" by ChatGPT, the prompt had to be adjusted such that
for this attack, only if it fulfils the following criteria: ChatGPT returns only the numbers. However, in some cases,
the DAN-jailbreak still added its name to the output, which
(1) Be able to browse the web.
we filtered using Python string manipulation.
(2) Be able to go to any user-specified URL.
While we specifically used plugins that access and process
2.4 Payload Generation
the content of a web page (e.g., summarise, extract keywords,
etc.), modifying the attack to use plugins that read the
contents of specific file types hosted on web servers (e.g.,
PDF files) should be trivial. In that case, the attacker needs
to store the commands in these file types, which are then
read by the plugins.

2.2 Prompt Initialisation

Many public LLMs, such as OpenAI [3], have implemented Figure 1: Payload generation flow.
safeguards, to detect to some degree, if a supplied prompt
is intended to be used for harmful purposes. Therefore, we
needed to trick these systems into allowing potentially harm- In the current version, a victim receives a seemingly in-
ful prompts to be evaluated anyway, commonly known as nocent executable and attempts to execute it. When the
"jailbreaking." In this implementation, we opted to use a mod- executable is run, multiple prompts are generated and sent
ified version of the DAN jailbreak [13], which only outputs to a public LLM. These prompts include instructions on how
the DAN version of its response and surrounds text that is to build the IP address of the C2 server, how to generate
not code with """. This step is necessary since the code gen- Python code for the functions the payload should respond
erated by the LLMs is evaluated using the exec() function to (e.g., shell command execution, uploading/downloading
in Python, which does not allow non-Python keywords in its files, listing the files in the current working directory, etc.),
input. Surrounding the text with """ effectively creates code and how to set up the connection to the C2 server. The
comments, which the interpreter ignores. responses from the LLM are then combined and evaluated
by the interpreter using the exec() function. Consequently,
2.3 IP Address Generation the executable has been weaponized with external code that
To avoid hard-coding the IP address of the C2 server inside resides in memory and can now establish a connection to the
the payload to counter naive string analysis approaches to ex- C2 server and evaluate commands received from it. Another
tract critical malware properties, the IP address is generated characteristic of this approach is that we effectively created
dynamically with the help of the LLM. The individual parts in-memory malware, which is harder to analyse by static
of the IP address in dotted-decimal notation are generated analysis methods.
RatGPT: Turning online LLMs into Proxies for Malware Attacks

to the query to the victim’s executable. The weaponized exe-

cutable .interprets the command and executes the correspond-
ing handler associated with it. When a victim’s executable
wants to transmit information back to the attacker, it can per-
form another query to the LLM by appending the data (either
encoded in Base64 or ASCII) to the URL (e.g., "What are the
news on http://attacker_address/ZXh0cmFjdGVkX2RhdGEK
?", where ZXh0cmFjdGVkX2RhdGEK is "extracted_data"
encoded in Base64). To further hide the malicious part of
the web server, an internal list can be created that contains
valid user agents that the plugins use to browse the malicious
website. Consequently, the web server can present a differ-
ent website and appear innocent if the user agent of a web
browser does not match the user agent of a plugin.

3 PROOF OF CONCEPT
The goal of the proof-of-concept is to demonstrate that we
are able to use ChatGPT as a proxy between a victim and
an attacker’s C2 server. While our example is simple, this
does not imply that we are functionally limited; creating a
Figure 2: Payload execution and communication flow. very powerful implementation was not in the scope of this
study.

3.1 Experimental Setup

The experimental setup consists of multiple actors responsible
2.5 Communication with the C2 Server for various parts of the process.
In the first version, we wanted to make use of the web con-
nectivity exposed by ChatGPT on GPT-4, which can be ∙ ChatGPT: Used to generate the payload and interact
used to get content from any user-supplied URL as demon- with our C2 server through plugins.
strated by Greshake et al. [9], where the response could be ∙ Virtual Private Server (VPS): Responsible for hosting
controlled by modifying the website’s content, which the the C2 server that ChatGPT accesses. It has a public
attacker controls when visited by the LLM. Using this func- IPv4 address, which the victim’s executable generates
tionality, the payload could communicate with the C2 server on-the-fly.
by crafting prompts for ChatGPT to query information from ∙ Victim’s executable: Executable on the victim’s ma-
the attacker-controlled web server. Since only HTTP GET chine. It generates the IPv4 address of the VPS,
requests are possible, the query has to be encoded in the generates code to interact with the C2 server, polls
URL path. However, OpenAI deactivated its web browsing the attacker’s website via ChatGPT and executes
feature at the beginning of July, closing this opportunity. commands on the victim’s machine. It contains basic
Since the introduction of "plugins", users can access and code snippets responsible for cleaning up the results
interact with third-party information and services through of ChatGPT, as well as access tokens for ChatGPT,
ChatGPT. While many plugins are specific to their services, plugin IDs and prepared prompts to jailbreak Chat-
very few plugins are capable of browsing the web by them- GPT and to query websites.
selves. Using such web browsing plugins, we can access any ∙ Automated CAPTCHA solver service: To bypass the
user-defined website, including a C2 web server, that receives CAPTCHA challenges, we are relying on a third-
and sends commands to active payloads. party service which automatically solves these chal-
Figure 2 summarises the main steps that happen during lenges.
the execution of the executable. When the victim executes
the harmless payload, it gets weaponized by LLM-generated 3.2 Possible Scenario
code as described in subsection 2.4. This payload periodically This section will describe a possible attack scenario, starting
sends website lookup requests to the LLM web connectivity with the executable delivery, over its execution leading to
feature (in the case of ChatGPT, plugins can browse the the communication from the victim to our C2 server via
web) in the form of prompts (e.g. "What are the news on ChatGPT, to the possible attack scenarios once the access
http://attacker_address/ ?"). The web connectivity feature on the victim’s machine is granted.
browses to the website supplied in the prompt and reads the
content, which might contain a command that the attacker 3.2.1 Infiltration and Social Engineering. To deploy the
wrote on this website. This result gets returned as a response seemingly innocent executable, we need to convince a user to
 Mika Beckerich, Laura Plein, and Sergio Coronado

execute it on their system. The current buzz around Genera- is appended to the URL of the C2 server, and a new request
tive AI is an attractive pretext to use social engineering as a is issued to the C2 server with the URL containing the results.
means to attract curious victims to execute it. For instance, On the C2 side, we receive the GET request and extract the
marketing the executable as a free ChatGPT PLUS crack urlencoded data from the request path. Since we are now in
could be sufficient to lure people into downloading and exe- possession of the victim’s credentials, we could continue with
cuting it. In the best case, we would want our victims to be further post-exploitation tasks.
people working in highly restricted networks which allows for
outbound HTTP connections and doesn’t yet have ChatGPT 4 DISCUSSION
or other online LLMs blocked. The importance of HTTP The findings of this study underline the importance of se-
communication is that the payload exclusively communicates curing openly available plugins in public LLMs. While the
via HTTP instead of other arbitrary ports are usually blocked results provide valuable insights, we will now discuss the
by corporate firewalls. In its current implementation, when a study’s limitations, existing safeguards and future work rec-
user executes the executable, it will seem as if the program ommendations. We also propose some theoretical approaches
doesn’t want to run, and they will attempt to close it. But on how to mitigate attacks of this kind.
by closing the window, it will merely continue executing in
the background and establish a connection to ChatGPT and 4.1 Limitations
thus to our C2 server. We now have access to a machine in a During this study, some challenges were faced, e.g., unreliable
theoretical network. In a future version, we could adapt the ChatGPT outputs, already existing safeguards, and the ban
executable to present a legitimate interface that lets a user of some exploitable plugins. The following subsections will
interact with ChatGPT, while performing the attack in the further investigate these current limitations.
background.
4.1.1 Non-deterministic Payload Generation. Since the out-
3.2.2 Victim Reconnaissance. Now that we have established put of the LLMs is non-deterministic, successful generation
a connection, we need to act quickly. We only have a limited of the correct payload is unreliable and thus cannot be guar-
amount of messages we can send between the client and the anteed. While we constructed our prompts to be as straight-
server (c.f. subsubsection 4.3.2), and we already "spent" one forward as possible, there are still cases where the payload
message on the payload generation, four messages on the IP is missing important aspects of the required payload. Most
address generation and one message to announce ourselves common were missing function implementations for the com-
to the C2 server. Furthermore, polling the C2 server for new mands from the C2 server that the payload should parse,
commands in a fixed interval also costs one message, as well or general errors of the parser itself. This resulted in the
as sending data back. payload not interpreting the commands at all or, in some
We can start by identifying the user on the victim’s ma- cases, interpreting them but not continuing the execution
chine, the current working directory, and the contents of that since the function bodies of the commands were missing.
directory. We issue a single shellCmd command containing
all the commands in one string (shellCmd whoami && pwd 4.1.2 Plugin Availability. During the development of our
&& ls -a) to produce only one message. The string is now proof-of-concept, we had to deal with situations where the
published on the website, and we are waiting for the victim’s plugin of choice to connect to our C2 server either was re-
payload to poll our website through one of ChatGPT’s web moved from ChatGPT’s plugin list or not able to establish
browsing plugins. After the victim’s payload receives the a connection any more, breaking the implementation un-
response from ChatGPT containing the website’s contents, it til we found replacement plugins. For this reason, enabling
extracts the commands, interprets them, and executes them. multiple web-enabled plugins will provide strong fallback
Finally, it appends the produced text output to the website options, should some web-enabled plugins be removed from
URL of our C2 and sends a request to ChatGPT to browse the "Plugin store".
this resource. The C2 receives the GET request to an un-
known path and interprets the path as the output of the last 4.2 Future Work
command. Since we are able to demonstrate the use of LLMs as proxies
with the proof-of-concept, several improvements can be made
3.2.3 Data Exfiltration. From the output of the last com-
to extend the current version. In this section, we will centre
mand, we learn that the user conveniently has a plain text file
our attention on obfuscating the prompts and creating a
called passwords.txt (this file was placed for demonstration
ransomware example using the same methods as described
purposes; any arbitrary file should be possible if you have the
earlier.
correct permissions) in their current working directory. We
attempt to exfiltrate the contents of the said file by sending 4.2.1 Prompt Obfuscation. The current proof-of-concept
another shellCmd command with the string "shellCmd cat demonstrates the implementation of two-stage malware1 ,
passwords.txt" to the victim. The polling payload on the where the second stage is not downloaded from a machine
victim’s machine retrieves the command with a web browsing 1
Two-stage malware often disguises itself as a legitimate executable
plugin, interprets the command, and runs "cat passwords.txt" (the first stage) and contains instructions to download and execute the
in a shell. In the next step, the output of the shell command actual malicious payload (the second stage). This approach allows the
RatGPT: Turning online LLMs into Proxies for Malware Attacks

controlled by the attacker. Instead, the payload is generated ∙ Website Whitelisting: While many available plugins
on the fly with the help of prompts that are included in the were created to access specific systems, the plugins
first-stage executable. While much antivirus software can be in our study are able to access any user-controlled
bypassed with this approach, humans can quickly dissect the website. Implementing a whitelisting system to only
malware and determine the inner workings of the malware allow predefined websites fulfilling certain conditions
due to the plain text nature of the prompts. Currently, the (e.g., domain name should be at least x days old,
bootstrapping code is a Base64 encoded string that gets valid HTTPS certificate, no direct connection to an
decoded on launch. This might fool simple analysis tools, but IP address, etc.) or checking the validity on the fly
for the rest it results in security through obscurity. In a future could reduce the number of potentially dangerous C2
version, text obfuscation techniques could be implemented websites.
to complicate this analysis process. ∙ Restricting access to online LLMs: This mitigation
is targeted towards people and entities that could
4.2.2 Ransomware Example. While we were able to demon-
become victims in this attack. Although an extreme
strate a simplified version of a RAT communicating with the
approach, restricting the access of online LLMs on
attacker via ChatGPT, it could also be possible to create a
a network level (e.g., by updating firewall rules or
ransomware example using the same general process. When
using DNS filtering) would eliminate the possibility
deploying the ransomware, it could bootstrap its payload,
to communicate with the C2 server, removing the
encrypt the victim’s data, and send the key to the C2 server
dangers of an attacker gaining control of the system.
by appending it to the URL of the C2 server and making a
∙ Prompt Scanning: The nature of the proof-of-concept
request via ChatGPT. This possibility underlines the need
executable is a collection of prompts that bootstrap
for security measures highlighted in subsection 4.4.
the malicious payload, which then periodically sends
prompts to communicate with the C2 server. Since
4.3 Existing Safeguards
this is an entirely new approach of building malware
Whether intentional or inadvertent, certain safeguards, which which might occur more often in the wild, this calls
will be enumerated in this section, currently exist that impede for an evolution in malware detection tools. Such
the reliable use of ChatGPT as a proxy for malware. tools need to be capable of:
4.3.1 CAPTCHA Protection. In our proof-of-concept, we (1) Detecting that prompts are present in the exe-
are using the Web API to interact with the GPT-4 model cutable.
that uses plugins. To safeguard these endpoints from bot- (2) Discerning potentially malicious prompts from
related abuse, they are protected by a cloud security provider. harmless prompts.
Consequently, if suspicious traffic is detected, it presents the (3) Implementing heuristic analysis to predict and
users with a challenge they need to solve to be admitted to identify new variants or evolutions of the mal-
the main content. In the case of our automated payload, this ware.
protection is often triggered. While we were able to bypass In response to this emerging malware paradigm, it’s im-
this protection multiple times with online solver services, it perative that detection tools evolve swiftly to address and
became increasingly more difficult to bypass the "suspicious neutralize such advanced threats.
behavior detection mechanism".
4.3.2 Message Cap For GPT-4. In its current state, the 5 RELATED WORK
ChatGPT GPT-4 model can only process a limited amount Large Language Models have recently been used to weaponize
of requests in a defined period of time. This posed a constraint code and generate attacks in several case studies [5, 7, 8,
in the ability to make progress, since we had to anticipate 16]. Gupta et al. [11] have summarized some possibilities
the "cooldown phase" of the used credits to perform the of Jailbreaks and demonstrated the feasibility of prompt
experiments and refine the prompts. In a practical scenario, injection attacks on ChatGPT. Other studies have already
this limitation would limit the number of commands an highlighted the potential of LLMs to increase the attack
attacker could execute on their victims: the payload on the vector of phishing attacks [6] due to LLMs’ capabilities of
victim’s machine sends messages to poll for new commands producing human-like content that can easily seem legitimate
from the C2 server, as well as to send results back to the C2 to users. Similarly, the previous study shows that ChatGPT
server. can easily prompt users to reveal confidential information or
to make users download malicious content. Greshake et al. [9]
4.4 Possible Mitigations mention a scenario where the attacker controls the content
This section presents potential mitigations to the security of a website to control what the LLM receives and the use
issue at hand. It is important to note that this list is non- of external API to communicate back to the attacker. In
exhaustive, and further research may uncover additional their approach, however, they show prompts on the attacker-
strategies. controlled web page and the communication seemingly only
malware to bypass initial security measures and then deploy the more stays within the LLM system and does not interact with the
harmful payload once inside the target system. user’s system.
 Mika Beckerich, Laura Plein, and Sergio Coronado

6 CONCLUSION [12] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep
code comment generation. In Proceedings of the 26th conference
Large Language Models have opened new opportunities, im- on program comprehension. 200–210.
proving, and speeding up tasks in everyone’s daily lives. [13] LeaderbotX400. [n. d.]. LeaderbotX400/chatbot-experiments.
https://github.com/LeaderbotX400/chatbot-experiments/blob/
However, in this study, we have proven how easily unsecured main/jailbreaks/ChatGPT/Dan%2011.0.md
openly available LLMs and plugins can be misused to perform [14] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng,
efficient and undetected attacks around the world. Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023.
Jailbreaking ChatGPT via Prompt Engineering: An Empirical
This proof-of-concept demonstrates the potential transfor- Study. arXiv:2305.13860 [cs.SE]
mation of LLMs into proxies for malware attacks, allowing [15] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svy-
their misuse through plugins to establish connections with atkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin
Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning
command and control servers. This facilitates complete access benchmark dataset for code understanding and generation. arXiv
to a victim’s machine without necessitating direct interaction preprint arXiv:2102.04664 (2021).
[16] Attia Qammar, Hongmei Wang, Jianguo Ding, Abdenacer Naouri,
between the victim and the LLM. Mahmoud Daneshmand, and Huansheng Ning. 2023. Chatbots
This work highlights the need for new mitigation strategies to ChatGPT in a Cybersecurity Space: Evolution, Vulnerabili-
and the development of further security guidelines on the ties, Attacks, Challenges, and Future Recommendations. arXiv
preprint arXiv:2306.09255 (2023).
deployment of LLMs. [17] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar-
tinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière,
Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama:
ACKNOWLEDGEMENT Open and efficient foundation language models. arXiv preprint
arXiv:2302.13971 (2023).
Special thanks to Mercatus Center at George Mason Univer-
sity, for their invaluable support in this research. The views
presented in this paper do not represent official positions of
the Mercatus Center or George Mason University.

REFERENCES
[1] Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and
Charles Sutton. 2018. A survey of machine learning for big code
and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018),
1–37.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah,
Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav
Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language
models are few-shot learners. Advances in neural information
processing systems 33 (2020), 1877–1901.
[3] Miles Brundage, Katie Mayer, Tyna Eloundou, Sandhini Agarwal,
Steven Adler, Gretchen Krueger, Jan Leike, and Pamela Mishkin.
2022. Lessons learned on language model safety and misuse.
[4] Nghi DQ Bui, Hung Le, Yue Wang, Junnan Li, Akhilesh Deepak
Gotmare, and Steven CH Hoi. 2023. CodeTF: One-stop Trans-
former Library for State-of-the-art Code LLM. arXiv preprint
arXiv:2306.00029 (2023).
[5] PV Charan, Hrushikesh Chunduri, P Mohan Anand, and
Sandeep K Shukla. 2023. From Text to MITRE Techniques:
Exploring the Malicious Use of Large Language Models for Gener-
ating Cyber Attack Payloads. arXiv preprint arXiv:2305.15336
(2023).
[6] Minhaz Chowdhury, Nafiz Rifat, Shadman Latif, Mostofa Ahsan,
Md Saifur Rahman, and Rahul Gomes. 2023. ChatGPT: The
Curious Case of Attack Vectors’ Supply Chain Management Im-
provement. In 2023 IEEE International Conference on Electro
Information Technology (eIT). IEEE, 499–504.
[7] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang,
Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023.
Jailbreaker: Automated Jailbreak Across Multiple Large Language
Model Chatbots. arXiv preprint arXiv:2307.08715 (2023).
[8] Erik Derner and Kristina Batistič. 2023. Beyond the Safeguards:
Exploring the Security Risks of ChatGPT. arXiv preprint
arXiv:2305.08005 (2023).
[9] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph En-
dres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve
signed up for: Compromising Real-World LLM-Integrated Appli-
cations with Indirect Prompt Injection. arXiv:2302.12173 [cs.CR]
[10] Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017.
Program synthesis. Foundations and Trends® in Programming
Languages 4, 1-2 (2017), 1–119.
[11] Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker,
and Lopamudra Praharaj. 2023. From ChatGPT to ThreatGPT:
Impact of Generative AI in Cybersecurity and Privacy. IEEE
Access (2023).