A Survey on the Memory Mechanism of Large Language Model based Agents

In the field of LLMs, Zhao et al. [70] present the first comprehensive survey to summarize the background, evolution paths, model architectures, training methodologies, and evaluation strategies of LLMs.  Hadi et al. [71] and Min et al. [72] also conduct LLM surveys from the holistic view, which, however, provide different taxonomies and understandings on LLMs. Following these surveys, people dive into specific aspects of LLMs and review the corresponding milestone studies and key technologies. These aspects can be classified into four categories including the fundamental problems, evaluation, applications, and challenges of LLMs.

Fundamental problems. The surveys in this category aim to summarize techniques that can be leveraged to tackle fundamental problems of LLMs. Specifically, Zhang et al. [8] provide a comprehensive survey on the methods of supervised fine-tuning, which is a key technique for better training LLMs. Shen et al. [9], Wang et al. [10] and Liu et al. [11] present surveys on the alignment of LLMs, which is a key requirement for LLMs to produce outputs consistent with human values. Gao et al. [12] propose a survey on the retrieval-augmented generation (RAG) capability of LLMs, which is key to providing LLMs with factual and up-to-date knowledge and removing hallucinations. Qin et al. [18] summarize the state-of-the-art methods on enabling LLMs to leverage external tools, which is fundamental for LLMs to expand their capability in domains that require specialized knowledge. Wang et al. [13], Yao et al. [14], Wang et al. [15], Feng et al. [16] and Zhang et al. [17] present surveys on the direction of LLM knowledge editing, which is important for customizing LLMs to satisfy specific requirements. Huang et al. [19], Wang et al. [20] and Pawar et al. [21] focus on long-context capabilities of LLMs, which is critical for LLMs to process more information at each time and enhance their application scenarios. Wu et al. [22], Song et al. [23], Caffagni et al. [24] and Yin et al. [25] summarize multi-modal LLMs, which expands the capability of LLMs from text to visual and other modalities. The above surveys mainly focus on the effectiveness of LLMs. Another important aspect of LLMs is their training and inference efficiency. To summarize studies on this aspect,  Zhu et al. [30], Xu and McAuley [31], Wang et al. [32] and  Park et al. [33] systematically review the techniques of model compression.  Ding et al. [81] and Xu et al. [29] analyze and conclude the studies on parameter efficient fine-tuning.  Bai et al. [26], Wan et al. [27], Miao et al. [28] and  Ding et al. [81] put more focuses on the efficiency of resource utilization in a general sense.

Evaluation. The surveys in this category focus on how to evaluate the capability of LLMs. Specifically, Chang et al. [34] comprehensively summarize the evaluation methods from an overall perspective. It encompasses different evaluation tasks, methods, and benchmarks, which serve as critical parts in assessing LLM performances. Guo et al. [35] care more about the evaluation targets and describe how to evaluate the knowledge, alignment, and safety control capabilities of LLMs, which supplement evaluation metrics beyond performance.

Applications. The surveys in this category aim to summarize models that leverage LLMs to improve different applications. More concretely, Zhu et al. [37] focus on the field of information retrieval (IR) and summarize studies on LLM-based query processes. Xu et al. [38] pay more attention to information extraction (IE) and provide comprehensive taxonomies for LLM-based models in this field. Li et al. [50], Lin et al. [51] and Wang et al. [52] discuss the applications of LLMs in the field of recommender system, where they utilize agents to generate data and provide recommendations. Fan et al. [39], Wang et al. [40], and Zheng et al. [41] concentrate on how LLMs can benefit software engineering (SE) in terms of software design, development, and testing. Zeng et al. [42] summarize LLM-based methods in the field of robotics. Cui et al. [43] and Yang et al. [44] focus on the application of autonomous driving and summarize models in this domain based on LLMs from different perspectives. Beyond the above domains in artificial intelligence, LLMs have also been used in natural and social science. He et al. [45], Zhou et al. [46] and Wang et al. [47] summarize the applications of LLMs in medicine. Li et al. [48] focus on the applications of LLMs in finance. He et al. [49] review the models on leveraging LLMs to improve the development of psychology.

Challenges. The surveys in this category focus on trustworthiness in LLMs, such as hallucination, bias, unfairness, explainability, security, and privacy. Hallucination in LLMs refers to the problem that LLMs may generate misconceptions or fabrications, impacting their reliability for downstream applications. Zhang et al. [53], Huang et al. [54], Rawte et al. [55], Ye et al. [56], Ji et al. [57], Tonmoy et al. [58] and Jiang et al. [59] summarize the mainstream models for alleviating the hallucination problem in LLMs. The bias and unfairness problems refer to the phenomenon that LLMs may unequally treat different humans or objectives, which can lead to the propagation of societal stereotypes and discrimination. Gallegos et al. [60], Kotek et al. [61] and Li et al. [62] comprehensively discuss these challenges and summarize existing methods for alleviating them. The problem of explainability means that the internal working mechanisms of LLMs are still unclear. Zhao et al. [63] systematically discuss this problem and summarize previous efforts on improving the explainability of LLMs. Security and privacy are also challenging problems, which have been comprehensively surveyed in Yao et al. [64], Shayegani et al. [65], Neel and Chang [66], Smith et al. [67], Dong et al. [68] and Das et al. [69].

Based on the capability of LLMs, people have conducted a lot of studies on building LLM-based agents, which can autonomously perceive environments, take actions, accumulate knowledge, and evolve themselves. In this field, Wang et al. [3] present the first survey paper to systematically summarize LLM-based agents from the perspectives of agent construction, agent application, and agent evaluation.  Xi et al. [4], Zhao et al. [77], Cheng et al. [78] and Ge et al. [80] also summarize LLM-based agent studies from the overall perspective, but they have different focuses and taxonomies, delivering more diverse understandings on this field. In addition to these overall surveys, there have also emerged several papers reviewing specific aspects of LLM-based agents. For the fundamental problems, Durante et al. [79] summarize studies on multi-modal agents. Huang et al. [74] focus on the planning capability of LLM-based agents. Guo et al. [75] pay more attention to the scenarios of multi-agent interactions. For the applications, Li et al. [76] provide a summarization on LLM-based agents that are leveraged as personal assistants.

Position of this work. Our survey summarizes the studies on a fundamental problem of LLM-based agents, that is, the agent’s memory mechanism. To our knowledge, this is the first survey in this direction. We hope it can not only inspire more advanced memory architectures in the future, but also provide newcomers with comprehensive starting materials.

For clear presentations, we first introduce several important background knowledge as follows:

In the above definitions, task and environment are the most coarse-grained concepts, while step is the most fine-grained one. They together describe the complete agent-environment interaction process.

A Toy Example. Based on the above definitions, we show a toy example and use it throughout the paper to provide more intuitions. In this example, the agent has two tasks: (A) Making a trip plan for Alice in Beijing from 5/1/2024 to 5/3/2024, where the agent has to (i) buy a round-trip flight ticket, (ii) determine several attractions according to Alice’s preferences, and (iii) arrange the visiting order of these attractions. (B) Recommending a movie for Alice on 5/10/2024, where the agent has to (i) determine the time that Alice is free to watch movies, and (ii) select a movie according to Alice’s watching preference. For each task, we present a potential trial in the following (also illustrated in Figure 3(a)). For task (A), the trial is:

[Step 1] Agent: I need to book flight tickets from Shanghai to Beijing on 5/1/2024, and from Beijing to Shanghai on 5/3/2024 through the online ticket office. Environment (Online Ticket Office): The order has been confirmed. The departure flight takes off at 9:00 AM, and your ticket number is EK240501SX9021. The returning flight takes off at 4:30 PM, and your ticket number is LG240503FN6317. Please arrive at the airport on time for your flight.

[Step 2] Agent: I have acquired external knowledge from the magazine Attractions in Beijing. It shows that the Great Wall, the Forbidden City and the Summer Palace are famous. According to Alice’s preferences, I think she prefers ancient architecture, so I determine the Forbidden City and the Summer Palace as the destinations. Environment (Alice): Well done!

[Step 3] Agent: Now, I need to arrange the visiting order for Alice. According to the information above, I recommend visiting the Summer Palace after she arrives in Beijing because it is suitable to spend a whole afternoon there and watch the sunset. Then, she can go to the Forbidden City the next day, which would give her plenty of time. Finally, she can return to Shanghai on the third day and take a rest. Environment (Alice): I love this plan. Thanks!

For task (B), the trial is:

[Step 1] Agent: According to my previous experiences of movie recommendations for Alice, she prefers to start watching movies at 9:00 PM, because she has finished work by then. I should ask Alice if she is available at that time. Environment (Alice): Of course, it is a nice time!

[Step 2] Agent: Now, I need to find some movies that Alice has marked "want to watch" on Netflix. Environment (Netflix): Alice’s want to watch: Interstellar, Planet Terror and Candle in the Tomb.

[Step 3] Agent: Since Alice never watches horror movies at night, Interstellar would be more suitable for her. Now, I need to confirm with Alice about the movie. Environment (Alice): Great! I like it!

In a narrow sense, the memory of the agent is only relevant to the historical information within the same trial. Formally, for a given task, the historical information of the trial before step $t$ is $\xi_{t}=\{a_{1},o_{1},a_{2},o_{2},...,a_{t-1},o_{t-1}\}$, and then the memory is derived based on $\xi_{t}$. In the above toy example, for task (A), the agent at [step 3] needs to arrange the visiting order for Alice; at this time, its memory contains the information about the selected attractions and arrival time in [step 1] and [step 2]. For task (B), the agent has to choose a movie for Alice at [step 3]; at this time, its memory contains the arranged time to watch films.

In a broad sense, the memory of the agent can come from much wider sources, for example, the information across different trials and the external knowledge beyond the agent-environment interactions. Formally, given a series of sequential tasks $\{\mathcal{T}_{1},~{}\mathcal{T}_{2},...,~{}\mathcal{T}_{K}\}$, for task $\mathcal{T}_{k}$, the memory information at step $t$ comes from three sources: (1) the historical information within the same trial, that is, $\xi_{t}^{k}=\{a_{1}^{k},o_{1}^{k},...,a_{t-1}^{k},o_{t-1}^{k}\}$, where we add superscript $k$ to label the task index. (2) The historical information across different trials, that is, $\Xi^{k}=\{\xi^{1},\xi^{2},...,\xi^{k-1},\xi^{k^{\prime}}\}$, where $\xi^{j}$ $(j\in\{1,...,k-1\})$ represents the trials of task $j$¹¹1For each task, there can be multiple trials for exploring the final solution, and all of them can be incorporated into the memory., and $\xi^{k^{\prime}}$ denotes the previously explored trials for task $\mathcal{T}_{k}$. (3) External knowledge, which is represented by $D_{t}^{k}$. The memory of the agent is derived based on $(\xi_{t}^{k},~{}\Xi^{k},~{}D_{t}^{k})$. In the above toy example, for task (A), if there are several failed trials, that is, the feedback from Alice is negative, then these trials can be incorporated into the agent’s memory to avoid future similar errors (corresponding to $\xi^{k^{\prime}}$). In addition, for task (B), the agent may recommend movies relevant to the attractions that Alice has visited in task (A) to capture her recent preferences (corresponding to $\{\xi^{1},\xi^{2},...,\xi^{k-1}\}$). In the agent decision process, it has also referred to the magazine Attractions in Beijing for making trip plans, which is the external knowledge (corresponding to $D_{t}^{k}$) for the current task $\mathcal{T}_{k}$.

As mentioned at the beginning of Section 3, there are three key phases in the agent-environment interaction process. The agent memory module implements these phases through three operations including memory writing, memory management, and memory reading.

Memory Writing. This operation aims to project the raw observations into the actually stored memory contents, which are more informative [7] and concise [6]. It corresponds to the first phase of the agent-environment interaction process. Given a task $\mathcal{T}_{k}$, if the agent takes an action $a^{k}_{t}$ at step $t$, and the environment provides an observation $o^{k}_{t}$, then the memory writing operation can be formally represented as:

where $W$ is a projecting function. $m_{t}^{k}$ is the finally stored memory contents, which can be either natural languages or parametric representations. In the above toy example, for task (A), the agent is supposed to remember the flight arrangement and the decision of attractions after [step 2]. For task (B), the agent should memorize the fact that Alice hopes to watch movies at 9:00 PM, after [step 1].

Memory Management. This operation aims to process the stored memory information to make it more effective, for example, summarizing high-level concepts to make the agent more generalizable [6], merging similar information to reduce redundancy [7], and forgetting unimportant or irrelevant information to remove its negative influence. This operation corresponds to the second phase of the agent-environment interaction process. Let $M_{t-1}^{k}$ be the memory contents for task $k$ before step $t$, and suppose $m_{t}^{k}$ is the stored information at step $t$ based on the above memory writing operation, then, the memory management operation can be represented by:

where $P$ is a function that iteratively processes the stored memory information. For the narrow memory definition, the iteration only happens within the same trial, and the memory is emptied when the trial is ended. For the broad memory definition, the iteration happens across different trials or even tasks, as well as the integrations of external knowledge. For task (B) in the above toy example, the agent can conclude that Alice enjoys watching science fiction movies in the evening, which can be used as a default rule to make recommendations for Alice in the future.

Memory Reading. This operation aims to obtain important information from the memory to support the next agent action. It corresponds to the third phase of the agent-environment interaction process. Suppose $M_{t}^{k}$ is the memory contents for task $k$ at step $t$, $c_{t}^{k}$ is the context of the next action, then the memory reading operation can be represented by:

where $R$ is usually implemented by computing the similarity between $M_{t}^{k}$ and $c_{t+1}^{k}$ [82]. $\hat{M}_{t}^{k}$ is used as parts of the final prompt to drive the agent’s next action. For task (B) in the above toy example, when the agent decides on the final recommended movie in [Step 3], it should focus on the “want to watch” list in [Step 2] and select one from it.

Based on the above operations, we can derive a unified function for the evolving process from $\{a^{k}_{t},o^{k}_{t}\}$ to $a^{k}_{t+1}$, that is:

where LLM is the large language model. The complete agent-environment interaction process can be easily obtained by iteratively expanding this function (see Figure 3(b) for an intuitive illustration).

Cognitive psychology is the scientific study of human mental processes such as attention, language use, memory, perception, problem-solving, creativity, and reasoning²²2https://en.wikipedia.org/wiki/Cognitive_psychology. Among these processes, memory is widely recognized as an extremely important one [84]. It is fundamental for humans to learn knowledge by accumulating important information and abstracting high-level concepts [85], form social norms by remembering cultural values and individual experiences [86], take reasonable behaviors by imagining the potential positive and negative consequences [87], and among others.

A major goal of LLM-based agents is to replace humans for accomplishing different tasks. To make agents behave like humans, following human’s working mechanisms to design the agents is a natural and essential choice [88]. Since memory is important for humans, designing memory modules is also significant for the agents. In addition, cognitive psychology has been studied for a long time, so many effective human memory theories and architectures have been accumulated, which can support more advanced capabilities of the agents [89].

To accomplish different practical tasks, agents have to self-evolve in dynamic environments [90]. In the agent-environment interaction process, the memory is key to the following aspects: (1) Experience accumulation. An important function of the memory is to remember past error plannings, inappropriate behaviors, or failed experiences, so as to make the agent more effective for handling similar tasks in the future [91]. This is extremely important for enhancing the learning efficiency of the agent in the self-evolving process. (2) Environment exploration. To autonomously evolve in the environment, the agents have to explore different actions and learn from the feedback [92]. By remembering historical information, the memory can help to better decide when and how to make explorations, for example, focusing more on previously failed trials or actions with lower exploring frequencies [93]. (3) Knowledge abstraction. Another important function of the memory is to summarize and abstract high-level information from raw observations, which is the basis for the agent to be more adaptive and generalizable to unseen environments [82]. In summary, self-evolution is the basic characteristic of LLM-based agents, and memory is of key importance to self-evolution.

In many applications, memory is an indispensable component of the agent. For example, in a conversational agent, the memory stores information about historical conversations, which is necessary for the agent to generate the next response. Without memory, the agent does not know the context, and cannot continue the conversation [94]. In a simulation agent, memory is of great importance to make the agent consistently follow the role profiles. Without memory, the agent may easily step out of the role during the simulation process [95]. Both of the above examples show that the memory is not an optional component, but is necessary for the agents to accomplish given tasks.

In the above three perspectives, the first one reveals that the memory builds the cognitive basis of the agent. The second and third ones show that the memory is necessary for the agent’s evolving principles and applications, which provide insights for designing agents with memory mechanisms.

In previous works, the memory contents may come from different sources. Based on our formulation in Section 3, these sources can be classified into three categories, that is, the information inside a trial, the information across different trials, and the external knowledge. The former two are dynamically generated in the agent-environment interaction process (e.g., task internal information), while the latter is static information outside the loop (e.g., task external information). We summarize previous works on memory sources in Table 1.

In general, there are two forms to represent the memory contents: textual form and parametric form. In textual form, the information is explicitly retained and recalled by natural languages. In parametric form, the memory information is encoded into parameters and implicitly influences the agent’s actions. We summarize previous works on memory forms with their implementations in Table 2.

We separate the entire procedure of memory into three operations: memory writing, memory management, and memory reading. These three typically collaborate to achieve memory function, providing information for LLM inference. We summarize previous works on memory operations in Table 3.

This type of approaches regards the memory of the agents as a stand-alone component and evaluates its effectiveness independently. Previous studies can be categorized into two classes: subjective evaluation and objective evaluation. The subjective evaluation aims to measure memory effectiveness based on human judgments, which can be widely used in the scenarios that lack objective ground truths. Objective evaluation assesses memory effectiveness based on numerical metrics, which makes it easy to compare different memory modules.

Besides the above method that directly evaluates the memory module, evaluating via task completion is also a popular evaluation strategy. The intuition behind this type of approaches is that if the agent can successfully complete a task that highly depends on memory, it suggests that the designed memory module is effective. In the following parts, we present several representative tasks that are leveraged to evaluate the memory module in indirect ways.

Compared with direct evaluation, indirect evaluation via specific tasks can be easier to conduct, since there are already many public benchmarks. However, the performance on tasks can be attributed to various factors, and memory is only one of them, which may make the evaluation results biased. By direct evaluation, the effectiveness of the memory module can be independently evaluated, which improves the reliability of the evaluation results. However, to our knowledge, there are no open-sourced benchmarks tailored for the memory modules in LLM-based agents.

Role-playing represents a classic application of LLM-based agents, where memory plays a crucial role inside the agents. It endows roles with distinct characteristics, differentiating them from one another. Many previous studies have explored methods for constructing role memories [105, 143, 145, 146, 147]. Shao et al. [105] construct the memory of roles by experience uploading, which utilizes SFT to inject memory into model parameters. Li et al. [143] enhance large language models for role-playing via an improved prompt and the character memory extracted from scripts, where user queries and chatbot’s responses are concatenated to form a sequence as memory. Wang et al. [145] infuse role-specific knowledge and episode memories into LLM-based agents, where context QA pairs are concatenated to form episode memory. Zhao et al. [146] aim to generate human-like responses, guided by personality traits extracted from narratives, which can be stored and retrieved by relevance and importance. Zhou et al. [147] generate character-based dialogues for different roles and empower LLM-based agents with corresponding styles by SFT.

Social simulation is basically an extension of role-playing, which focuses more on multi-agent modeling. The memory module is an important component for such applications, which helps to accurately simulate human dynamic behaviors. In previous studies, Kaiya et al. [148] propose a Summarize-and-Forget memory mechanism for better self-monitoring in social scenarios. Gao et al. [2] focus on social network simulation systems. Each agent in the system has a memory pool, which consists of diverse user messages from online platforms to identify the user. Li et al. [163] maintain conversation contexts, encompassing the economic environment and agent decisions from previous months, in order to simulate the impact of broad macroeconomic trends on agents’ decision-making and to make the agents grasp market dynamics. Li et al. [109] simulate the job-seeking scenario in human society, where the memory of agents includes profiles and goals initially and is further enriched with other information, like dialogues and personal reflections. Hua et al. [150] simulate the decisions and consequences of the participating countries in the wars, where the conversations of agents are continuously maintained into memory.

There are several insights in designing an agent’s memory for role-play and social simulation. First, the memory should be consistent with the roles’ characteristics, which can be used to identify each role and distinguish it from the others. This is crucial for improving the realism of role-play and the diversity of social simulation. Second, the memory should appropriately influence the subsequent actions of the agent to ensure the consistency and rationality of its behaviors. Additionally, for humanoid agents, their memory mechanisms should align with the features of human memory, such as forgetting and long/short-term memory, which should refer to the theories of cognitive psychology.

LLM-based agents are well-suited for creating personal assistants, such as agents capable of engaging in long-term conversations with users [94, 101, 153], as well as those tasked with automatically seeking information [164]. These agents often need to memorize previous dialogues to maintain the consistency, and remember critical styles and events to generate more personalized and relevant responses. Lu et al. [94] maintain the context consistency for dialogues by saving contents and information of conversations, which helps to find proper relevant information by retrieval. Lee et al. [101] summarize conversations to extract important information, store it, and retrieve it for future inference. Pan et al. [164] focus on information-seeking tasks, which design memory modules to store user’s context information, and empower external knowledge with tool usage. Wu et al. [153] retain important context as memory to maintain conversation consistency.

In summary, most memory implementations for personal assistants adopt retrieval methods in textual form, because they are better at finding relevant information from pieces of conversations. For the memory storage, the agent should remember the factual information during user-agent interactions, as well as the personal style of users, in order to generate responses that are tailored to the user’s situation. Additionally, when recalling memories, the agent should identify and retrieve the memory that is relevant to the current query and context. This principle can enable the agent to correctly understand the user’s requirement, and maintain the consistency in conversations.

For games and open-world exploration, LLM-based agents always maintain post observations as task contexts, and store experiences in previous successful trials. By leveraging past experiences, agents can avoid making the same mistakes repeatedly and achieve a high-level understanding of environments, thus exploring more effectively. Some of them can acquire external databases or APIs to obtain general knowledge [99, 93, 159, 161]. Wang et al. [99] save obtained skills into memory for further usage in Minecraft. Zhu et al. [93] store and retrieve successful trajectories as examples for similar tasks, and utilize external Minecraft Wiki by API calls. Wang et al. [159] construct multimodal memory as a knowledge library and provide examples for prompt by retrieving interactive experiences. Yan et al. [161] maintain working memory for decision-making, save and retrieve relevant past experiences, and implement external datasets for general knowledge.

In summary, no matter inside-trial or cross-trial information, the key aspect of memory is to reflect on past interactions and draw experiences that can be applied to the subsequent exploration. In addition to accumulating experience through self-involving trials, absorbing external knowledge as part of the agent’s memory is also an important way to enhance the exploratory capabilities of the agent.

In the scenario of code generation, LLM-based agents can search relevant information from the memory, thereby obtaining more knowledge for development. They can save previous experiences for future problems, and also maintain context in conversational development interfaces [142, 144, 1, 109]. Tsai et al. [142] construct an external non-parametric memory database, which stores the compiler errors and human expert instructions for automatic syntax error fixing. In [144], personal information will be stored in the memory, and helps in retaining context and knowledge for decision-making. Qian et al. [1] adopt multi-agents to develop software, where each role maintains a memory to store the past conversations with other roles. Li et al. [109] also focus on software development, and the agent can retrieve its historical records preserved in memory when errors occur. Zhang et al. [114] can search relevant information when they face problems on code generation.

By leveraging external resources, the agents can learn from code-related knowledge and store it into their memory, thereby enhancing the capabilities of code generation. In addition, the memory can improve the continuity and consistency in code generation. By integrating contextual memory, the agent can better understand the requirements for software development, thereby enhancing the coherence of the generated code. Furthermore, the memory is also crucial for the iterative optimization of code, as it can identify the developer’s targets based on the histories.

In the field of recommendation, some previous works focus on simulating users in recommender systems [95, 108], where the memory can represent the user profiles and histories in the real world. Others try to improve the performance of recommendation, or provide other formats of recommendation interfaces [149, 102]. Wang et al. [95] simulate user behaviors in recommendation scenarios to generate data for recommender systems, and the agents store past observations and insights into a hierarchical memory. In Huang et al. [108], the memory in LLM-based agents can archive the user’s conversational history over extended periods, as well as capture the most recent dialogues pertinent to the current prompt, to simulate interactive recommender systems. It also uses an actor-critic reflection to improve the robustness of agents. Item agents and user agents are equipped with different memories in [149], where item agents are endowed with dynamic memory modules designed to capture and preserve information pertinent to their intrinsic attributes and the inclinations of their adopters. For user agents, the adaptive memory updating mechanism plays a pivotal role in aligning the agents’ operations with user behaviors and preferences. Wang et al. [102] memorize individualized user information like reviews or ratings for items, and acquire domain-specific knowledge and real-time information by web searching tools.

For both simulating users in recommender systems and capturing their preferences, retaining personalized information through memory is essential. A critical challenge lies in how to align the personalized information and feedback with LLMs, and store them into the memory of agents. It is also an important task for bridging the gap between conventional recommendation models and LLMs.

Medicine Domain. In the field of medicine, most of the previous works empower LLM-based agents with external knowledge in their memory [107, 129, 132, 151, 115]. Wang et al. [107] fine-tune LLaMA [127] with medical knowledge graph CMeKG [165] in QA form, in order to enhance their medical domain knowledge. Xiong et al. [129] adopt LoRA [131] to efficiently fine-tune on foundation models for healthcare. Wang et al. [151] empower LLM-based agents to acquire text-based external knowledge as reasoning reference. Besides, Shi et al. [152] build memory upon the most relevant successful cases from past experiences, and use similarity metric for the retrieval of relevant questions in the medicine domain.

Finance Domain. Some previous works also apply LLM-based agents in the finance domain, whose memory can store financial knowledge [113], market information [154, 156], and successful experiences [157, 155]. Yang et al. [113] construct financial investment dataset to fine-tune LLaMA [127] to empower knowledge on investment. Li et al. [154] design a layered-memory structure to store different types of marketing information. Wang et al. [155] record the ongoing interaction like exchanges and information to ensure consistent response, and record prior outputs as experiences for retrieving relevant examples to provide a diverse learning context for agents. Koa et al. [157] store past price movement and explanations, and generate reflections on previous trials. Yu et al. [156] adopt a layered memory mechanism to provide abundant information for reasoning.

Science. In the domain of science, some existing works design LLM-based agents with a large amount of knowledge in memory to solve problems [158, 160, 162]. Chen et al. [158] include molecule database and online literature as external knowledge for memory in LLM-based agents, and retrieve them when they need related information. Zhao et al. [160] and Chen et al. [162] empower domain knowledge by fine-tuning in Chemistry and structured materials respectively.

To build an expert system based on agents in a specific vertical domain, it is necessary to retain the domain-specific knowledge in their memory. However, there are several challenges. First, domain knowledge is specialized and requires higher accuracy, leading to difficulties in constructing memory storage. Second, domain knowledge is often time-sensitive, which can become outdated in the future. Therefore, the memory needs to be partially updated when some of the knowledge has been out-of-date. Furthermore, the substantial volume of domain knowledge makes it difficult to recall from memory based on the current query.

There are some other applications of memory in LLM-based agents. Wang et al. [166] focus on the task of cloud root cause analysis, using memory to store framework rules, task requirements, tools documentation, few-shot examples, and agent observations. Qiang et al. [167] solve the problem of ontology matching. The agents save conversational dialogues and construct a rational database for retrieving external knowledge. Wen et al. [168] investigate autonomous driving, whose memory module is constructed by a vector database and contains the experiences from past driving scenarios. Wang et al. [169] propose to improve user acceptance testing, which employs a self-reflection mechanism. After each trial, the operation agent summarizes the conversation and updates the memory pool, until the goal of the current step is accomplished.

For different applications, the focus of memory varies, as it inherently serves the downstream tasks. Therefore, the design should also consider the requirements of tasks.

At present, the memory of LLM-based agents is predominantly in textual form, especially for contextual knowledge such as observation records, trial experiences, and textual knowledge databases. Although textual memory possesses the advantages of being interpretable and easy to expand and edit, it also implies a sacrifice in efficiency compared to parametric memory. Essentially, parametric memory boasts a higher information density, expressing semantics through continuous real-number vectors in a latent space, whereas textual memory employs a combination of tokens in a discrete space for semantic expression. Thus, parametric memory offers a richer expressive space, and its soft encoding is more robust compared to the hard-coded form of token sequences. Additionally, parametric memory is more storage-efficient, where it does not require the explicit storage of extensive texts, similar to a knowledge compression process. As for the memory management, such as merging and reflection, parametric memory does not necessarily design manual rules like textual memory does, but can employ optimization methods to learn these processes implicitly. Moreover, pluggable parametric memory is similar to a digital life card, capable of endowing agents with the requisite characteristics. For example, Huatuo [107] aims to enhance agents with expertise in the biomedical field by refining the Llama [127] model on Chinese medical knowledge bases. MAC [106] is designed to create a parametric memory adaptation framework suitable for online settings, employing meta-learning techniques to replace the traditional optimization phase.

Although parametric memory holds great prospects, it currently faces numerous challenges. Foremost among these is the issue of efficiency: how to effectively transform textual information into parameters or modifications of parameters is a critical question. Presently, researchers can transfer vast amounts of domain knowledge into the parameters of LLMs by SFT. However, it is time-consuming and requires extensive text corpus, making it unsuitable for situational knowledge. One viable approach is to employ meta-learning to let models learn to memorize. For example, MEND [134] leverages the method of meta-learning to train a compact model that has the ability to produce adjustments for the parameters of a pre-trained language model. Moreover, the lack of interpretability associated with parametric memory can be a hindrance, especially in domains requiring high levels of trust, such as medicine. Therefore, enhancing the credibility and interpretability of parametric memory is an urgent issue that needs to be addressed.

The exploration of memory mechanisms within LLMs has burgeoned into the dynamic domain of multi-agent systems (MAS), marking significant advancements in the realms of synchronization, communication, and the management of information asymmetry. One pivotal aspect that emerges in the cooperative scenarios is memory synchronization among agents. This process is fundamental for establishing a unified knowledge base, ensuring consistency in decision-making across different agents. For example, Chen et al. [170] emphasize the significance of integrating synchronized memory modules for multi-robot collaboration. Another important aspect is the communication among agents, which heavily relies on memory for maintaining context and interpreting messages. For example, Mandi et al. [171] illustrate memory-driven communication frameworks that foster a common understanding among agents. In addition to cooperative scenarios, some studies also focus on competitive scenarios, and the information asymmetry becomes a crucial issue [172].

Looking ahead, the advancement of memory in LLM-based MAS is poised at the confluence of technological innovation and strategic application. It beckons the exploration of novel memory modules that can further enhance agent synchronization, enable more effective communication, and provide strategic advantages in information-rich environments. The development of such memory models would not only necessitate addressing the current challenges of memory integration and management, but also explore the untapped potentials of memory in facilitating more robust, intelligent, and adaptable MAS. As evidenced by pioneering research, the evolving landscape of LLM-based MAS sets a promising stage for future innovations in memory utilization and management. This exploration is expected to unravel new dimensions of memory integration, pushing the boundaries of what is currently achievable and setting new benchmarks in the realm of MAS.

Lifelong learning is an advanced topic in artificial intelligence, extending the learning capabilities of agents across their life-long span [173]. Agents can continuously interact with their environment, persistently observe environments, and acquire external knowledge, enabling a mode of enhancement like humans. The memory of an agent is key to achieving lifelong learning, as it needs to learn to store and apply the past observations. Lifelong learning in LLM-based agents holds significant practical value, such as in long-term social simulations and personal assistance. However, it also faces several challenges. Firstly, lifelong learning is temporal, necessitating that an agent’s memory captures temporality. This temporality could cause interactions between memories, such as memory overlap. Furthermore, due to the extended period of lifelong learning, it needs to store a vast amount of memories and retrieve them when needed, possibly incorporating a certain mechanism for forgetting.

A humanoid agent refers to an agent designed to exhibit behaviors consistent with humans, thereby facilitating applications in social simulation, studies of human behavior, and role-playing. Unlike task-oriented agents where greater capability is typically preferred, the proficiency of a humanoid agent should closely mimic that of humans. Consequently, the memory of humanoid agents should align with human cognitive processes, adhering to psychological principles such as memory distortion and forgetfulness. Additionally, humanoid agents should possess knowledge boundaries, meaning that their knowledge should correspond to that of the entity they replicate. For instance, in role-playing scenarios, an agent embodying a child should not possess an understanding of advanced mathematical concepts or other complex knowledge beyond what is typical for that age [174].