Name: Graph-Based Long-Term Memory: How Agentic Workflows Adapt Through Experience with Alessandro Negro, GraphAware
Uploaded: 2026-03-31T18:16:13.769Z
Duration: 2 h 8 min 48 s
Description: Graph-Based Long-Term Memory: How Agentic Workflows Adapt Through Experience with Alessandro Negro, GraphAware

Transcript for "Graph-Based Long-Term Memory: How Agentic Workflows Adapt Through Experience with Alessandro Negro, GraphAware": Hello, everybody. Good morning, good evening, good afternoon. And it's almost a little sad now, but this is the final notes a road to notes AI workshop we're hosting today. Graph based long term memory, how agentic workflow adapts through experiences, what what you also can see on the screen already. Super happy to have Alessandro Negro do the workshop with us. Alessandro is the chief scientist at GraphAware. Hi, Alessandro. How are you doing? All good. What about you? Yeah. Good. No. I'm excited. I'm happy about these workshops. They've been very, very popular. We had lots of lots of great engagements, great in excited engagement audiences throughout all these these sessions all of March, still March for one more day. So we we kept that in in the loop. And, yeah, in two weeks time, almost exactly on the day. Right? I think it's exactly on the day. No. It's it's Wednesday. So it's two weeks time still tomorrow. Know it's AI is taking place, and we'll do the full the full event. But I'm super happy to do the road to notes as as always as a little warm up series with all of you to get a little bit more hands on experience on notes, graphs, relationships, and AI. Specifically about memory for agents, I think this is a super hot topic. I heard a lot about this. People were talking about this. I was at KubeCon in Amsterdam last weekend. Now lots of people were talking to us and say, hey. I heard about this memory thing. How does it work? Context graph memory agents. What's what's going on? So I think this is very, very timely that you are covering this today, Alessandra, gives us give us a bit of an insight in in how how things are are working and what from your perspective is a is a good a good practice. And for all of you, the session is recorded, so we will share a video link with you afterwards so you can watch it in any time if something was too quick or if you if you had to skip over something. We will also share some links that Alessandro has prepared for us throughout the session, but also in the in the email afterwards so you can look at everything again later on. And if you have any questions throughout the the the workshop today, Alessandro told me there will be some some breakpoints, we have some time to cover a little bit of questions. So if you have any, please use the q and a function. You should see that at the top right over the chat where it says, you know, chat messages and q and a. Click on q and a, add your question there, and then we'll see it pop up in the q and a segment, and then we can we can look at questions and answer them. But, obviously, also, write in chat. I see people from Mexico and France who did dare to write where they're from. I think the other chat there's another chat. Please use the session chat. Bunch of people from United States. Welcome all. Great to see you. And, yeah, I think with that, I can re return to the backstage and let Alessandro have the stage. Have a good good time, everybody, and I'll be I'll be back in for for questions. Cool. Thank you, Alexander. So good evening, afternoon, or morning depending on where or when you are watching this workshop. So today, you will learn how to empower your agentic workflow with graph based long term memory and evolving the playbooks. By introducing these mechanisms, you will get your AI to learn continuously while you are using it without tuning required or at least not at the beginning. It is going to be a mixed workshop with a lot of theory, but also a lot of practical examples, demo, code, and architectural design details. So in the next days, as Alexandra said, I will also release the source code, which is a GitHub repository. Consider that it's not supposed to be a real project, so I didn't spend too much time on read me or project setup, but you will find all the prompts and land graph configuration that you will see during this presentation. So no need to put a screenshot because it will be available immediately after this workshop. So, moreover, considering that this is quite practical and it is a two hours workshop, we have all the time, so feel free to interrupt me and ask questions if you have any. There will be some moments in which I will stop on purpose and I will ask you if you have some questions. But really, feel free to interrupt me whenever. But before we dive in, let me quickly introduce myself so that you understand why I'm here in front of you. As Alexander said, I'm the GraphAware chief scientist. And at GraphAware, we spent more than thirteen years helping customers adopting graph technologies and in particular Neo4j for mission critical tasks, spanning from recommendation to data analysis to advanced search to natural language processing. And in the last six plus years, actually, we also developed our own graph powered intelligence analytics platform called YUM that is built entirely on top of Neo four j that is specialized in law enforcement domain. So some of the screenshots and the demo that you will see today come from YUM. But what I'm going to present you today can be reproducible completely by using the code that I will give you and Neo4j, of course. I'm also the author of two published books that are graph parallel machine learning and knowledge graphs and LLM in action that are, let's say, in the middle or that are spanning between graphs and the machine learning natural language processing. And so what does it mean? That really I spent more than half of my career working at the intersection between graph machine learning and natural language processing, and in particular, the last years, the LLMs in the graph space. So you can imagine how much excited or glad I am to present here at NODES AI. And if this is not enough, I also have two hours actually that is like the paradise for for an Italian like me. So in order to help you to follow the content of the workshop, I like to give a full picture first, including the problems, the type of problems that we would like to to solve and the high level idea of how we are going to solve it. And then we will go in the details, and I hope that this will help you to follow in a better way. So generative AI have become incredibly powerful as you have seen in the last years. The agentic approach has become a common pattern. You know, these autonomous agents combine multiple tools, reasoning steps, actions, and knowledge sources to get users what they need. We've gone from the early stages with impressive but somehow available results to ones that can to systems that can access a specific domain knowledge through our knowledge graphs or other approaches. So the initial problems that we had, like hallucination, for example, or lack of domain expertise, have been tackled or heavily mitigated by these technologies and these approaches, and agents played a key role in that. So now these agents can give you accurate, grounded answers by pulling from structured or unstructured knowledge, including a lot of amount of code. They can cite sources that they can give you they can, let's say, can navigate complex domains, and this is absolutely a huge progress. But there's a critical piece missing here. I don't know if you remember Dory from funding NEMO. So she was helpful, enthusiastic, ready to assist, actually, but she had no short term memory. Every few minutes, she'd forget where she was going and what she'd learned. They this situation describes pretty well the main problem with the vanilla agentic AI systems. They can access knowledge and tools, sure, but they can't learn from experience in their original format. Not at least without going through complex and extensive training processes. They can't remember what worked the last time, what failed, adapt based on user's feedback. They reset with every new conversations. They make a mistake, well, they will make it again tomorrow. Give them feedback, it's gone in the next section session, and every iteration starts from zero. So to be truly intelligent and supportive, then AI needs two things. One is the knowledge foundation, and we've built a, you know, RUG or GraphRUG and knowledge graphs to solve these issues. So we consider this as sorted out. But, also, we need the ability of this system to grow, to adapt, to retain memory on long term from experience and use this experience to get better next time. And this is what we are going to discuss in this workshop. And here is the bonus or the, you know, the stake. Because for mission critical decisions, that is the, let's say, domain in which we work, this learning should be transparent and explainable. Analysts needs to know why AI is suggesting a particular approach, what experience led it there. So here's the question. What if AI agent can could learn exactly like just human humans do? So that's what we are going to solve today. And trying to solve it, we discovered that this is not an easy problem. Well, actually, quite complex. So because while we were solving it, we faced two major challenges. So the first one is that we can't just use a full past experience as another Rag data source. Sure. We can store every interaction, every output, every piece of feedback. But if we treat it as another knowledge source to retrieve from, we will be back in the from where we started. Again, the indirect cows and the complexity. So we needed to convert experience into something distilled and useful, like patterns, strategies, lesson learned. So we need a lot of like, this is what happened 147 times, but here here is what we learned from this 147 experience. The second challenge is that even though we store only patterns, lesson learned, etcetera, so we store smart, we can't dump all of these into LLM every time that we ask the question. Imagine if every time that you had to solve a math problem, your memory will brought back historical information about Roman invasion or how to prepare the best picanha ever. You know, definitely, you'd lose focus completely. With LLM, we face the same problem, at worst, because we are limited by the context window we can use. You so this context is growing. I I teach you more the iteration. We hear that, okay, the number of tokens you can send grew quite significantly. Absolutely, that but this will never be infinite. Moreover, even though this will happen by chance, you know, passing all this information will cause LLM to lose focus. And so if we bring this relevant memory, this would bury the signal in noise. So we need a selective memory, the right information at the right moment for the right for the right question. So how do we solve this? Well, let me spoil a bit what we are going to discuss today. This is a very high level architecture or vision of what we are going to see. So what we are doing literally is to combine or bring in together two of the hottest breakthrough research areas in the AI agents right now, and not only, of course, in the AI agent, but in general in AI, and also we extended them a bit. And these are the context graph and the agentic context engineering or ACE if you prefer the short version. So let me describe them first, and then we will see how we combine them. So first, graph. These are comes from a very recent research on capturing the complete story of decision making. At the early stages, or this is generally referred to human processes, you know, what decision is taken, why, what were the other options, and why they were discarded, for example. The more context, the better because we can get more information while we are taking the next decision. Applied AgenTiC AI, the idea is elegant because it tracks every decision your AgenTik system makes along with the context in which it happened, the options that were available, the reasoning around that, and most importantly, the outcome including the human feedback. So it's not just about what did the agents do. It's also what what else sorry. Why? What else can be done and how did it turn out? So the complete picture, as I said, with its provenance, and this is what enables both real learning and explainability. And thanks to graph databases that extend our storage capabilities and even offers a flexible way of storing all this information, we can literally store everything, even connections among the different steps, iterations, etcetera. And this is not only useful, of course, as we will see for growing our knowledge, but also it's also useful for traceability, reference, debugging, and of course explainability as we will see. The second important research area that we are going to merge here is this concept of agentic context engineering. I'm sure that we have seen many articles and papers on this topic, including the one that I mentioned here from where the charts are taken. Also, Anthropic, the the Cloud AI the Cloud AI, let's say, company, you know, has an interesting blog post about it. So it's the evolution of the prompt engineering. You know, because instead of having the static prompt that is manually or automatically edited, this context is passed as an evolving, a dynamic piece of content that could help the system to perform better next time. So ACE creates a leading playbooks as we will see of experience that evolve as our understanding of the world deepen, as a genetic understanding deepen. And these playbooks are passed as guidance to the prompt to improve the result of each iteration as I said. And as you can notice from these charts, actually we've got, thanks to this, a 10% improvement in many different scenarios actually. So and they achieved this by preventing two critical problems. You know? One is the brevity bias, so where summaries lose crucial details, and the second one is the context collapse where knowledge erodes over over time. So coming back to our high level schema, we see in this picture our virtuous loop. So in the context graph, we track agents' decisions and human feedback. And we use this source for crafting and evolving playbooks that are used to provide guidance to agents so that they can do better next time. So context DAF gives us a comprehensive auditable experience tracking, perfect for mission critical decisions, as we said, because we can keep track of everything. We can know exactly why did the system do this. And the living playbook that is evolving, not extending all the time, gives us a created actionable guidance that actually get better over time without growing infinitely. So because they are constantly refined and not extended, this allows us to really keep it as something manageable that can be passed or as we will see can be, let's say, used as a source to extract the proper guidance the next time. And these two technologies actually can solve what researchers call the biggest bottleneck in the agentic systems right now, the memory that learns. That is exactly what we what we need. But let me show now how it works in practice. But so in order to do so, we need a real problem to solve because this will help not only to make the examples that I'm going to show you more concrete, but also it will justify the complexity of their solution as we will see. Because as you will notice, but as you can guess, the final solution will be pretty complex. So we needed to tackle a complex problem so that all these effort will make sense. So the value delivered to our users or to our company should work, you know, should be worth the the time that we will spend and the effort that we will provide by doing this. So in particular, I asked myself, you know, what makes a good test case for me? You know? So I found out that there are four critical things to make it, you know, reasonable. So first of all, we need a problem that requires multi step reasoning. It's not something that you can just ask a question and get the answer immediately. Second, needs domain knowledge that goes beyond what generally LLMs know because of their pretraining process. Third, it should benefit learning and feedback, of course, because this is the core of what we are going to see. It should interpret the human feedback and use these to get better next time. So we really need to learn from it. And, of course, since we would like to approach this problem in a a proper way, we need a clear metric to or clear metrics in general to know if we are actually improving, so we need also a proper dataset. So by reading a a few papers and and blog post, I found out what I needed. In terms of problem, I found out that these financial document analysis or financial document intelligence has the right level of complexity because it's a real problem with proper, let's say, sites to make it complex enough for our tests. Of course, it's critical because we want to address mission critical problems and definitely this is the case because the wrong answers or wrong decisions can generate bad decisions in terms of, you know, finance decision and so we can lose actually money or people using it can lose money. And these problems should learn, should get domain knowledge from the interaction with the users or from the answer and the question actually at each iteration. And this is, of course, really relevant. So luckily, I found also a good data that is called FinanceBench that I had to extend a bit to make it even more complicated, but it was definitely a good starting point. Just to give you again an example how much complicated is this problem, I also tried several for several questions inside the finance bench, a few well known cloud providers of these LLMs. You know? And, of course, my favorite one, and it is always very good, let me say, just that in this case, I'm really raising the bar quite a lot, is Cloud AI. So I asked a few other questions, and actually it was pretty accurate but not 100% accurate. As you will see in this screenshot, I first asked it the question and then I passed the real answer and it noticed that there is an issue in one of the documents, in one of the years in particular. And this is happening because it struggled to find the full document, to go through the full document because it was too big. But of course, it can do better with the rug, etcetera. But this is exactly what we are going to do. So as I said, you know, just now, I mean, the problem of answering proper complex financial question can be solved by using a rug. You know, any particular, you know, financial rag is exactly what we are trying to to address here. So the problem is is clear and but it's complex even with with this hint because there are a few aspects that we need to consider. And here we have a few example of the questions, again, to understand why these questions are complicated. For example, the first one is a three ms capital intensive business based on the fiscal year 2022 data. So this is complex because of the the word capital intensive. It's something that is close to that is definitely not available in, let's say, in the in any of the documents that we have. And so it make text search or even the the vector search quite complex. You know? So the first test that I run, I noticed that there was no way for me to retrieve the right documents for this specific question, so I struggled a lot to to find a solution. And then the other issue from the second question is that for such a large set of years, let's say, range of years from 2019 to 2022, actually, we need four different documents, and each document can contain easily more than 65 or 70 pages. So we are talking about a size letter will go way above the capability in terms of the window. So we have these basic issues, but we have also more, let's say, complex issues. For example, even though we will find the the word capital intensive, you know, what's the threshold for considering it if three m is capital intensive or not? So actually what happens in particular for this question is that some people can consider capital intensive the CapEx over revenue and the threshold could be 50%. Other might use CapEx over assets with a threshold of 25%. So this is really, first of all, domain specific, but also it is user specific. So different users can have literally different needs. And so if we come out with with some models, maybe that this won't work for another use case. So actually, it needs to follow the requirement or the the feedback of the user while the user is using it. So it's not only about the domain expertise, but it's also about the user preferences. And as I mentioned, you know, there will be multiple documents in this sense. So let's try again to explain better why, for example, a simple RUG or a one shot RUG would fail. For who doesn't know how the vanilla RUG works, here is a simple image. You know, here we have the question starting, you know, the the type process, then this question is converted in a in a vector, the same embedding function that has been used to vectorize the the chance of the documents we have. So by using a similarity that can be cosine or whatever, you can get the most similar chunks related to your question, and then you can pass these to the answer generation. And then you can take all these and pass to LLNs, and at that point, you will get the answer from LLM. So basically what happened in this case is that as I mentioned with the capital intensive question, you won't find that much or you will find wrong documents probably. So you won't be able to answer. This is exactly what happened to Marie during my test. Of course, just to be clear, there will be way more complex implementation, some using a graph probability that's super powerful if you like to answer very complex question. But still, what I'm challenging here is the concept of one shot so that you decide upfront which documents to retrieve and then you pass them to the answer generation phase. So there are techniques to make it better, but still the problem here is not the retrieval phase, but the fact that it happens before the answer to the question, and it's one shot. So you can't go back and, you know, improve the result because you don't have the the right content here to answer the the question. So and and even the the brute force won't work because passing everything, as we said, it's impossible. So in this question that we've seen before, passing four ten k document is absolutely a no go. None of the current models will support such a large token size or token window. You know? So static retrieval strategies can handle the dynamic reasoning task. So that's why this is a perfect problem for using an agentic approach. And in this case, since it is applied to to RUG, this is called agentic RUG. So basically, we flip completely the way in which it works because the paradigm. So instead of retrieving everything upfront, it's retrieve, think, decide what's next, or think, retrieve, decide. I mean, it can be organized I mean, structured in a in different way. So in AgenTeq Rug, the AI becomes an active participant in the retrieval process. It decides what tool to use. For example, if it's vector search for conceptual query or keyword search for exact terms, structured query for more complex but organized data, document navigation to hop between sections, etcetera. It makes this decision during the reasoning process, not upfront. So here's the power. It can adapt. The agent retrieves the document, evaluate it, or many documents actually, and think, okay. Is this set useful for computing, for example, capital intensive signals? If not, it can come back and retrieve other type of information. So if the vector search returned nothing because of the capital intensive issue, then it will try with a keyword approach. But more interestingly, doesn't have only the the question. You know, it can ask for specific type of information inside the documents. So this gives us a focus and context. We only pull what we need when it's needed. So no brute force, no drawing in irrelevant documents. And, crucially, we can use a different tool, so for different type of questions or for different subtasks. So the right tool for the right job. But how can we orchestrate this is the real question because agenticRAD is a generic concept, then we needed to implement it in a real, let's say, architecture. So IdentityRug, as I said, is a generic concept and then we will see how we are implementing it. Let me clarify that this is just one of the many possible implementation. Maybe some of you already used IdentityRug in the past and they've seen a different version of it, and it will be perfectly fine. Remember that the scope here is to have a complex problem that we would like to solve with an agentic AI system. So the problem has been identified, that is the financial document intelligence. Now we need to identify the agentic architecture to solve it. We identify the RUG as the problem or let's say a sort of solution, in particular, agentic RUG that is exactly what we would like to use as an example. But what we are going to discuss later in this presentation can be applied to any other type of argentic configuration for different type of scopes definitely. You know, so just to clarify where we are here. So we are using this agentic RAC implementation as a base, but it can be applied to whatever you need. So let me explain this implementation in particular. Again, the starting point and the end point is exactly the same as before. So we have the question, enter new system, and we would like to get the answer. So the problem, let's say outcome doesn't change. But what happens in the middle is the measure issue. So here what I did was to divide the problem in two major, let's say, subtasks. On one side, we have the retrieval phase and on the other side, we have the answering phase. So the purpose of the retrieval phase, as you can guess, is to get the right set of documents to answer the question. But this is not done by a single element. It's done by these three elements. Actually, let me say that this is the explosion of what is commonly called the React agent. But instead of using a simple React agent where I have literally no control over, I prefer to split it so that I can keep track of every single decision and I can implement it as I wish. And in particular, in the React agent, have only one prompt that I can use for the entire retrieval process. Here I can have, as we will see, two different prompts. One for the retrieval strategy and one for the evaluation. But let's see how it works. So the retrieval strategy is the element that literally decide what is the the next strategy to take the right information out of the documents. Then once the tools are identified with the parameters, etcetera, there are the actual calls to these for these tools and the documents are retrieved, and this can happen in parallel. So actually, the retrieval strategy node can decide to run multiple calls in parallel. Then there is another, let's say, element that actually does two things. On one side, evaluate the quality of the retrieval. So it takes the question, the result documents, and evaluate, okay, these documents, these chunks enough to answer the question? And then it, if this is the case, produces a so called, let's say, growing knowledge. So it extracts from these documents what is necessary to answer this question. If at this point the extracted knowledge is enough, then it will move to the next phase that we will discuss in a second. But if this is not enough, you see there is a feedback loop that will come back to the retrieval strategy with the feedback to say, okay. This is what we have gone we what you gave me. This is what is missing. Please focus on what is missing and and extract these from the tools from I mean, via the tools or from the documents. And this is exactly what the retrieval strategy node will do in the next iteration. We'll actually take this into account and make a different call to extract only the missing pieces. So as I said, if everything will be fine after one or multiple iterations, we move to the answer generation. Let me say and stress once more that here we are not passing all the documents because we grew up this knowledge that is all the answer generation node needs to answer the question. So there is no documents passed anymore. At this point, the answer generation will answer the question. And at this point, we have, let's say, LLM as a judge whose role is to evaluate the answer and, again, take, in this case, not two, but three type of decisions. The simplest one is, okay, the answer is fine. The computation is fine. Let's move to the end and provide the answer to the user. The second option is that there are some computational, I mean, errors, but still the data provided by the retrieval phase is correct, is enough, so let's go back to the answer providing some feedback to the answer generation. But there could be that the answer evaluation will come back and say, there are some missing items in the generated knowledge. Please retrieve other information and again pass some retrieval feedback to the retrieval strategy and these will again extract the missing information and the iteration will continue. So that's how it works at a high level. So as you can imagine that the process is complex. Of course, as we will see, the accuracy will be enormously higher than the one shot drug, but of course we moved from one single LLM call to actually four different calls if everything will go well. Otherwise, this can be literally, you know, a very long number of iterations, you know, and actually this means a lot of LLM calls to the you know, that can generate more or higher costs for for the company. But, you know, the return is that the answer are answers are accurate. So this is not, let's say, a talk around RUG. I spent already a lot of time discussing how it works, but again, I needed you to understand what kind of agentic system we are putting together. Here in the next three slides, you can find some key difference between the one shot and the agentic drug that justifies why you will need an agentic drug or when you will need an agentic drug. What I would like to stress on because this will be useful later is that the preprocessing requirement is totally different because in the one shot drug, you have to spend a bit more time on the, let's say, preprocessing because for example, in the case of graph rather you need to decompose the text in the entities, in the relationships and so on and so forth. Or you're able to summarize the content of some documents or create the hierarchies, etcetera. In the case of agentic RUG, as we will see, this will be way simpler. So because we moved the complexity to the retrieval part, not to do the preprocessing part. And of course, the complexity, as you can guess, is completely different because of the in complex agentic RUG, you have very complex structure as you have seen. But the latency, of course, will be higher. And on the way back, of course, the answer quality will be higher. Arrow recovery is in place while in classical one shot RUG is no recovery or at least not in the standard way. And it can adapt things that one shot drug cannot do. And of course, in our system in particular, we will see how the full trace will help to to keep track of every single decision here. So the problem of keeping track of what's going on in the in the agents or in a conversation is definitely not new in the sense that this is a very well recognized problem. So there are out there a few other solutions that might be you have seen before. I would like to mention this too because in my opinion, they are they are more production ready and actually, they have a lot of interesting use cases. So it's important to mention them to compare what I'm going to propose you to what is available in the in the market, you know, as other other options. One is MEM zero. So MEM zero is the most production ready, let's say, memory system available. Its focus is on keeping track of conversations, and its core insights is that you don't need the full conversation, but you need a compressed I mean, the duplicated set of important facts. So it has these mechanisms through which after each step or after each iteration, you get some manipulation, some merging of the content. And this is important because you will see something similar even in my approach. You know? So you have that at the end of the process, you will get some addition, some update, some deletion, or some, I mean, skip or some no operation, you know, once you it received a new message. Of course, in terms of latency, it is one of the best tool because it reduces a lot of the latency. And actually, it has this seven k concept, which means that it is able to keep the full, let's say, memory around the seven k tokens. And the problem that we have here compared with what we are going to see is that no answer, let's say, quality will come after thanks to feedback loop. You know? So the memory updates just based on conversation content and unless you implement it, there is no natural way to get feedback from the user or at least not so immediately. Then the other platform or tool that I would like to mention is ZAP or ZAP. The reason is because it is, in my opinion, the the most, let's say, ambitious memory system available. And it's also interesting because, of course, we are at NODS AI, and it uses a heavily temporal knowledge graph as a structure of the memory. You know? Of course, it needs, let's say, richer structure because, you know, it stores entities, relationships, and the entire timeline, so in these three tiers, which is a bit complex and is quite a huge overhead. So the benchmark provides strong, let's say, results, but this must be justified by the reason why you are doing it because the overhead is higher, so the latency compared with MEM zero is higher. But, of course, there are scenarios in which you don't care about the latency as we would see even in our case, but you care about accuracy. And so definitely that is one option that you should take into account. But again, here there is not that much related to a feedback loop, etcetera. So I will stop here for a second because we reached the point where I gave you the full picture of what problems we are going to solve, what is the agentic, let's say, system that we are, let's say, addressing to prove that what comes next can improve the quality of this system. And now we also have a better understanding of how critical are the agents in the generative AI field. And I think it is clear how long term memory can help these AI to learn. What is missing actually is how we are going to implement this. So we have our multi agent system, so we can start by discussing about the the details of how to implement it, what are the different components that we needed to to implement. I barely discussed about the preprocessing when I was comparing a RUG, one shot RUG with a Genetic RUG. As I said already, the preprocessing is pretty simple. And in this case, what I'm using in the system the table we see live, is a very, let's say, simple preprocessing step. Simple because differently than what happens, as I mentioned, with the GraphRAD, I'm not extracting entities. I'm not extracting relationship. I'm using only one summary, not a RTO summary, etcetera, etcetera. So there are all these simplification. And actually, I'm using only one call to LLM per document, which is definitely way lower than what you will get if you are using GraphRAG. So the process is described in this picture. So we have, of course, the first OCR because the documents are all in PDF, but this is up to you. I'm using Docling on my computer, but we also tested Azure, the old form recognizer now called Azure Document Intelligence, that allow you to extract what you need from the text actually, so the content even quite well. Then on top of these, we apply some semantic chunking. So instead of using just a fixed side chunking, as we will see in the demo in a few slides, what we do is to organize the content so that the content per se is self contained and we have also a description of this content because this is easier to read and to understand and actually, let's say the content helps us to find what we need in a better way. And definitely this is proven in our test to be more effective on the long run but also for understanding the final results. What we also tested was to, in particular for tables or for images, to pair these previous OCR step with some large language models, let's say multi model language models post processing that literally take the result of the OCR and semantic chunking, take also the picture literally of the image and extract or fix the content, you know, because it can happen that, of course, the basic OCR cannot extract relevant information for images or in particular for tables, they cannot be structured in a proper way. So this post process can help a lot to get better results of fixing a lot of issues during the process. At that point, we have a process that involves LLM that extract because the post processing is optional because before I said that I have only one. This is true because in most of the cases, we are not doing the post processing. If you have these, of course, it will be that you have to close to LLM. But in general, we are not using that much for our use cases, but you could need it. In our case, as I said, we are using LLM for extracting titles, summaries, and tags. Actually, we are passing the first couple of pages just to get for our 10 k or quarterly documents the title because we don't have it in the PDF metadata, the summary so that we will be able to better add some header to each chunk and some tags to improve the extraction. Then we embed the chance and the the summaries, and we literally index these as a vector but also as a as a text. And in this case, Neo four j helps us quite a lot because with the same structure, we can we can same data structure, we can store the embeddings and and the text. In terms of, let's say, of the graph database, this is very minimalist. We have the documents that have the pages and these pages have chunks. We also shortcut a bit documents with chunks and we included also some aggregation in this process because we noticed that with semantic chunking, it can happen that you will get very small chunks. So what we did was to store both small chunks and big chunks. You know? And we let vector search or text search to find what is what is better or more accurate during the the retrieval phase. So this why we have the chunks including other chunks. And, of course, as I mentioned, we have also the the tags for, you know, for the retrieval even though, to be honest, in this at this stage, we are not using it at all. So in terms of tools, even here, we were very basic, I I would say. We have, and you will see soon, three basic tools. One is, as you can guess, vector search. So we use a similarity, in particular, Cosine for semantic search. We use a Neo four j vector indexes, as I mentioned before, and this is extremely fast. Then we can apply some filters on on documents or on other aspects, but, you know, this is optional, and this is a decision that that retrieval strategy agent will will take. We have the second tool that is a full text search tool that actually use these Lucene based queries, again, provided by Neo four j through the full text search indexes for extracting specific terms. I'd like to spend a few seconds here because in my opinion, you know, using the the retrieval strategy, say, with this full text search, actually, we don't need real entities extracted. Because as I said, you know, we simplified a lot of the preprocessing, the tanks to the agents. And actually, one of the reason is because of that. Because as you will see if you you know, in a few examples, but also in the if you run it, is that the strategy agent will be able to decide which entity to use, if it is a company, if it is a specific term, if it is a person, etcetera, etcetera. So it will recognize that this is an entity and will search for this entity in the text without having a specific node containing it. And with the understanding that it will get about Lucene query, actually it will be able to get exactly what it needs to even though it's not stored as a single node in the graph. That's why this simplifies a lot of the preprocessing phase. And then I noticed that at some point, the system was struggling to find what it needs. So I said, okay, we should mimic what a financial adviser or analyst do. They literally read through the documents. So there is this tool that is the page search tool that literally get the document name and the page, actually the list of pages. And actually it goes through these pages looking for information. So when there are I mean, the tool before failed, actually it has this fallback that literally goes through pages of these documents to find exactly what they need. But, of course, it is never the first one because it needs at least to know what is the document file, you know, the the the name of the file to get through this. And, of course, these are just three tools available among many others that could be because we can have a more complex structure. You can definitely use a graph rug if you log it if you log here, and then you will have an agentic graph rug in which you can use the graph structure to get better queries. Or you can have also your article summaries. You can do whatever you want, but here the point is have a simple implementation. So it's time now for a quick demo, actually. So let me switch to this tab. Okey dokey. Okay. So, Do we want to answer a couple of questions before we do the demo? yes. Absolutely. Because we had a couple coming in, and I'll I'll I probably won't be able to have the time for all of them, but maybe we can we can start with Joey asked pretty much in the beginning, how would you distinguish between short and long term memory if you duplicate the long term memory into short term memory or do you dynamically link it? Maybe you showed us in the demo as well, but that's probably a good good starter to to distinguish what what is what and how how does what become long or short term memory. Okay. That's definitely a good question. Actually, the the reason why I called it a long term memory, and I even didn't mention short term memory, is because I I think that we refer to short term memory to what MEN zero or ZEP are doing. You know, they keep track of what happened in the, let's say, in the specific conversation. You know? So to me, these tools are focusing on a short term memory because they keep track of the what's going on in the conversation. Of course, they store it and these automatically will become long term memory. But here, you know, what I meant for long term memory instead is that actually we are keeping track of everything is happening in the system as we will see forever, literally. So how you will see in a few next slides, actually what we are doing is to keep track of every single decision and this will be tracked forever. And from this memory, we are extracting from this long term memory, we are extracting our, let's say, playbooks or our evolving playbooks. So that's the the distinction. In my system specifically, I'm keeping track of every single decision, which, you know, if we look at it as a single question answer, it's short term, but the fact that I'm using it, you know, and you can use it literally by condensing them, by aggregating them, for example, it's immediately becoming long term memory because it's not connected to a specific session, so answer and question. And actually, I'm not using it during the session, but I'm using after the session to extract knowledge. That's why I distinguish it between these two terms. Okay. Makes sense. Ayesha asks, when you talked about the the agentic loop and the with the LLM as a chart judge segment, how the feedback implicit or explicit, is is is that stored in the graph to refine future actions and how how how that makes you know, Well, is that best practice maybe? yes. I mean, absolutely, these will be visible in the examples that we will show later, Super. you know, once. we will go through. the concrete examples of how this loop is closed with, you know, inside the graph and also inside the prompts that we are, you know, using to answer this question. So it will be visible in a few. minutes. Okay. Super. And then Jay asks, because you talked about that you had the best results with ZEP, if you have any benchmark data used or how how do I mean, was this just personal experience, or how did you decide for it as you come to the conclusion? Well, I mean, just to be clear, I didn't say that I I had the best result with ZEP. What I said is that I like ZEP the most because, as I mentioned in the beginning, I'm a a great fan of graphs and the knowledge graphs. You know? And and so the basic idea is that ZEP can I mean, is entirely based on the concept of the temporal knowledge graph? So that's why I like it. You know, I didn't test. yet on this infrastructure, but this is something that I have in the in the road map to compare these results with what I will get by using the ZEP and the MEMS zero, and I will publish something in the in the next months about around this. But, again, I didn't test it. I just went through how it works, and I checked the you know, how these could solve my problem, and then I decided to. go my own way and I implemented it. But, really, Yeah. ZEP is, It's it's mind, what's the best solution, you know, because I love knowledge graphs, and it is. based on knowledge graphs. Yeah. And then always, you know, make your own tests. Let's try try something what works with one use case or one specific, you know, stack doesn't work with another is is maybe not as as well performing with another. So even if we show something here, you know, take your own, you know, time and and try something, play something, explore different options and tools, and see what works for you, how you feel, you know, results wise, but also, you know, what's more fun to work with, what's what's easier to to do and and to build, and and and, you know, just just go with that. Totally agree. Yes. This is absolutely true. You know? And that's why I would like to make a comparison for this specific problem, and I'm sure that there will be tons of other problems in which MEM zero, for example, will be the best solution and others in which ZAP will be the best solution and a few others in which what I'm presenting today. could be instead the optimal solution for what you need. So absolutely test. everything and use the right tool for the right problems, you know, for the right job. Yeah. And then maybe maybe one more question. Marc Henri asks, do you also cons did you also consider more traditional tools for preprocessing like spaCy? I mean, they would probably not, he says, probably not they would not be able to to summarize, but but still is that is that still an option, or are they outperformed by by all the other more AI driven things that are currently available? Yeah. I mean, let's say that if you go through my first book, all the natural language processing part was done by using spaCy. And actually, at Grafelberg, we have used it for several scenarios many years ago. I have to say that the reason why I don't see it useful in generic terms is because it struggles when we come to specific domains. You know, as I said, we operate in the law enforcement space, but we also operate in healthcare, for example. And we struggled a lot not to recognize people, for example, where SPACI can excel, but when we started saying, okay, we would like it to recognize the specific terms or specific, how to say, relationships, unfortunately, all these tools failed. In that sense, LLMs can do better because you can easily, let's say, tweak the prompt to extract what you need. So in this sense, I would always go for LLMs unless, of course, unless the data you have is so secretive, you know, or so you have to protect the privacy of people. that you can't use public LLMs. In that case, of course, I will go for spaCy. But again, honestly, it works really in specific cases. We also had in Hume many years ago, actually, a tool for annotating documents to customize spaCy to extract what you need, but it was so tedious. So personally, I would be far from that. I tested. We also promoted in a historical part of our life, and, actually, we decided not to go for that because it's too complex, you know, when you come to something that is different than people, organization, location. No. Yeah. Makes sense. Alright. Maybe we do one more because I think it's also interesting from Orlando. How quickly do we do we expect storage needs to grow with continuous memory ingestions? Do you think that that could become a problem with with this pipeline? Well, another very good question. So let me say that now I'm storing everything. So every iteration, I'm I'm storing all the details. But let's say that the way in which it works, and you will see in a few minutes, is that once I extracted the track of what happened to the question answering phase and also the entire process, I have a step that will literally extract the evolving knowledge or update the playbook according to what happened in that specific session. In that specific moment, in theory, you can completely remove the trace. You don't need it anymore because you extracted already what you need and so you can keep to the playbook. And the playbook, as you will see, is very, very small. So it's very, very limited. Of course, at this stage of the project, I'm even not thinking about removing it because I need to keep track of every single decision to verify if I'm literally improving thanks to guidance. And you will see as a part of this demo how much information I can extract thanks to to this history track completely. But, of course, in production, I agree. You know, you can't keep everything. But for that, you can decide to go for, okay, obsolescence. So after, I don't know, one week, you can remove all the questions older than that, or you can keep these by daily, and so you can remove maybe relevant question, or you can decide a different policy. Okay. If it's an interesting question for whatever reason because done by an interesting user or because it generated a longer path than usual, then you can retain it and remove the rest. Actually, you can put a cleanup strategy or a TD up strategy after, and this won't affect. what you are going to see today. Cool. I want to keep you longer from seeing it actually in in action. So yeah. Let's let's let's continue with that, and we'll have we'll have some time for a couple of more questions later on. I'm sure. Sure. Yeah. So, actually, it was good to have a moment to to answer some questions so that we can keep in any case this part, I mean, shorter because this is just about a few details of how the documents are stored in the graph. So let me show you this screen. So here, I can perform some search to show you, for example, how these documents are stored, and then I will show you an example of the so as I said, this is the document that has chunk that and the sorry. That has page, that has chunks, and then we have the the bigger chunks. Now then we can easily filter, for example, for some document like starts with three m. Okay. And then we can create a preview, and we can expand so that you can see the structure. So here we have the the documents where we have the document I mean, document summary. We have the the source file. We have the, you know, the tags, actually, and we have, I mean, just basic information, like the number of pages. Then we have the pages, and then we have the the chunks. Let me show you the chunks. Here, you can see the the embeddings because they are filtered in the view, but they are there. I would like to show you an interesting thing here because here we have, for each chunk, two text. One is the text as it is extracted from the, let's say, the the semantic chunking. So this is exactly what is extracted. And we have also an extended text because what I noticed at some point was that some of the chunks didn't contain, for example, the name of the company or the the year of reference of that document. So, actually, I since I extracted the title and the summary, I have a small portion of it that is an header of the real text. So the real text is what you see here. You know, in the text, this is what has been extracted as a result of the chunking process. But in order to improve the quality, actually, I had to put this error upfront and this dramatically improved the quality. This is just a trick that I wanted to share. Also, what I didn't mention is that the embedding has a the embedding function or the the model has a great impact on the final quality of this process. You know, in this case, I'm using a b g m three, and I'm using it through a LAMA just to save some costs on embeddings. But, you know, I tested many others and the quality literally changed the final results according to the type of embedding that I'm using. Another thing that I would like to show you quickly is is this one. So let me show you this this document if I can get it. Okay. This document is an example of three m fillings for the 2018. Yes. Because it is yeah. 2018. So actually, you see that these are documents with a complex structure and very interestingly, they have a lot of tables to process. I have to say that the DocLink did an amazing job. Azure Document Intelligence did even a better job, but they are, I'd say, comparable. What I would like to show you is how tables can be stored in chunks to make a better retrieval. So you see here, you you have a table, then you can decide to store it as a, let's say, metatables, so actually with the same structure, the columns, etcetera, using, you know, the common structure. But you can actually store it in the way that I'm going to to show you. So let me get some some chunks for three and company 2018. So if I go to page 11 or eight, material, you see that the title is always useful of okay. Executive officers. You see that instead of storing the table, you know, as table, actually, it is stored in this way where we stored facts. And also this tooling guy has age of 65. This tooling guy has the current position, executive chairman of the board, etcetera. So Dockling was able to process this table and actually extract the information in a way that is better for retrieval, for example. And this, again, improved a lot the quality. But again, don't I want to discuss too much about the reprocessing because definitely not part of this. But if you like to replicate these these results, you need to consider also these aspects. Otherwise, you will get worse results and say, I suddenly did something weird and make the result fake, but the reality is because I spent significant time to make it better and better. So that's where we are. So at this point, we have our multi step GenTIC RUG that we described, and this is exactly what we would like to to improve. And as somehow you asked, in order to get the full picture and see where the, you know, the feedback comes into play, where the guidance comes into play, we need to see a bit of the code actually. So let me show you just a bit of code to show you where you will be able to find the prompts. Here we go. So, actually, this is the the the project, you know, that I will share after this this event. The entry point for me is this Evaluate Rug implementation that actually runs through this benchmark and run all the tests. And it can use a different type of configurations. So changing the RUG type, you can switch from AgenTiGRAUG to classic RUG and other configurations. And then you can also switch between different embedding models. As I mentioned, I'm using BG M3. I'm also using different tests, different databases actually. As you can see now, while we are speaking, I'm still running the last test as ablation because I would like to be sure that the quality is improving. But what I would like to show you at this stage is the agentic part. So this is the the description of the the workflow. As you can see, I'm using land graph. And I'm sorry just for the names because I didn't have the time to match the the slide's name with the name of the agent node here, but, you know, I will fix it before giving you the the final version. So here we have the the retriever strategy agent. Of course, we are logging the question, but this is just a log. So we have the retriever strategy agent, the retriever, that is the the element of making the calls to the tools. Then we have these great documents that is what you've seen as a retrieval evaluation and logging knowledge. Then we have the question answering and the reflection and then of course we log the final answer as an independent node. Then of course we have these, let's say, description of the graph structure where we have all the edges plus a couple of conditional edges that are the squares that you have seen there where we decide where to go if we have to go back to the strategy agent or we are good to continue to the question and answering. Or like in this case, we have the three options that I mentioned before. So ending up and providing the answer to the user or go back to the retrieval strategy or go back to the answer and generate a new answer. So this is easy to describe in LangGraph. That's why I love it because what you have on the schema, you can easily represent in LangGraph. So if you are curious about a few of the prompts, I can show you the retrieval agent prompt. Here, you have the prompt is, oh, you are a financial research analyst, blah blah blah. Your task is to retrieve to use the retrieval tools available to extract the most relevant parts. So here, there are critical use, something that I had to fix because of some misunderstanding of the tool calls, and here are some suggestions on how to use it properly to avoid to go through the same step again and again. So here, I keep track of the duration, the question. Here, we have the guidance block that we will see soon. And here, we have the feedback that can come from the step inside the retrieval phase or from the evaluation at the end of the process. I have, of course, the growing knowledge, the list of tools available. I'm also passing the list of previous attempts so that, I mean, prompt and LLMs know what to avoid because it didn't work or it worked, but this is already tested. So we avoided to do the same things again and again. And what is really relevant for this is that I'm also keeping track of the thinking strategy. So I'm explicitly, as you can see here, asking to always provide thinking with this structure because this is what I need later for taking lesson learned. And actually, we will see a few examples in this direction. And again, if this was not enough, I'm also repeating again, you always must provide thinking. And this way, I was able to provide to get the thinking all the time. If I go down, I have the retrieval to call. Of course, it just make the calls, and then we you can find here the hold on. The prompt for the for the system that evaluate the documents retrieved and it creates the growing knowledge. So again, here we have the description of what is going to happen, the document sorry, the question, the retrieval documents, the growing knowledge because this is iterative, so we need to keep track of what has been instructed already because just to be clear, we are not passing again and again the same document. So these documents are the only one retrieved by the previous step, the previous node, not all the previous iterations and etcetera, etcetera. So we have the current strategy thinking, we also pass what we've got from the previous strategy agent node so that this element, this node has the full picture of what happened and whatnot and of course, the guidance that we will see in a while, and then a clear guidance on the explanation. The answer is the same, you know, structure. We ask you to answer the question using the knowledge, and this evaluation is an LLM as a judge in which we provide some specific let me show these in particular, the examples oh, it is here, actually. So here, we have the, you know, the the answering node because this is a different phase. So you will find in another in another file. Okay. I don't want to spend too much time here because you will get access to these documents at some point. So let me share my screen again with the workshop. There we go. Okay. So so far, so good. So at this point, we can start discussing how to store all this information. And, actually, this is the the schema of our context graph. Again, this is exactly matching I mean, this is matching exactly what we have in that architecture. So you can imagine easily that we have the question that is our entry point somehow, then we have the two phases, actually the retrieval phase and the answering phase. Then we model the iterations, so we have multiple retrieval phase iterations and we have the, let's say, the answering phases. All of them have these AI step. Of course, it is of different type, but I created just a simple one to have a single model, a single, sorry, node type. And then we have the answer feedback because I needed to track the the feedback provided by the, let's say, by the user. And interestingly, we have the opportunity also to come back to the to the chunk because we are storing in the same graph database both the data that we needed to answer, you know, the the the documents that we just processed. On the other side, we also have the, let's say, the the trace of the execution, and we can connect them easily because they are literally in the same graph database. And again, this is a greater power for us to analyze really how things are going, you know, in the system. So this structure matches perfectly what we just described. And here, you can see a few example of the thinking that I was just describing. In this case, the retrieval evaluation and knowledge builder said no, so this is not a success with this reason because some documents have not been found. The question is exactly what you have seen on the three ms capital intensity and it explain exactly what is missing and provide suggestion on what to do next. Here, we have the thinking again because even the retrieval provided some thinking about why it says no, what were the options, and what are the recommendations for the next steps. Here, you can find them, and then you can see the growing knowledge that is exactly what has been extracted. And you will see a few examples in this sense. So now I'd like to show you that some concrete examples of, you know, how we are storing these data inside the graph. So let's go for some example like the three ms since we mentioned it too many times. Let me zoom it a bit. So here, we have, you know, that as I mentioned, we have the question with all the details. So here we have, you know, the the UID that uniquely identified with that specific instance of the question. Of course, Alessandro? if you ask Alessandro? Sorry. We still see oh, now it comes. Okay. no. It is. No. Sorry. Yep. Sorry. What am I I forgot to switch to this one. So. sorry. I was showing this. I mean, this is a real result, actually, because we I asked the question, is the three m capital intensive business based on, of course, financial fiscal year 2022? And this is the actual execution that, you know, generated this graph in which we have the ritual phase and the answering phase, and we have that the ritual phase required two iterations to get to the answer, and then we needed just one iteration of the answering phase. Then we have the answer, and we have the answering. In this case, since this is green, it means that it completed successfully. So the feedback was positive. I was just showing here, I mean, minor things, but I can show you the the full trace to understand how it works. So here we have the question. This is the answer provided. So the answer is clearly no. Three ms does not appear to be a capital intensive for this and that reason with all the computation that happened. And interestingly, we have a directed trace back to the source. So this is literally the ID of the chunk from where this information has been extracted. This is the chunk of the from where this other information has been extracted and so on and so forth. So in this way, of course, for mission critical tasks, this is mandatory. So if we, you know, say something, we needed to trace it back to the source of this information, and this is the case for whatever is reported here. And again, the fact that we are splitting the retrieval phase with the growing knowledge with all the details about the sources with the answering phase allows us to keep the focus. So we have that for each piece of information before the calculation, we have the trace of the source, and then we have that the answer just have a limited amount of information with the related sources. So it's easy to, you know, provide exact and precise information about the the actual source. So here we have the feedback, so the gold answer was no with a bit of information. And, you know, the the evaluation is not done by humans in this case. So the answer is provided by humans, but the evaluation, if the answer is correct, of course, cannot be done for 200 times and for all the tests that are done auto I mean, manually. So I actually asked the LLM to evaluate it and to explain why this is correct. And, actually, we see here that the the computation was correct, the threshold was correct, and that's why it worked it out. So this is a, you know, a better form of what what happened. So in this case, this is the first iteration. We see that we don't have any retriever slide yet. So here, the system is saying, okay, my approach missing this is what is missing. My approach is to use tools. So we are tracking here the decision, the thinking, the reasoning that we needed to get lesson learned, etcetera, etcetera. And here is the the case for the first iteration, but unfortunately, we are missing some information here. So and here is clearly stated sorry. So this is the query to run. We run vector. As you can see that since we have the strategy agent, we don't have the initial question. Actually, this has been changed and actually it is a dedicated, let's say, sentence that could increase the chances of getting the right documents, and actually we've got 10 documents, but still this is not sufficient because you see current retrieval found the flow measures, blah blah blah, but here you see that it is using some playbook views as we will see later, and it's saying exactly what is missing because of the playbook that we passed. But still, you didn't see how we created this playbook, so ignore for a second. But just keep in mind that the retrieval feedback is driven also by the the guidance that we provided in this case that are dynamic, of course, and we pass through, you know, this first iteration, and we go to the second one in which you see, now we know exactly what is the the document we need, the name, actually, because we asked it for fiscal year 2022. So we have exactly the the the right document, and now the system is going page by page to extract what is necessary. Actually, and thanks to that, we've got what we need and we've got our answer. That is exactly what we were expecting, and the evaluation is, again, done by LLM as a judge, so the answer is correct, complete, follows the playbook. Again, you see that the playbook will come back again and again because it learned how to evaluate. And trust me that this was one of the most complex questions to answer because without this playbook, the threshold was not known and the concept of capital intensive, it was not clear to the basic LLM without proper guidance in this case. But thanks to that, we have got the proper answer. Okay. I have other examples, but I think that we will not have time to go through all of them. But I think that the concept is pretty clear. So I will stop here again for a second just to summarize a bit what we learned so far, and then we will move to playbook that is the other relevant part of this talk. So we've seen that we prefer to have a simpler preprocessing step because the use of tools, the use of these agent doesn't require to have complex structure behind. So we simplified a lot of the preprocessing in favor but on the other side, we have, let's say, a more complex retrieval part that actually compensate a simpler preprocessing, but still it will deliver better quality in the end. So the context graph matches perfectly the structure that we have in the agent. So as a suggestion, if you have a different agent, should do something very similar because this will help you in modeling the graph, but also will help you to understand what's going on and understand and interpret the results so it increases the traceability. Interestingly, and let me stress again, we use the same graph, literally the same instance to store both the documents, so the document graph, and the context stuff and we will see even the playbook will coexist with that for a reason. And tools, I showed you three, but you can plug and play tools so you can add as many as you wish actually. So, okay, now we have the agentic RUG implemented. We have the context that is keeping track of what's happening in the agent, and it's also keeping track of the feedback provided by the human. Okay. So how can we learn from this? So what we have seen so far in this schema is here. In the ACE paper, this part, even though it's done by a simple LLM call, is generally called the generator. So I put it into parentheses because, you know, if you read the paper, you will find this generator mentioned many times. In my case, it's not a simple LLM call, it's an entire agent system, agentic system, but it works exactly in the same way. So we keep track of the decisions. This is the important part. And the reason why I decompose the problem in multiple smaller steps is because I would like to keep track of more and more. I would like you to provide more guidance tailored for specific type of tasks as we will see. So this is where we are. We have an antigenic rug. We have the answer provided to the user that will provide feedback, and everything is now inside our context graph that we have just seen in one example. But if we have time, I can we'll show you more. Then what happens is, in my opinion, very, very cool. So we have two other agents somehow because this is what the ACE paper suggested, and this is actually something that I liked from from this approach. They have two specific and different scopes. So the first one is their reflector. So the reflector actually extracts or creates lesson learned from the context graph. So imagine that you have just completed a question. I mean, you've got the feedback from the user. At that point, you have all the information you need, all the iterations, what worked, what doesn't work, all the reasoning, all the feedback, etcetera, and these are in the context graph. So the reflector extract all these information and create lesser learned. So just describe what has been learned by this execution in a clear way, what pattern worked, what tools was working, and, actually, it also extracted what of the existing playbook items was useful were useful actually to to get to the with result or what was harmful for that. Again, this is extracted from the context graph. At this point, reflector passed the ball to the curator. The name curator comes from the fact that its role is to curate the playbooks. As I said, the playbooks are not growing indefinitely. In this specific case, I have as a fixed size that is the the 30, actually. So I don't want to get to more than 30 playbook items, actually. So what it does is that it takes the output from reflector. It takes the current playbook and actually say reason around them and say, okay. This is what I learned from this. This is what I knew already. Is there something that I can do to improve the quality of the playbook? You know, what should I remove because this was harmful? What I should improve because this can be clearer the next time? What I should merge because maybe these two items can be merged in a single item and make it clear, etcetera, etcetera. So the name creator comes from this. It really improve the quality of the playbook at each iteration. This is what happens here. I would like to make also another point. Actually, you can have as many playbooks as you wish. In the current implementation, the last one that you will find in in the code, there are many playbooks, and you will see a few examples today. Because first of all, what I did was to split between the phases. So I had a playbook for the retrieval phase and the playbook for the answering phase. At the beginning, I had only one, but this didn't work because they had very specific needs so I decided to split them. And I said okay, let's create one for the answering and one for the retrieval. And actually this improved a lot to the final quality. Then I also said okay, maybe that with different type of questions or with the evolution of these tools, I could have different type of needs, so I need a different type of playbooks. So what I did, just an experiment, and actually the results are quite better, I divided these per type of question. So now I have a classifier that will take the current list of classification items, and let's say, type of questions, and decide if the new question can be one of the can be classified as one of the existing. Otherwise, we'll create a new a new version, a new class, actually. And then I have a playbook I mean, actually, two playbooks, one for the retrieval and one for the answering for each of these class. And in this way, as we will see, we have like the class of capital intensity that has specific rules for answering, specific rules for retrieval because actually retrieving specific elements for capital intensive is important because we need specific tables, we need specific elements inside the system. So just to give you the full picture. What happens after is that I have two options here. I can pass the entire playbook. It can be done. You know, you can pass because it's limited. You can keep it even shorter. It will never exceed the size of the context window, so it's fine. But what I noticed is that it can be distracted as I told you about the examples of the Romanian invasion. You know? So what I did as was to include another element that actually will take, according to the type, of course, of the question, the current playbooks and generate five very focused guideline, five for the answering and five for the retrieval. And pass this information that are not generic anymore, but they are again specific for the type of problem that you are going to solve. So you will see in the examples that these are very focused on the type of question because as you see here, it takes guide the the question, the current playbook, and creates these five guidelines for the retrieval and five guidelines for the answer that will be feeded as a part of the prompt as we have the as we have seen and now we will see some some examples. Before doing that, I'd like you to see again how we are extending our graph model to include also this part. So actually, what we have here is that each question is classified in a category, and actually, for this category, we have a set of playbooks playbook items, actually, And actually, we have an entire playbook with the playbook items. And one I mean, these are for the one playbook is for the retrieval and one playbook is for the answer. And these are connected to the specific type of questions, so you always have the right set of items from the playbooks that you need according to the category of these questions, and this is literally traced exactly the same way the question. And what is interesting is that we have we can keep track of the, let's say, the items that were relevant to build the guidance. So what we passed here is also traced back to what were the items that we needed to create this guidance. So this is tracking here relevant for building the guidance, and you see they are both for the the content and the the think. And and then we have also the consequence of the execution of the question. So we know exactly what are the, you know, the items in the playbook that we changed because of these questions. So generated a new one or deleted an old one or updated an existing one. And also everything is traced. And again, here is the power of the graph. You know, we again have another entire area completely traced in this model and that flexibility of the graph allows us to say literally what happened and so that's exactly what we needed from this graph model and everything is here. So with a single graph, we keep track of everything from tracing to answer to feedback to, let's say, playbook. Okay. So let me show you a few examples of the playbook and the guidance to better understand the value of these results. So here are just a few. Again, here we have the question class. We are taking always the same examples just to keep track of the entire process. So if we the three ms capital intensive business has been, let's say, traced as being categorized as question sorry, as a capital intensity assessment. And we have a few interesting section in the retrieval phase. So for example here, we are saying that when extracting GAP, canonical metrics across multiple years, targeted the most recent annual filling with a Doxx copied extract phase, etcetera, etcetera. Other examples are perform one discovery call, do not run-in parallel, do not declare retrieval success when required metrics for the users who requested the calculation, etcetera. So you see that we have a clear, let's say, indication, clear items about how to use it. What is interesting is that even though this has been extracted from specific questions regarding specific years, specific companies, etcetera, I asked explicitly that these items, these playbook entries are absolutely generic. There is no indication about the, let's say, the year, no indication about the the company, and actually these are always generic comments. Let me show you another example from another question class. Again, here we are in the retrieval phase playbook. Again, the playbook here is completely different because it is suggesting something else. It's always picking about a vector retrieval template, include the filling type and reporting period, the plus, the key statement labels. So here is literally telling the strategy agent how to generate the question or the sentence that would be then, you know, embedded and used for searching for the relevant element. So, again, here, we are not searching for inventory. We are searching for consolidated statement of earning. That is what we need for searching for inventories, and inventories, of course, is also is also there. So when doc name is known, issue a page retrieval. As I said, you know, this has been clearly stated, and also the indication about the plus minus one page has been also mentioned. This is for the retrieval. So a few examples from the retrieval, now a few examples from the answer phase. So again, same class as before, but here you can see what we needed from this system. Operation for example, in this case, it is explaining exactly as a domain knowledge what we need for capital intensive computation, CapEx over revenue and CapEx over depreciation and also some threshold indicated explicitly. This is what the system learned. I I I didn't know and I didn't write anywhere in the prompt about this specific thing, but the system learned by mistakes probably because of the first iterations, the system made mistakes, and there are many questions about capillary intensivity. It's not the only one that I showed you. And it learned what is the threshold for the user to consider that business are capillary intensive or not, and what were the, let's say, the values and the computation that it needs to do to get this third shot done. And again, there are also indications for the reflector because these guidelines are provided not only in the process of answering, but also in the process of evaluating the the results. So there are even specific indications coming from the trace that clearly stated must not accept the draft what that compare CapEx with something else. The same is for inventory turnover, same issue as as before. So I have just money your time. Let me complete this part, and then I will run a unique demo for both things. I would like to be sure that we have enough time to comment on the guidelines. So as I said, the playbooks are generic and they are stored as a result of their creation process, and then there is this guideline builder that will take even the question and will customize these playbook items based on the specific question. So you will see the total how this would be totally different because if the first you know, the playbook items were generic with no indication, this will become very specific. So let me give you some concrete some concrete examples. You know, here we have these three m capital intensive. So for the retrieval, so we see that immediately we can recognize the specific years that were not mentioned in the generic one. So before the client complete, no, this is not generic, but for example here, always include a fiscal year token, for example, fiscal year 2022. That is exactly what is coming from the question and this was not there. If we look at other answer related, here we have something that came directly from the items, but also there are some specific elements that are coming up exactly from the question. So among the 30 items available in playbook for answering, they took these five top, and of course, the first one is exactly describing what is required to compute the capital intensive ratio or parameters to get it right. The same is for the other question. In this case, again, in the retrieval, we are indicating explicitly what are the period because it is converting a generic item in a more specific one. And the same is here. So let me show you then a bit the code for the reflector and curator and for the guidance builder. So hold on for a second because I need to go to the other screen. Graph benchmark. Okay. So we are in another area. So here you see the process described. So actually, again, I'm using the Landgraf, as you can guess. There is a lot of question data node that literally will perform the query to extract the history. So here, I mean, you will be able to see it, but the basic idea is exactly that we start from the question that we are processing right now as a next step after the user provided the feedback. And we actually are extracting the entire path. We are extracting all the information we need like for the retrieval strategy we need the feedback, the thinking process, the generation feedback if any. Then we have for retrieval call, we need the tool calls, we need the number of documents returned, we needed also a portion of the chunks. Actually, are not returning the entire chunks, but just a small one. As you can see here, there is a substring somewhere. Yes, here. So we have just the initial part. Excel. So we are passing a lot of information, and we are doing the same for the retrieval data. So here is the retrieval part and for the answering part. If I go back here, then I'm loading the answering question the answering details, and again, here we have the generate answer step and the other step in which, again, we keep track of the thinking, the reflection feedback, and the answer actually provided by the system. So again, we are treating these as a two parallel processes. So let's see what happens with this data. This is converted in JSON and passed the prompt. So let's see an example of the prompt that will happen, for example, here in the refractor. Let me show you the quickly the the prompt. So here. So the first thing that we are doing, because this improved quite a lot of the quality, is to pass the the final signal. So I said, okay. This worked. This didn't. And this passed as a first thing because, as you know, LLMs tend to forget what is in the middle. So as a first thing and the last thing, we generally try to put what is very relevant. The rest also is relevant but not as part that you put at the beginning and at the end. So you're an expert to review our strategies and your goal is literally to extract, to carefully analyze the full iteration trace and abstract what worked and what doesn't with particular regard with the term that generated the results and what didn't because this is exactly what we needed. The input is the question, the human feedback, the agent final answer, and the playbook IDs oh, sorry. This is the Corrado the reflector. Yes. The playbook ID because we we will need to keep track of what guidance worked and what didn't. So, actually, we are passing these and say, okay. This is what happened. This is what we passed to the system. Tell me what worked and what doesn't in terms of what guidance was very useful, what instead was harmful for this process because in the thinking, it's explicitly said that I use this playbook suggestion to do this. So if it's wrong, it will be marked as harmful, etcetera. Then we have the curator. As you notice here, we are passing the entire retrieval playbook. If before we are passing just a portion of it, here we are passing the entire the current playbook, and the role is to keep the, you know, the the number of items below the 30 entries. And here are the the instruction that you would be able to to check within the the system. Then just quickly, the the retriever that is the node the part that generates the the guidance. So let me show you the the retrieval playbook, for example. No. Sorry. This is the playbook. So the this is loading. And at some point, here we go. So this is the playbook guidance prompt template. In this case, here I'm asking specifically to come up with five clear guidance item that then can be used by the system to accomplish the retrieval part or the answering part. And the result of this that is actually what is returned is exactly what will be passed to the, let's say, to the agents inside the agentic rack. And as you can see here, we have the playbook. We have the query actually, and some specific guidance that are specific because we are using a single prompt for both answering and retrieval. So here, there is some differences between what are the retrieval, let's say, critical rules and what are the answering the critical rules that are used for the entire process. So with that in mind, I will like to come back to the few examples to because we saved some time now, and it's time for me to show you a few examples. So let me share the screen here. So here we are back. So let me show you a few more examples of what happened. For example, coming back, I mean, to a different scenario, we have here the same question done twice. And the reason why I wanted to show you this is because it's not only about the quality of the answer. It's also about the entire process. In this case, specifically, what happened is that in the first iteration, that is this one, we've got a wrong, let's say, a wrong answer. So let me show you the the full process here. So we are You need you need to switch. We still see the demo in the call. Right? Yeah. you. Thank you. I. will never actually, these are the question is this one. So who are the primary customers of Boeing as of fiscal year twenty twenty two? Then we have this one that was run almost at the early stages without a proper guidance, and actually it failed. So I was gonna show you the the process. So this is the question. This is the answer. And as you can see, the answer was wrong. And the interesting thing is that here, we see the prompt. So, you know, we see what I showed you, and, actually, we have no guidance for the retrieval, and we have no guidance for the for the answer. Let me show you. Sorry. Prompt. Blah blah blah. Okay. This is the the the content of the text provided, and there is no guidance as you can see. Here, there are no guidance, and that's why in the end, the answer was wrong because it didn't help to find what was necessary. In the second iteration, which was longer because I needed more information to answer properly to the question, but we have got a proper answer, as you can see from here, because the previous answer was wrong because it didn't count on the government part, actually. So let me show you again the reason why it failed. So here, you see the reason why it failed a bit but omits the key goal, the fact that the US government accounted for almost 40% of Boeing on a revenue. So this was why the answer was wrong. So partially, it was correct. The rest was fine, but this part was missing. So I'm expecting that the guidance provided here will help to do better, and now you see the guidance. And here you see a specific indication about Boeing. Use vector retrieval as the first one with a short semantic discovery query. Example, measure customer's disclosure in annual filling, etcetera, etcetera, for precise document swabbed, text search, apply Lucene, etcetera. So you see that the first call exactly as indicated by the guidance was a vector with disclosure primary customers. You you see, measure customer disclosure. So actually this was exactly what the guidance suggested here. And actually, regardless of these, the retrieval was not successful because there's some other information is missing. So here, again, we are saying that it's missing customer concentration. That was the reason why the previous version failed. So let's see now what is the reasoning in the retrieval strategy again. So here we have you know, that is specific regarding customer concentration is missing because this is what the retrieval feedback came from. And, again, here we have no major indication because of the feedback is enough. We are using text search retrieval this time with specific measure customer, one customer accounted for, etcetera, etcetera. Again, this was not enough and as you can expect, the system switched to page retrieval. So now it's getting the first 40 pages almost and got exactly what it needed in this case, and now the answer was correct. Again, here in the prompt, we can see that the let me show you the guidance from playbook is present here, and this is taken into account both in the answering phase and in the thinking around the the evaluator. And blah blah blah. I guess I got to go to identification, etcetera. So, actually, this was correct because, you know, it answered all the the items required for the evaluator, and this is why it was considered as final, and then it is actually final and correct because of this. Okay. So a few other things that I would like to show you, and then I think that we have some space for questions, is an example of the retrieval playbook. For example, for whatever turnover ratio, and the fact that we are keeping track of how much helpful some of these pitfall or pitfalls or strategic reviews are. So So you see, for example, these content was helpful six times and natural one time. We have some harmful one. This one was helpful seven times, five times this one, etcetera. So actually, these strategies and views and tools call were actually useful in many different occasions, actually. Sometimes they are also harmful, to be honest. I checked for a few of them, and they were harmful. Okay. So with that in mind, let me switch to the final part of the presentation very quickly. I run a few a few tests, actually, and the one was running while we were speaking, so I will maybe update the the presentation after this talk before giving to you. You can see few interesting things. So first of all, this was initial quality, so we were able to answer to only one hundred and fifth seventeenth questions, actually, correctly. So with a percentage of 68.5% with the measure issues happening here, you know, for the multidocument because one shot RAG cannot do this, and this for the domain relevant. As soon as we introduce the vanilla agentic RUG, we immediately got a jump. So we actually jumped to 74% overall, but what is interesting is that we improved the a bit of the domain relevant type of questions, but in particular, as expected, we improved the multi documented type of questions part. And once we introduce the agentic, the learning, let's say, loop, actually, we've got a dramatic improvement in the quality for domain relevant because, of course, we are learning by user feedback, and also we improved a bit in the multidocument. Overall, consider that the quality was already quite high. So what we improved significantly was, you know, these 72 that became 84 and these 54 that became 64. So we've got quite a, you know, a big improvement, but we are still evaluating. And as I said, I'm speaking, there are still some tests running for completing this view because I'm testing after one, two, three more iterations with some changes in the questions, etcetera, etcetera. So with that, it's time to close this workshop. So what are the key takeaways that I would like you to bring home after this presentation? So first of all, the role of AgenTeq AI systems in the generative AI space because they are literally shaping the future of generative AI as we have seen, and we've seen just an example of agentic rug, but in reality, there are tons of other use cases of co generations accomplishing complex tasks, etcetera. So definitely, the the world is now agentic for sure. We've seen our memory and learning on top of memory are key features to improve AI capabilities and improve the traceability, of course, in particular for for memory. We have seen the example of a GingerGauff that can deal with a very complex fashion and early results are definitely very promising. And what emerged also from this conversation in a few examples that I showed you is that having the graph offered the flexibility that I needed for keeping track of the document, I mean for storing the documents, for keeping track of the context graph, so the decisions and the outcome, storing the playbook, the guidance, etcetera, etcetera. So actually, the flexibility of the nodes and relationship model helped me to have everything in a single data source. And actually, Neo four j in particular helped on the technical side because it offered a Cypher query language, of course, for structured type of query like pager, but also others that are not mentioned in this talk. The vector index for searching, I mean, for semantic type of queries, and the full text search help that for, let's say, item based or keyword based type of search. And as I mentioned, the edit tests are very promising. I will share more results later on. And but what I would like also you to remember is that all these is a complex infrastructure. You know? You know, the tests that I'm running are taking almost twenty four hours to complete, you know, because of all the iterations and such. So it's a complex scenario. So you need to have a complex problem with a high value, high stack to justify the complexity of this implementation. So don't get a bazooka for killing a mosquito. Of course, you need really to use the right tool for the for the right problem. And with this sentence, I would like to thank you for listening today, and I'd love to hear some of your questions. We have a few a few minutes, actually. Super amazing session. Alessandro, thank you very much. You got also high praise in the chat, so really, really appreciate your your time and and the lot the work that went into this workshop today. It's very. it's very was a lot. But it was very very very precise, a very good overview, and it also went in into the into the depth of of how how it's done and and and what you can recommend as as somebody that built this yourself. So I was I was very impressed. The chat, know, reflects that as as I can see right now. It's lots of thumbs ups and and thank you. So really good. But like I said, we have a couple. I wanted to wanted to dive in and and see if we can cover a couple. One comes about the curator and, you know, if the curator won't mess up your whole pipeline when he determines something, revising a playbook, create new ones, changing data, there's always the issue breaking the playbook by adding something new or is possible to, how does retest work with that? How do they manage with with this this this this potential issue of of, you know, improving something that breaks the nineteen ninety. use case. well, definitely, let me say two things. First of all, you can definitely explore other type of models and extend these models by, you know, keeping track of the history. In this version, I also keep track of the entire history. So whenever there is a change in something, I'm keeping the old version. This is. still connected with the questions that were you where this guidance or this playbook was useful. So this is always phased. So the entire history is there with all the details. So you can always come back and say, for this question I use this playbook, why this is not working? So you can always run the postmortem analysis if you like. Also, the second aspect that you need to consider is that the prompt is structured so that the changes are always justified or are always limited. So what I'm asking to the creator is to take into account the current playbook and decide what really needs to change. Not I mean, I'm forcing it not to to make a stupid changes in in the structure or the of the sentence, etcetera, but something that concretely he has, let's say, a very verifiable reason why these needed to change. Of course, this won't solve the problem, but the combination of the full history for a postmortem analysis and the, let's say, proper prompting for the creator can help it. Plus the fact that you keep isolation between answering and retrieval and between different type of questions improves the, you know, the safety of these overall scenario because, of course, even though these will affect a specific type of questions, all the others will not be affected. So this isolation will help you a lot in this sense. Okay. Cool. Another question I saw from a couple of people, either in chat, also in the q and a functions about, you know, how how do they cope with the playbooks, but, you know, generally, the pipeline. How how how do they cope with dynamically changing environment? So, you know, imagine you have a supply chain and then, you know, one supplier is not available anymore or routes change through, you know, reasons. And then all of a sudden, the the whole you know, everything that that used to work by now is no longer working because of of sudden sudden variations and and and changes in in flow of, you know, let's let's say with supply chain management here, transportation issues or something like that. How, how would. that be, reflected? consider that these playbook is evolving over time. So I'm. expecting that if there is a major disruption in the, let's say, in the context in which this agent operates, I'm expecting that at the beginning, there will be some issues because it assumes something that is not real anymore. But I noticed that the capability to recover is quite, let's say, faster. Let me say that based on my experience, I noticed that the system learns more from failures than from, you know, right path because it learns exactly. what doesn't because the fact that something works doesn't mean that it will forever or for every case. But if something doesn't work, actually is a very good indication. So what I noticed is that the system is capable to learn from it quite quite well. Yep. And then maybe maybe another point that's interesting in in this point is, you know, obviously, made your playbooks as generic as possible. So and through the power of the graphs, you you have the the possibility to say, okay. I 'm sticking with the supply chain management. I want to know from, you know, location a to location zed, what's the best way. And then through changes in the graph, maybe I know, okay. This this route is no longer possible, then another path becomes available or becomes the next preferred way because it may be might might have been not the preferred way because it was slower, but because the other was no long is no longer available, then the other one switches in. So we have the the dynamic concept and the the interconnectability there. You you don't loop you know, this don't doesn't go away. This still will be available and then dynamically can can make it work. Yeah. Well, it also depends on the type of tools and how much, let's say, flexibility you provide to. the to the agent or to the main agent. You know? Because in theory, the problem that you just mentioned can be solved by the agent flexibility to understand that there is a better path that suddenly, you know, jump out from the test. So the guidance. should be very generic in some way, you know, and provide enough context but never, you know, force the agent to avoid to explore more or at least have some some fault mechanisms that will help, you know, to also explore other ways after some appear to be not available anymore or just to explore other options. And let's maybe do one or two more questions. Here's one from Chris, and I think I saw this from other people's as well, so that's why I wanted to to touch about this. And Chris has are the playbooks similar to skills that can be updated by the agent? Well, that's definitely something that we we could explore. You know? The concept is very is very similar. The idea of the playbooks here and the name comes from this concept that they are, let's say, dynamically evolving, you know, inside the system. The skills somehow are made for being shared between, you know, a provider of the skill and the and another company. But that's a good idea. You know, you can easily convert the playbook that you created by using the system as a in a skill that you can then translate or move to another scenario or to another agent, for example, if, for example, there are some commonalities. That's absolutely a good a good idea. But in my mind, the skills are generic and across, let's say, multiple agents. Playbooking, they are generated as a result of the experience, direct experience of the agent. So the agent and also consider that the playbook matches very well with the prompt. So they understand how the that's why I'm passing the prompt because I would like. the system to say, okay. Yep. This is the prompt where you fit this piece of playbook. You know? So it consider also what is before and after. So it's very, very specific for that agent. Skills are made for genetic reasons, for genetic purposes, so without considering the actual history or the experience made by the system. Cool. And then just close it out. One more question from Ayala. The the full pipeline latency, is that acceptable for a chat experience, or do you think this is more like a research kind of a thing you let around a little bit in the background and you don't know real time or close to real time answers given? Yes. Absolutely. This is not for real time answers. You know, these are full process that takes minutes sometimes. What I can suggest, and this is something that I didn't add because I I could miss the, you know, the real improvement is that I would like to give it a sort of a a quick shot. Because with the proper embedding and proper, let's say, let's say, approach to the preprocessing, you can get these questions to be answered very quickly. So as you've seen for 117 over 200, that is more than 50%, you don't need all this complexity. You know? So in theory, what we can do is that you can have either a system that will see the question and immediately decide, okay, let's go for the simple solution first. And maybe it will answer in a few seconds because it goes. for the. simple one shot. or then we say, no, no, no. This is a complex question. It require more time. So in this way, you can balance between fast way and the slowest way. I don't know if you remember, I mean, but in many of the current tools, you have a sentence coming out sometimes. This is difficult, so let me think through it through. You know? And I think that this is exactly what they do. If the answer cannot be provided immediately, they switch to a different reasoning model, and they will give you a better answer, but it will take more time. So they ask you to be patient. We can do the same. Yep. Cool. Alright. Super. Lots more question, but let's let's continue the conversations elsewhere. Thank you very much again for for watching today. Thank you for for your questions, your your contributions throughout the day. Thank you very much, Alessandro, first last and foremost, you. obviously, for for this amazing workshop. If you have, you know, any questions, people can can reach you as you shared in in the at the end of the presentation. Yep. You can also you know, you'll get an email with the video link. You'll an email with the GitHub and the slides. So you can also do do some some commenting there on on this part, and we hope to to get that out over to you before before the weekend so you will you'll see see that coming in. And just briefly before everybody goes away, I wanted to share that next stream is taking place tomorrow, actually, with Luan who is talking about Nodes AI agenda before Aura Agents. So we have just launched Aura Agents a couple of weeks ago, and we will build a AI agentic companion that runs on Aura that helps you structure the the Nodes AI agenda that is taking place on April 15, as you all know. So if you're interested in that, join us tomorrow in on YouTube or on LinkedIn. And, yeah, Nodes AI is coming up. This was the last road to Nodes workshop. I hope to see you all on the April 15 to discuss more AI and graphs on a full day with lots of interesting sessions, lots of interesting tracks, conversations here about use cases, adapt applications from Neo four j folks, from community, but also from, you know, implementers that build stuff with with Neo four j and and AI. So looking forward to see you in the fifteenth. And, yeah, until then, have a good time. See you soon. Join the community. Join the Graph Academy or run an Neo four j or a free instance if you like to get involved yourself. And I see lots of people who really want to dive into this, Alessandro. So we'll we'll definitely provide the material as soon as we can. Yep. Yeah. And with. that, I will do it shortly. cool. Thank you. Thank you very much. Thank you, Alessandro. It was a pleasure. It was an amazing session, a great finale to. a great series this this season. And, yeah, see you soon next time. Take care, everybody, and yeah. Bye bye. Thank you. Bye bye.