Name: Agents in the Wild: Building AI That Learns
Uploaded: 2026-10-29T19:16:13.686Z
Duration: 1 h 59 min 12 s
Description: Agents in the Wild: Building AI That Learns

Transcript for "Agents in the Wild: Building AI That Learns": Okay. Good morning, good afternoon, good evening, everybody. My name is Adam Cowley. I'm filling in for Alexander who is, otherwise engaged today. Filling in, you in... Filling in, sorry, for the, this session, on our road to nodes program. So road to nodes is our, lead up to our developer conference, nodes twenty twenty six, which takes place on the... On November 12, this year. So this is session number, six of session... Six of seven, sorry, where we will be talking with, with Alessandro Negro, chief scientist at GraphAware. So I guess without further ado, Alessandro, happy to hand it over to you. Thank you, Adam. Let me start, sharing my screen. But, you sure? Okay. Cool. Okay. So, hi, everyone. In this talk, we are going to speak about building generative AI systems that learn while they are being used. I don't know if you've seen my previous talk for road to nodes, AI 2026, because this is a continuation of that, of that work. If you didn't, don't worry because I will provide you with a quick recap because, this work is literally taking from, what you have seen in the previous talk and extend it quite heavily to answer even, more complex question. But, in this case, rug is not the the main topic of this talk because we will instead, focus on, how, you know, a complex system like a, agentic rug can be improved over time by leveraging, memory, so agentic memory, and, the way in which this memory can be converted in lesson learned that could improve the execution, next time. In this context, graph technologies and Neo four j in particular will play two roles. One that will be the the main repository for the knowledge on top of which, the agent will provide, answers, and also will store the the the trace of the agent execution as well as the, the playbook, so the lessons, learned. So as last time, this, will be a very, practical, workshop. So I will mix, theory with practical examples, and, we will see a working version of, what I'm going to present in a Vibe Coded application. And then we will use, Yoom, which is our analytics platform built on top of Neo4j, to explore the, the graph. So really, I like this session to be interactive, so feel free to, interrupt me whenever, you would like to ask me, any, any question. I will check time to time to see if there is any question here in that, in the screen or if you like to ask, something in particular. So let's say that, even though a few months passed, from, my last talk for Rhodes, many things changed in the meantime because, GraphAware has been acquired by Neo4j, so we merged our efforts in promoting graph technologies, so my title changed it from chief scientist, for GraphAware to senior director, for GraphAware product. My day to day job remained the same. My passion for promoting, AI and the knowledge graphs remained the same, but, the title changed. And, actually, last time I was promoting my, fresher printer, book knowledge graphs and LLM in action. But in the meantime, with the valid, we brought in, a MIP status, another book, that is a building, specialized AI systems, that, has, as one of the topic, definitely the way in which we can customize the behavior of the AI agentic system via, you know, lesson learned as we will see today. And of course, with fine tuning. So let's, let's start this talk by the end, by talking about what's the the end, you know, what is the the final purpose and why learning matter? In order to do so, let's discuss about the context in which this is happening. First of all, let's say that agent are becoming more and more, complex. They are also becoming part of our, day to day, work. They, now can, decide, they can retrieve data, evaluate results. And, real solutions now are not based on a single agent, but in reality, they combine multiple agents interacting to each other with different prompts and, leveraging different tools to achieve their, their scope. But in order to improve this application, in order also to make them, more transparent, it's important to keep track of the, decisions taken by these agents, for transparency, as I said, but also to make them more reproducible and for debugging purposes. Since we have to do this in any case, why not using these traces for learning purposes, and using it to abstract what worked and what didn't work, and get lessons that can be applied to improve the full process or as small, as a small task. Let's say that now, this is inserted, this concept of improving is inserted in the current different approaches that are jumping out every day. Time ago, we were hearing about context engineering, RNS engineering, recently loop engineering, and graph engineering. And probably there will be a star engineering, coming out quite, quite soon. Let's say that in this context, where all these solutions are trying to improve the process, you know, to improve the way which these agents are applied even to complex, scenarios like, coding or, decision, taking. This approach that you will see today is, inserted in the context, of context engineering. Because, what we would like you to do is to pass more information to the prompt, that our guidance is, as we will see, to improve the way in which the answers are provided to the end users. So supposing that, everything will work as expected. So what we will get from this learning? So first of all, we will get, our agents to behave better, you know, to improve, automatically without, too much human, interaction. Or when I say human here, I mean, the developer side. So we would like to develop, solutions that can learn and adapt according to the, the needs of the... Of each user or each customer. And this can happen because of specific needs of the user or specific needs of the, the specific domain or the the technology or the, the terminology in a specific domain. And the approach that, we will take, as I mentioned, will be not to fine tune, open source model. We don't want to do, in this specific case, to change the the weights of these, models, but we would like to keep them as they are and, instead adjust the way which they behave by passing the guidances, by passing, you know, more context in order to get a different behavior without changing, this for the reasons that you can imagine in terms of, simplicity and adaptability. And of course, the fact that each lesson would be, as we will see, in a plain language, clearly this makes this solution way more, explainable. That, in certain domains like law enforcement, for example, or the banking sector is fundamental to provide, such solutions to the end users or to, the, to the customers. And, let me also say that this work that you're going to see is inserted in a broader effort in Neo4j of providing, agentic memory that, actually keep track of the decisions taken by the system. It's inserted in the broader concept of the context graph, you know, that we are promoting as a way for modeling, not only the work or the decisions taken by the agents, but also decision taken by humans, you know, modeling processes, etcetera. And, we are working in that sense to distill knowledge in the form of skills or in the form of guidances as we will see, today. So let's recap, a bit of what we've seen, last time at, Node, 2026, Node Zei 2026, actually. So, at the time, the goal was, a bit simpler, let's say. We would like to find a good setup for answering the complex questions. We wanted to, scan different approaches to, RAGs or retrieval augmented generation, to learn how they works, how, when to use one and one... Use another one, and why one is better than, than than another. And in that specific, presentation, I went through the... My personal journey from one short run that we will see in a second to, a custom or complex Agentica, rug. So, just, again, to set the ground and to give you all the information that you needed to properly understand what we are going to present, let me quickly explain what is the simplest version of a system, what I call the one shot rug. In this system, actually what happens is that the user asks a question, then this question is converted in a vector, so it's embedded. And then this embedding is compared with chunks of different text or different documents that have been already preprocessed using the same embedding strategy. And in this way, by using similarity functions like, etcetera, you can find those chance that somehow are closer to the specific question from the user. Then this chance in, let's say the top k, that could be 10 or 20 or more, depending on the context window you have, are passed to the, LLM together with a question and together with a bit of prompt, and the system will provide an answer. This solution is simple and actually it works for simple question, but it has several, drawbacks. So first of all, there are many questions in particular in the, type of questions that we were analyzing last time. You know, there are questions like, is a three ms capital intensive? Where a capital intensive is not probably available in any of the texts available, because it requires a better understanding of what is necessary to compute to the capital intensive, I mean, characteristic of a company and then, extract the value. So this will never match with any of the chunks, available. So embedded is not always useful to find the relevant information to provide an answer. And, because it it requires somehow to to match a text or to find something that is close in terms of the semantics. And, also, it doesn't have, as you can see from the simple schema, a second chance. So The system is linear. I have the question, I embed it to find the closest chunk, then I generate the answer. In this case, it can go well or not, you have no chance. You can't navigate, again through multiple documents, or you can't adapt for sure. And, clearly, passing the full of text is not an option. Probably, you know, you have customers, but I definitely have customers having tons of documents and passing all of them to provide an answer will be literally impossible. Even though these LLMs are getting better and better in accepting larger windows, of course, there will be never the option to pass the entire, and it will be in any case very expensive. So these static retrieval has been proved, to be, simple, effective only in a limited amount of cases, and definitely can be improved. So in the last presentation, I went through the the multiple steps, but, the final result, is this one, where, what we have done is to disassemble the simple process in multiple steps. In particular, we isolated the two phases. One is the retrieval phase and the other one is the answering phase. So the purpose of the retrieval phase as a whole is to, retrieve the relevant information from the documents available and provide, a knowledge, and then the knowledge you needed to answer to the next phase. With the task, it's just to provide an answer, a final answer to the, to the end user. So in particular, the retrieval phase is a further split in, let's say, in three major steps, in this, in our implementation, of course. This is not generic. This is very, customer. This is our solution to the problem. So the retrieval strategy agent has the role to identify what are the, the tools to call and how to call them. Because in this case, what we have done was to provide not just a simple vector similarity, but we provided also, queries, so queries and the Cypher queries against the graph. Because what we have done was to take all the documents, chunk them, and store them in Neo4j. Neo4j offers a different type of indexes you know, to access this data properly. So vector, if you like to use embedding strategies, Lucene, if you like to do, textual search, and of course, you can use Cypher to go, for example, page by page or to navigate, a specific area of the, of the graph. So, we provided all these, tools with the description to the, retrieval strategy agent, whose role is to decide which one to call and how to call, them. So what is the question to pass or to the the query, you know, to ask through Lucene. Then there is another step that is the evaluation and knowledge building agent that takes that... The the the chance, returned by the... All the calls after they are, of course, duplicated. And, it verifies. First of all, if the data, the information passed, is enough to answer the question, And, as a second responsibility, it creates, what I call a knowledge, a growing knowledge, which is a list of sentences of parts or relevant data that is, necessary to answer the question. So that, to the next phase, we won't pass the entire set of documents, but we will pass only the, the relevant, sentences with the, with the sources just mentioned so that we can keep, you know, track of, from where these sentences are coming because it's important for traceability. At this point, the evaluator can decide if, you know, the information is enough, and then we will move to the next phases. Otherwise, can come back to the retrieval strategy agent passing some feedback specifying what is missing to answer the question. And, immediately, we see a bit of advantage here because we have this loop. So it's not necessary to go right to the first iteration or to go to complete in the first iteration, but we can iterate, again and again. But supposing that the information at the end of this process is enough, the, governing knowledge contains all the information necessary to answer the question, that, there is the second phase where we have again here two elements. One is responsible to answer the question. The second one is responsible to evaluate the answer provided against some criteria, like traceability, verifies the computation is correct, etcetera, etcetera. Here we have three options then. If the question is correct, complete, and satisfies all of the criteria, we will pass it back to the user. Otherwise, if there are some mistakes in the computation or there is some issues in the way in which the the sentences are structured, we, the system comes back to the, answer generation, providing some feedback, so that it can improve. And, if some information is missing, then it will pass back to the retrieval strategy agent with the, feedback again to provide, better information to answer the question. Okay, so before moving up to the next slide, I'd like to clarify a few important points that I noticed are very relevant. And I would like really you to get this. Because in theory, all these can be modeled as a as a single or a reactor agent. You pass the tools, you pass the question, and the system, in theory, will be able to answer to any question. This is what we learned from from the books, you know, from the papers, etcetera. Well, actually, using, or decomposing, the the the full process in this way, so we have literally five different agents instead of one, has, several advantages and some disadvantages that we will speak about. So the first advantage is that, I can have, first of all, different prompt for different, roles. So I can be very specific with the strategic, agent on what I need and how to call each, each tool. In the evaluation and knowledge building, I can be again with a prompt specific about what's the role and the same for the others. So this specialization, it's something that, pays off quite well because, the agent will focus on specific tasks. So the strategic agent will focus on the best way to call the agent, on the best agent, sorry, the best tool to call and, the best way to call these tools. So passing not exactly the question, for example, but passing a different type of question, etcetera, etcetera. That's one great advantage. The second advantage, which is not a small advantage, is that since we have four, in this case, distinct, type of calls, we can have four different models. So for example, for the strategic agent, I can use a model that is specialized in the reasoning. While of course, the answer generator should be specialized in generating text. So, this is the way in which we can actually, provide different type of of specialization, which means that, you can save money, which is not a small thing, or you can, in the case in which you would like to use some local models, call specific models that are so, that are very specialized, and they probably can be better suited to answer or to accomplish the task, which means that the system can be migrated to use not big LLMs, but small LLMs without too much effort because of the specialization of these tools. This was what we created and there was a first complex, agentic system for which we decided to start, somehow, reasoning about, how can we make it better, after a few iterations. So in the previous presentation, what I did was to just introduce this idea of, keeping track of every single decision taken by each of the agent that you've seen and see how we can improve the next time. So we created these context graph made by the traces of the execution. From these, we distilled some playbooks and we use this playbook to provide the guidance. This was the idea that I just provided very quickly last time. And this is what we concluded, let's say, my talk there. So to clarify, in that scenario, the goals were different. You know, first of all, the main goal was really to find the best rug setup for complex, question, you know, and we, started with one shot and, landed in the agentic, solution. We tested, as I just said, this idea of the context graph, you know, so I fine tune, for example, the prompt of forcing it to provide always the reasoning. We started understanding what kind of information we needed for, providing enough information to generate proper playbooks. We noticed that we can adapt a day to day behavior and improve the execution in terms of number of iterations, for example, and the quality to the final answer via a few, of these suggestions. And, we had a specific simple case there. We had just, PDFs, of... That are, 10 k filling from different companies, and it worked pretty well. So in terms of results, and what we've done after that, so this GenTICARAG was stable enough and was tested enough so that now it empowers, our, u Maestro feature that is a sort of document intelligence that allow our users to upload the documents exactly as I was doing in the demo last time and ask any type of question. So it works perfectly well and it will be released in the next version of the of the home. So immediately, we were able to convert that experience into something concrete, and if today we will have time, I will show you how it works in practice. We prove that the context graph provides enough information to build playbooks, to build guidance apart from full traceability, and at the same time, we found the limitations. So let's go through these limitations first, and then we will start discussing how these can be addressed in this second iteration. Let me say, this is, again, not the final version, you know, this is, again, another step in the process of learning and applying this knowledge to real concrete, options. But as you will see at the end, you know, we discovered interesting, things that we would like you to apply immediately, but still, the journey is long and we would like to achieve a way higher, goals here. So in that case, we had a very limited data. You know, we were focusing only on 10 k's, that definitely are not enough to take real decisions, in particular in the investment space. The setup was a tedious, I'd say, because you have to download first the PDF, so you needed to know upfront what type of questions you would like to ask, download the PDF, let them be processed using the doc link, and then you can start asking question. The reasoning was limited because it's true that we were using the finance bench dataset as a scenario, as a use case that has a very linear questions. Interesting question, it was a standard, benchmark that we can... We use it to compare different, solutions even against other people doing the same. So it was definitely very useful, but the type of questions were quite limited and, doesn't justify, for example, the the complexity of an energetic system that learns by, by doing. And we didn't have an interface, so my presentation last time was quite dry because I used just a human to navigate through the graph, but it was not interactive. So from this, what happened was that I had a a a bigger dream. You know? So the the idea was to, create a real financial advisor, and we were quite far from that. So that's why I, you know, extended my dream. And this is just to be clear, connected with the content of my second book about knowledge graphs and LLM. Because along the entire book, what we were trying to do there was to build intelligent advisors, another system, that somehow combine knowledge graphs as a knowledge repository and LLMs for providing guidance, you know, providing the concrete help in a complex, scenarios. So on that idea, I came out with, another type of application that could benefit this combination of, graphs, in particular knowledge graphs and LLMs. And it, it is a financial, let's say investment advisor. You know, I think that many of you, time to time, think about investing or they are doing already in stocks, etcetera, and always you have the, you know, the, the questions for yourself about, okay, is this company reliable enough? Or what are the... What is the management doing? Etcetera, etcetera. So based on this clear idea, what I wanted to do was to create this project that I called, but I think that I should change the name, make it right. So that means exactly make your decision in terms of investment right. And the basic principle is that I would like to start by, you know, by asking the user what kind of investment he or she would like to do, the vertical, if it is a long term or short term, what is the company size, etcetera. And the system will provide a list of candidates from which the user can decide. And this list is created by, you know, querying different ETF providers, so they list all companies they invest in, can access to Warren Buffett company or letter to investor, etcetera, access the news and provide a set, the user select the company that he or she would like to invest in, then start the reasoning. And, in the end, get a recommendation. So this was the high level characteristic. Again, I'm not one to... I don't want to sell you the idea, but just trying to give you a broader understanding of how these complex system can find their natural place. So basically, the big picture, the high level architecture is represented here in the, you know, in this slide, in this schema. So we have the discovery, phase where, you know, the... There is this interaction between the user and the the system just with the purpose of identifying a single candidate. Then, there is, this phase that I call a diligent diligence that is a mix of research and analysis that produces a company card, like a baseball card, literally, you know, with all the numbers about the company, but also, and, this is where we are going, a list of answer to very, very complex questions. That is at the core of this implementation. I mean, the rest is available elsewhere, not, I mean, of course, combined here, but the real goal was to get there, to get to the point at which I needed to describe a company, from different angles, you know? So I, since I read many, many books on how to manage companies, you know, like, Good to Greet from Jim Collins or, you know, a bad strategy and good strategy, etcetera. All of them describes what are the best practices for managing complex companies and how to succeed in competitive markets. So what I did was to take all these ideas and condense them in a set of questions, a question back. And I noticed that answering these questions was exactly the type of complexity that I needed for, you know, challenge my, agentic rug approach. And of course, the idea was even to understand how... The way which we answer can affect the the quality of our decisions. Because once you build these, company card, what happens is that, we, have these, set of agents, actually two agents. One is, bullish, let's say, the other one is bearish that debate to each other about, okay, the pros and cons of buying stocks for that company, using the company card as a reference. And there is a really dichotomy between them. One going in one direction, the other one going in the other direction. And then, of course, there is an arbiter that will just take the debate transcript and make the final decisions on what is the best solution, then the reporter will create a report with a company card, the discussion about the debates, and the final recommendation that ends in the ends of the users that can just take a decision. This is the complex part, but we won't speak about all the details. Just to give you enough information, at the end of the presentation that I will share with you after this call... This, workshop, there are a list of references that you can use if you like to, you know, investigate these ideas that I presented here. Now there are a few papers that I will highly recommend if you like to enter in this, in this topic. But, where we will focus is exactly the diligence research and analyst that produces this company card. And even in more detail, we will focus on the, the way which we can provide answers to such complex questions. So the architecture, you know, the entire system, is an extension of what we have seen, before. Of course, Neo four j remains the the main database, the main graph database for the knowledge core where, for example, we reprocess and store the feelings, the various feelings, etcetera, that in this case, won't be stored or downloaded upfront. As we will see, there is now a system that downloads everything, quite automatically. And, at the same time, Neo four j, keeps track of the, the decisions from the agent and the the playbook that results from these decisions. Of course, since, we needed, to compute a proper baseball card for the company, we needed also some, some, I mean, numbers, actually. I included, Sharadar, for, fundamental numbers, and I'm using inducted b plus parquet files for storing the numbers because it will be a waste, storing them as well because there are so many numbers. And to be honest, the access via Adapted b, on top of Parquet is the best solution for the for the job. But of course, all the rest, you know, since we are using, Seca, rather, etcetera, will be, or news or fund holdings. Everything else will be stored in Neo four j, properly organized and stored. For the agents, we will use, several, LLMs providers. And whenever it will come, the moment I will describe what are the, the different options that we use and where we use what. LandGraph remains our tool for orchestration, for combining these different agents. And of course, Yoom, on top of Neo four j, happened to me quite a lot in exploring and navigating the graph while it was composing. And the the web coded application has been developed in the... In Python. Okay. So I think it's a moment for stopping for a second if there is any, any question that, jump in the meantime. So let me just check. I see many, many questions here. Have you considered the rearing, the intent in retrieval phase? Well, I I definitely think that this is, currently a part of the process. As we will see, there is a a component that, actually decides the type of question, you know, if it is just numbers, if it is, a narrative, etcetera. So this is, let's say, somehow intrinsic in the process. It's not called the intent detection anymore, but definitely the retrieval phase and another phase that you will see that is the, the planner takes this into, into account. Okay. So another question is... Sorry if you already said, but how many agents altogether in the agentic rank? Okay. We will see it because, there are several agents combined, that, in the final version are way more than what we have seen now in the in the previous version. So we extended it to quite, quite a lot. Okay. But I think that's pretty much it. Calling the... Oh, this is interesting. Calling all the tools consumers' tokens. Although you use a progressive disclosure to only to only... A bit more. To only call those tools so that are, needed. This is also another interesting questions. Well, consider that, we are not calling all the tools, but the strategic agent is able to select the tools that are necessary. And definitely, this is a discovery process. So what I noticed by following many, the executions is that, in this loop, the the strategic agent is, together with the, evaluator, is able to find good strategies. Probably not the first one, that's why the guidance can help. But definitely after one iteration, immediately recognize the type of documents to extract, how to extract the relevant information, etcetera. And I have to say that recently, the quality of the, for example, or the question translation and the Lucene query is way higher very recently. Okay. Let me see if there is a... I have a question. Either there will be a recording. Yes. There will be a recording, and then everybody registered for this talk will receive an email with a, with a link at some point. But it... In any case, it will be available as a part of the NODES, 2026, repository in the the YouTube. Okay. So as I said, the hardest part here is to, let's say, answer these, complex questions available, in the question bank. You know, of course, the the numbers... So, you know, processing the numbers, is the easiest part. These, probably won't even require complex, gentry crack and definitely won't require, complex tools. But the real problem is when we come to the, to judgment, you know, some way. Because, it's true that, all these books contain, nice ideas, but translating the ideas in something that can be retrieved and visible some... Somehow outside is the difficult part. So that's why we... It was necessary for me to to gather information from multiple sources now. Currently, as I said, I'm using Shraddha, and I'm paying, you know, an extended license to get really all the numbers. But also I'm, extending it not only to the 10 ks anymore, but even to the eight q and other, another source of information. I'm planning to, you know, implement a crawler to extract the information from a news feed or to literally crawl, you know, I don't know, a YouTube transcript, etcetera, you know, to gather a lot of information from leaders in the organization, etcetera. So, this requires a lot of data to be processed, and this is where I think that a solution like the one that I'm proposing here could shine, in particular. And, these would require definitely more complex, agentic driving. That is exactly what we are going to do, to do here. And, Putting altogether actually is where having a system that can learn how to combine data sources retrieval, querying specific components, etc, is a real improvement together with some customization based on the, based on the type of questions that we have to answer. Here, I I added a few, questions from, you know, from the bank. This is, taken, for example, from the book, Good by Choice from Collins. Then, there are others taken from, other books like this one from, Moby Singh, and there is also another one from, Good to Great. Good strategy, sorry, Good strategy and a bad strategy from, Rumelt. Actually, let's say that, in the end I came out with 70 questions, you know, for different tiers, different type of aspects, different type of things. And, that's why, the problem was so complex that I needed really to focus on the specific areas of the entire project and the area where I focused was exactly how to answer to send... To these 70 questions with the information I have, if I need, further information. And the results are quite promising now in terms of the capability of the system to build the the, the the company card, the company baseball card. Okay. If you are interested, you know, I can provide you the the full list. And if you have any suggestion, definitely, you know, it will worth, extending it. Okay. So let's say that, in order just to provide an answer to this complex question, I needed to extend heavily the, let's say, the orchestration of these agents to include the new agents. So first of all, as you can guess, if you already played with these, complex agents already, I needed to, introduce a planner whose role is to literally take the the questions. And, as we have seen, we are talking about complex questions. You know? They... Most of them span multiple years, like in this case, you know, the... For each of the last ten years, and then you need to compute several aspects. And, each question generally is composed by multiple sub questions already. So this is what the planner does. So take... Takes the question and, start composing it in multiple sub questions. Okay? Then... And this is done by using the LLMs. Then there is a, a resolve plan, that, is not calling any LLM, but it takes all the information passed by the planner and start checking if there there is all the information about the company. For example, there there are some issues in the way in which each company, let's say, get to the the years, then, it uses external, tools. It makes a call to, Edgar, Filling, and others to check if the data is somehow available or convert to a company name in a proper CIPK code, etcetera, etcetera, to get all the information necessary. In this case, of course, if something is going wrong, there is no way to convert the the company name in a proper, you know, CAK code, then the system fails immediately because it's impossible to get any valuable, information, not from Shraddha, not from, Adri Filling, etcetera. So... But supposing that everything is going well, then there is this phase that is called the load guidances that will be the core of what we are going to discuss that based on the questions that arrive, it will, let's say, distill some guidances for each of the questions with the the details that we are going to see. And then we have this loop here where for each of the, the sub questions, we go through the the question answer, we check if it's enough, and we continue. And then we, we can go to the synthesis phase that write the final answer, based on the results of the other subanswers and the evidence, etcetera. And, and then, as in the case before, we have always this reflection because this is another suggestion that I would like to give you. I found very, beneficial to have always a reflection phase. Rarely, you know, if you just have a specific goal, the agent will be able, alone with a single prompt, to accomplish the task perfectly. Adding a reflection phase in any intermediate step or in a final step will give you really high value, high quality in the result. So this will will help heavily in that sense. So, actually, the focus of today, in the end, will be on, the... This specific, component. That is, again, as you will see, similar to what you have seen before. But, here, is where we focus our attention in the way which we can extract guidances that then would be loaded here for all the sub questions. You know? And, we noticed that, of course, we can improve the planner, we can improve the synthesis, but, actually, in order to priority, we needed to improve the quality of the final answer first, and then, we'd improve it. So let me spoil a bit what I'm going to say later. Because in the first version of this work, what I tried to do was to try to optimize everything blindly. I said, okay, this is my question... Full question and answer system, optimize it. So optimize the planner, optimize the way which it creates some questions, etcetera, optimize the synthesis, optimize the question. Again, reaching responsibilities is the way to go. So I started it from the, the the core of the problem because without a proper answer, I won't be able to provide a proper final answer. So then I focus it on the, let's say, on the core, and then the goal, later is to start on the planner, and there will be a specific tracing, a specific, obstruction for the planner, and the same will be for the synthesis and for the final reflection. So all these will come in the same approach, but, you know, each for these different type of, of system. So now that, it's clear that we are focusing again on just on the agentic rug in this complex system for the purposes that I just described to you, here we have, an extended version of what we had seen before. You know, so the core idea remains the same. So we have these three phases for the retrieval and there's two phases for the answering. But, of course, as I said, one of the issues we had before was that, we weren't able to get the, how to say, the data, ongoing. You know? So I had to load it up front, etcetera. So in this case, I added a specific agent whose responsibility is literally to ensure that, the data necessary to answer the sub question is available. So, of course, it checks in Neo4j, it checks in the, in Shraddha, it checks on Internet, etcetera, to assure that if the data is not downloaded, it will start the process of downloading and pre processing the data. So this component actually is responsible to assure that whatever it happens, will have the data to answer to it. So, this replace the humans to download it. So then the rest is, more or less the same as before. And, of course, we have now another, you know, phase that just condense, let's say, the answer as a part of the, the list of sub questions. So so prepare, in the in the state, the the the sub question. So here, I'd like to stop for a second and share with you, how this is translate. I mean, these complex, let's say workflows are translated in quite a clear code. We will not go too much in detail, but again, I would like you to highlight how much Lambda Graph was the, let's say, a big help for me. Then I will take some questions before moving farther. So let me share another screen here. Here we go. Okay. So, Well, so, Alessandro, before we move on, yeah. one one thing I would love to, to ask you is, why did you choose LandGraph, and, did you evaluate other frameworks at the same time? Well, let's say that I didn't evaluate a properly other, other framework. The reason why I like the line graph is honestly because, it is a graph, structure, you know, as, we will see, now. Actually, it, allows them literally to represent these, complex workflows as a graph where literally you can define, nodes and relationships. You know, the the interesting extension compared with the, for example, the way which we will represent this in Neo4j is that you have the conditional edges, which I found, very, very useful. And, you know, so that the idea of having a mental representation as a graph and can represent it very easily in a Laing graph was one of the main reasons why I literally started using the Laing graph as my initial solution. I'm pretty sure that there are other solutions that are absolutely valuable. So far, I'd say that for my tests, I didn't find any good reason to move out of our long graph, despite the fact that I'm using it for very complex scenarios. And another advantage that I don't know if other solutions have is that you can compose subgraphs. You know? So literally here, I'm composing at least two subgraphs, one inside the other, you know, in a very simple way. So it, even mentally, again, gave me the opportunity to specialize my thinking to what I'm going to do. So for all these reasons, you know, I found familiar working with, with LandGraph, and I decided to to use it. And also there is a Java version that we are using in Maestro. So even the translation from, Python to Java was quite straightforward because of this of this reason. So more or less because of this. But I I I guess that other people could have a different opinion about what is the best solution for implementing this. I tested a bit of QAI, but I I wasn't super happy about about it at the beginning. But I tested it many, many. months ago. sense. Yeah. So we we use line graph on GraphAcademy as well. So, we're we're in good company. Woah. Perfect. So even better now, you know, because, we can, definitely learn from from from each other. I I guess that, you know, I I spoke with many people, even during, you know, some, I mean, random conversations, and many of them are using, Langraph. There are defects. I I I know about it. But, honestly, if you would like to do something like this, you know, that quickly allows you to define a complex workflow architecture, well, long graph is definitely one of my favorite. You know, I've seen this applied by many people. And so, you know, getting code from others was pretty simple because, you know, it was easy for me to to just take the code and attach to my code because it was already in the same framework. Cool. Okay. So, based on that, let me just check since we are speaking if there is any other interesting question. Let me see. I'm going to the lot. No recording. Not if it is in the chat because I'm looking at the q and a, but it seems no. Okay. Adam, if you don't mind, time to time, have a look at if there is any relevant question, just, interrupt me. I'm checking back to there are not that many. Okay. So let me, just to reinforce on what we were, discussing now about these, land graphs. So this is the, let's say, the broader, let's say, land graph, they'll set up, you know, in which we have a different nodes exactly as you've seen in the schema. We have the planner node, we have the resolve plan node, we have the load guidance, the next sub question, then we have this subgraph, which is the graph running the question answering part of the subquestion, then the synthesis, etc. In this piece of code, what I would like to show you is, of course, the way in which we are having this dedicated node that, of course, I I didn't present in the in the schema because these are, let's say, utility nodes, you know, that allows me, for example, to start logging a question even before entering the planner because I would like to know if for any reason the system won't reach the planner or the planner will fail. So, these are specific nodes where I log the the the the entrance and, and the end somehow, you know, of the, of the system. And log final answer is here. You know? So these two are not represented in the schema that you've seen, but they represent literally the beginning of the end, you know, the start and the end that you've seen there. And then, you know, there is this context graph that is the, you know, the the the library, the, the class that is responsible for storing this properly in, in Neo four j. As, as I said, one of the future extension will be to replace this system with the agentic memory now available in Neo4j so that we can have a more standard way of representing this trace. Consider that this is a custom version of it, but we will likely to use, let's say, a more generic approach that will be easier to maintain and to leverage in the future. I started already, but it's not available yet in this code. And then, of course, you know, this, run sub question actually is empowered by another, you know, subgraph that is this one that has a different type of state because I don't need it to, you know, bring everything with, with me. And again, here, you can see exactly the same structure that you have seen in the schema. Again, I I I love, LandGraph for for this reason, you know, because I can represent in my schema and then map immediately with the, with the Land Graph in a very straightforward, straightforward way. What is, what is important is, of course, that I would like you to to see from from this, piece of code is what I mentioned before, this idea of the conditional edge, which is quite helpful because you can define some rules according to which you can switch in one direction or the other. That I found, super intuitive and super easy to to use and to, adopt, you know, in my workflow, definition. So here, for example, you know, I'm saying that, I will pass from ensure that an order to the next step or to the question answering in the case in which the system doesn't have enough information. It won't just send a failure. You know, in some way, it will pass the ball to the, to the question answering for that specific sub question saying, probably I don't have enough information for for this, for this sub question, etcetera, etcetera. You know? And probably the planner would, would restrict it. And this is very, common, you know, and as you can see, I'm using it many times. So whenever you've seen a square in the schema before, actually, you will find another conditional edge in the, in the code here. Another thing that I would like to show you just as a as a future evolution is that, you know, here we are implementing this, plan and execute part, you know, in which we are, able, to have a planner session that describes the the list of sub questions, and then we execute the sub questions. Then at the end, as you can see, we have this final reflection whose option, as we can see from here, is to, go to the... Back to the synthesis, go back to the next sub question if it requires to have another, sub questions or it, logs to the final, answer, you know, in some way. There is no way for the final reflection to step back to the planner, which I found out... I mean, this is common. Absolutely. These are the... Whenever you see the, plan and execute of the pattern, this is the way which it works. There is a planner, then there is a loop that executed the plan, and then there is a final evaluation. Rarely, you see in the examples that you step back, to the, to the planner, you know, so that the final decision will step back and say, oh, look. There is, probably a better way to execute it because this didn't produce the expected result. And actually, I found these in a few examples that I'm running very, very useful. So probably in the next, in the next version, you will see many different loopback coming from, I don't know, the... From next question, for example, coming back to the to the planner or from the, from the scene... From the final reflection coming back to the to the planner, you know, suggesting an improvement or a revision of the, of the plan. But, of course, this requires some, some other, improvement. The last thing that I would like to show, and then we will speak about this extensively later, is that there is something that you should consider. Because as I said, having a dedicated, tool calls for dedicated task allows me to have also dedicated models. So here, you can see that we have a greater model and we have a strategy model. And in this way, in our schema, we can use a different model for the strategic occasion and a different model for, the greater that, I mean, this is the... Call the name for the, let's say for the evaluation and knowledge building part. It is called greater, I mean, just for simplicity here. You see, we can pass both of them in this creator sub workflow that is exactly what it does here. And also that we can use a different... The grader is used in the grader document and the strategy is used, of course, in the strategic model. Of course, if... I mean, in the standard way, you will use the basic model, but if you would like to specifically assign, a different model, this is definitely the case. You know, and this is what the system allows you, allows you to do. In terms of the code, the other thing that I would like you to to show you just as a good suggestion is that, in my code specifically, I took the prompt altogether, of course, and in some way I'm also versioning them. So that in the case in which once I will make the call available, you would like to adjust it to different purposes, etcetera. Definitely, this is the entry point for you to understand each of these prompt, what they do. Of course, not always you have a single prompt. For example, in the case of the planner, there is an internal loop that I can show you actually that can revisit the plan just internally. So before passing through the next steps, as I've said before, it can just check if the plan makes sense. And, if not, before moving further, we'll just get back and, we'll, you know, try to improve it before moving into the next. It happens just once, so it's not like a loop, but it's something that can happen. So that's why you find the two planners here. And, this is something similar that can happen, you know, again in other points. Okay. So let's go back to the, presentation for for a second so we can continue there. Okay. Perfect. So now that we have seen how the, let's say, the the code work very quickly, and again, I will make the the code available later. Let's, Let's start discussing about the context graph, and how we are storing it. Actually, again, let me say that, here there are two different options that I would like to mention, clearly. You know? Because we need, to store the the trace of the execution. And here is, where I see the the difference and, why, I needed to migrate it to a broader approach. You know? Because what I did here was to literally model my, graph matching exactly the, let's say, the way which my agents, interact to each other. So as you've seen, we have two phases and each phase then has a different steps and can go through multiple iterations. This is exactly the way which I modeled the graph. Here is the question and the question which is in reality, there's a sub question just to be clear, not the the major question. In reality, there is this relationship here that contains the relationship between the question asked by the user or in the question bank and the sub questions generated by the planner. And here we have the planner. Okay? But let's focus for a second on the, on the context graph related to the question answering in the sub question part. Okay, so we have the two phases and here, we have the retrieval phase and the answering phase, exactly as I pretended you the first time. Then we have that of course, since I can go through these iterations multiple times, of course, I have a retrieval phase iteration and an answering phase iteration because I can't have half and half. So when I start with a a retrieval phase, I need to complete the retrieval phase. I left there with an error, of course, but in general, I have the entire retrieval phase. And the same is for the for the answer. And then, of course, each of these has a generic, AI step that can be the the... Let's say, the strategic agent, the evaluation and knowledge building, the answer, generator, and the other, evaluator. All of them are called the... Oh, sorry. All of them are called the, AI step. Of course, I have, you know, two calls that is also traced. Actually, I'm also keeping track for each of the tools. I mean, for each of the tool calls, actually, the documents returned by that call. So I can map exactly what was the query and what was the result. You know, that has, quite, an interesting approach because I can easily, for example, map if the the tool call is making a a call to the chunk that is related to a document of a company that somehow was in the plan, for example. You know, that's the power of a graph explained here. So that's why in this case, I wanted to store, everything in the in the graph because, there is an entire area that is not presented here, I will show you later, that of course contains the chance, as you can see, but also the playbook. So in a single graph, I can store literally three areas. The main ones are the trace and the documents, but also I can store the, let's say, the playbook, with all the information and the information about, you know, what playbook could work and for what question, etcetera, etcetera. So, actually, in this case, you know, I can close the loop of the tool calls with a with a chunk, with a, with a document, etcetera, and, and adding a full picture. That's, as you will see, it's important in the moment in which I have to distill some information. As I said, this is very specific. So I have my, LandGraph model. I have my, let's say, workflow, and I'm up with it in this case. Of course, using, you know, a Neo four j agent memory, you can keep it more generic. So it can work with, any type of system. Of course, there are, advantages in having such a specific one because, of course, you can tailor the extraction and the interpretation in a specific way, but, of course, the drawback is that it is specific. So if I have to migrate this solution or to, another approach, I have to implement it again. So probably this is good for a generic, agentic, rugged solution, but won't work for other, let's say, planner, etc. So I ended up ready to extend it a bit now that I introduced planner, that I introduced the data acquisition. So every time I add a piece, I needed to add another node type. So objecting memory in Neo4j will do this in a more abstract way, which has several advantages in that sense. That's why in the next generation, I will try to migrate these and use that type of Neo four j, object memory because that could be more generalized. So wait for, for that. So to better understand, with a concrete example, let me show you, how this is stored in Neo4j once you execute a query. Let me see if this example can work. This is the full query, actually, that identify for the two fiscal years, the moment last five, etcetera, with the lowest operating cash flow. This is the generic question, the question asked from the question bank. And also the complex one. Of course, this has been slightly, revisited through multiple iterations. So just to make it a bit more clear because, since these are predefined questions, I would like to to be sure that the the system understands it. Of course, this is not always the case. This is the plan that has been generated with all the information I need. You know? There's some question IDs because, I have, three types of questions, not the three questions, but three types of questions as we will see later. And then we have the type of the questions. So q one is a panel question, for Microsoft. So then we have a q two that is a narrative question, for each year and the q three that is the derived one related to the rest, etcetera. So this just to give you a plan. So based on this plan, we have different questions. So for example, here, we have a q one question. Actually, that is the the first one, that has been generated. Then we have the q two, and we see that the... We have the same questions, apply the two different years. So here we have a Microsoft score for 2026. Here, same question about 2025, etcetera, etcetera. Just to be clear, these questions can be run-in parallel if you like, at least these are... Because there is no, dependencies between between them. And here, we have, the, the different phases. Here, we don't need, for example, a complex retrieval phase because these are, you know, these type of questions, requires just numbers. So we don't need to complex the two phase. So we are, using just the answering phase. And, here, instead, we have the, you know, the the different phases, including a data acquisition here, you know, in which we, we describe exactly which kind of, feelings we need, etcetera, etcetera. So... And here, we were lucky because we had, an answering phase with a single iteration. Here instead, as you can see, we have a multiple answer iteration, you know, where we had some issues probably in the answering and it got through a longer process. Again, here we had a data acquisition step. Probably, we we added the data already or we downloaded it. But here, if we like, we can expand, you know, with the with the documents that have been acquired as a part of this, of this iteration. So again, very, let's say, specific. Of course, I can get to the same for other for the questions. So let me just try to run another another question just, you know, for the, for example, I can use an action here, and then I can select, for example, one from... I don't know. I did a mini test for Apple, for example, and, I can see if, it generates a complex... So this is a way more complex question. Again, same structure as before, we have these global question, totally different over the last five fiscal years, etcetera. And then we have a different plan, and again, we have, a different type of questions. So in this case, we have the same. You see repeated multiple times, all of each with the retrieval and answering, retrieval and answering. And then we have probably another type of answer that is this one in which we have, you know, some more complexity. You can see that here we have different iterations. Probably, it won't... It wasn't easy to find the information. And this is a very good example of what we would like you to obtain with, let's say, with guidance. So because now we will come to the discussion, but, here is the right moment because, this is exactly the problem that I saw when I thought about, okay, what can I do to make it better? Because, of course, we can make better the answer, you know, to make it more accurate, to, let the system answer, knowing better the domain. But also, we have tons of other, things that we can improve. So in this case, for example, you know, I see that clearly, in this case, for this specific question, I struggled, or not me, but the agent, or the agents struggled to to find the right information. You know? It's a attempt, in in a way... So we can see here, for example, you know, this is a, a step that, you know, this is the retrieval agent that, you know, was approaching without any, any previous guidance. Then we have another one. Again, this is another retrieval retrieval step, but in this case, you know, it has some some guidance... Not guidance, but some feedback provided by the the the previous... Here we go. So you see here the system provided some some feedback and say, okay. Missing annual report or reported filling figures, two thousand nineteen, two thousand twenty four. So clearly, you know, we have enough information here to say, okay. What can we improve? Why do we need one, two, three, four, five, six steps, you know, instead of just getting all correct to the first one. So what is the issue? What was the issue? What issue in the retrieval, issue in the in the in the generation of the answer, etcetera, etcetera? The same can be, of course, for the answer. You know, as you noticed, it's very common to get through multiple steps of answers. Here we go, we went through the process three times, same here, here not, here not, here not. So these are fine, these are not fine. Okay, so this is just to give you an idea of, what is the complexity, why the context graph contains, so much information that we can leverage for storing it. And based on that, we can move to the last and more important piece of this presentation because the concept is care. So we have complex agents that constantly, take decisions about what tool to call, how to retrieve. They fail mostly, and they will actually improve. So there, in the... While I was preparing the the the previous talk and I was working on this finance bench test for AgentiKorag, I read this article about Agenti context and here you will find the reference explicitly. That has a very clear idea that I love love, that's from where all this effort is coming. So the idea was, okay, why not creating curated lessons, learn from context graph? And they called the playbooks. And I found this name very, very useful, and very accurate in some way. So it collects literally strategies that work or pitfalls that the system went through many times that the system should avoid next time. And they had this idea of saying, okay, we should constrain these playbooks to, to a certain amount so that we don't have it growing constantly, but they should, somehow be constrained to a certain amount. And, here is where I find, let's say, my thinking a bit struggling because, of course, I don't want them to grow indefinitely. But, I mean, having a a playbook of 30 elements that could solve all the issues, it won't work. So we see how, I came with the idea of creating specific playbooks for specific tasks for specific users probably. The idea remained the same but applied at scale, let's say. They can be customized and retrieved based on the current task at that point. So, but, as you will read in the paper that has been re adapted very recently, it shows how these, magenta context engineering or ACE can improve the quality of the existing models quite a lot, which is super useful also because this allows us to customize without fine tuning even smaller models. So basically, the key elements of this process. You run your agent. You run your, task. In our case, the the sub questions, you know, and you record each of the each step. You know, that is exactly, what you've seen right now. I showed you through Yoom how the execution is completely traced. At that point, what they do is to split the process of creating this playbook in true phases, and they call the reflection and curation. So the reflection, again, with the idea of the specializing responsibility, has a specific, task. Take the execution and somehow abstract it, you know, understand, which decision happened, which heart, you know, and then propose a list of lessons learned. Of course, without our creation, we can store all these lessons, but the creation has a... The the creator role literally. It takes the the current playbook and say, okay. These are my 30, 20, whatever, lesson learned. These are the the new candidates. So how can we combine them? What can we merge? What can be, removed if it's a contradiction somehow it's arting us? And this process of curation will make these lessons better and better after each iteration. So this process, so this number two and three, produces the paybook that contains the current version of lessons learned, let's say so. Then there is a step that is not available in the paper because I'd say that is my personal suggestion that instead of passing all the playbook to the to the next, whenever is necessary, what I introduce is this concept of guidance. So identify the type of question and the step, we will see how. And then instead of passing the full playbook, I say to another level, okay, this is a question. These are the lessons learned. Give me the best guidance to accomplish the task, you know, condensed so that if there are lessons that I don't care because the question doesn't need it, it will be eliminated. This, of course, are the... Another step. But, again, with the idea of, reducing the confusion of the, in the front, this, improves the quality instead of just passing 30 ruse altogether to the, you know, to the system. Because consider that these are not alone. You know, not only within the prompt, we have probably, feedback, from, the... For example, if you think about the SAP Education, we have, probably feedback from the previous step. We have the list of playbook. We have, probably the the the... Not probably, but we have the the growing knowledge. So it's already a lot of information. In this case, we have a small step which takes a question and a playbook. Just this. I say, okay, is this relevant? Yes or no? And then it condenses these in a a list of sharp guidance for the next step, so it will be a more condensed way. Again, I know this adding complexity, but, I noticed that, it's better the agent to be very focused on the thing because it tends to forget things if we pass larger prompt even though it is improving. Okay. So let me give you a more concrete example just to, you know, to understand better. So the reflection, observes and proposes. You know, so observe, so from what it sees, you know, figure out what is the fastest or what is the worst case. And it says, for example, that in this case, the reading table was wrong. And really abstract something that is generic. Then we will see how we improved also this step. At this point, after it abstracts, it just proposes, okay, based on my knowledge, based on what I see, this is what I propose. Then of course, the creation check. For example, the creation can check if this is a very specific for a company. If it fits the specific stage or it is a generic and irrelevant, or if, for example, contradicts some specific generic guidance provided by the humans, for example. And, based on this process, it decided what to do. If, should be added, if we didn't reach the, the threshold, we can decide to add a new one, or we can merge with an existing one, which is very similar. So it will be a more, accurate lesson or can be rewritten entirely. So replace the completed by another one or just remove the, if we feel that there is a lesson that, is somehow not necessary anymore because the new one is way, way better. So for example, you know, this is an example of a... Of an entry. So for any subs... For any selection or sub question, first, the required rule over, the full in scope series, and etcetera, etcetera. I mean, this was even for the planner, for example. You know? And we noticed that, we get an improvement of the 28% less rejection, for example, and 9% less of partial sub answers. So actually, we have evidences, in case, even though I didn't speak about the planar implementation, but here, just to give you an idea that I also tested the planar, that even the planar can be optimized in this sense. Okay. So, I'd say that the final result wasn't simple to achieve, you know, because, I already run multiple, multiple tests for the Nozzei experiment, let's say. And I run way more for for this one. I tried to hear to summarize a bit, let's say, the type of test that I, that I did and, that we did actually, with Dario and, also the, the the lessons learned that helped us to improve the the the full the full process. So at the beginning, as as a big mistake that I did, was, you know, in the first campaigns, was to use, the full question. So the user question, the big one, the the one that coming from the the question bank. So I considered one company, I considered the the the full question, and I tried to optimize the entire process. I mean, I I say... I would say that, the the result was very, very bad, you know, because of the the questions were, too different from each other and the way which they were posed, the results were absolutely, not consistent, to the level that the, you know, the the reflection and the curation cannot really abstract anything, valuable. So we had this holistic creator, that wanted to optimize everything from planner to some question... To questions, etcetera. So it was way too complicated. And and then, we decided instead to do something slightly different because the planner was changing all the time. So if I asked the same question multiple times, the planner was so different that whatever I learned from the previous probably won't be useful. So I, I decided to run the same, questions by frozen, the plan so that we are sure that at least the sub questions were the same. But still, we were running the the process holistically. So the the the process about taking the final process into account. And again, Zelle improvement compared with the baseline, but nothing measure. Again, we were using the, you know, one company. Then we said, okay. Instead of using the same execution, the same planner for multiple executions, I try to pull multiple companies at least. Again, here, the problem was exactly the same. The problem was clearly that starting from the big question, it was too much to optimize. So the big change happened when we decided to optimize the sub questions that are way more controlled, that matches exactly with what we were doing before. Because, you know, before we were literally optimizing the smaller question, and it worked pretty well. So we were a bit struggling with the results because we said, okay, it was working before, why not now? So actually, as soon as we moved to some questions, we started getting more accurate results. But in reality, by doing that and by noticing that the planner was already, you know, categorizing these questions by a panel as you've seen or by narrative, etcetera. Actually, this allowed us to, somehow, categorize even the playbooks that we should, we should create. And this, gave us way better results. In this case, the better results were perceived or were, computed by considering the amount of, of steps, you know, reduce the amount of iterations. So the effort went down. It means that, we needed a less LLMs call. We needed less iterations. We needed, in general, the answer were more accurate because the data passed was already correct, etc. We noticed immediately an improvement. We created a playbook of error category of sub questions and the specific steps. So we created a playbook for that category of question for the strategic retrieval. We created let's say, the same, another playbook for the grader, etcetera. So this was improving a lot of the accuracy of the process. So the the number of steps were used. But the accuracy, as, you will see in the end of my conclusion, you know, it was already very high, you know, because in the meantime, we optimized a lot by using the code improvement. We improved the prompt manually. So these already skyrocketed our quality, in terms of the the answer quality. And, so we reached the the threshold. It was super accurate all the time. And, in reality then, in the last test, that is something that we we will discuss further. What we did was to say, okay, since the accuracy... I mean, we solved the issue about the hydration, so it went better, perfect. But the accuracy is something that we wanted to measure, but the accuracy was already high even without it because we improved the code. We said, okay, what if we see if the answer can be tailored to the specific type of users? Instead of getting generic, we said, okay, let's suppose that we have two different types of personas. One is executive that wants a high level answers, no details, and a few less sentence. And on the other way, we have the analyst that would like to have very detailed tables, very detailed, step by step of computation, etcetera. And see if by running iteration and creating paybooks for two different personas, we can customize it. Well, the result was amazing because we had a a range between the 5180%, of accuracy improved in the in the measure of how much the answer was accurate or was, let's say proper for the specific persona. So this was the key for us to get something that is meaningful. So with this approach, I can say, I mean, that definitely we were able to optimize the answer based you know, on the user preferences. It is exactly the purpose of this, execution. So... This experiment. So just to, summarize, what went wrong, because definitely this is from where, we learned the most. You know? Definitely, learning from one run is a no go. So, the initial idea was to run it once and immediately started, you know, the the reflection and the curation. Didn't work, so the solution for this was to group runs. Then we will discuss how to group them. But for example, we decided to run the same question for multiple companies or to run the same question multiple times and hope for, some variation. Definitely, one company, Faxuty, is definitely not good. That's why, you know, when we rerun multiple times the same question, we decided to run the four different companies to get a broader understanding of the options available. The other error or the other... What went wrong was to to to imagine that our lesson can, work as a generic lesson for the entire engine. Instead, we decided to to move toward a specific step. Again, a lesson learned in general is that the specialization works better than generalization. You know, the... In every single step, whenever we specialize the agent and whenever we specialize the playbook, we obtained better results, you know, definitely. And, and of course, you know, connected to that, what happened in the generic part, in the generic approach was that sometimes the same lesson, landed in the wrong set with the consequence of getting against what was supposed to be in that lesson, etcetera. So lesson per category and step was the final solution. But there is another major improvement here that, came as an idea discussing with Dario while we were trying to further optimize the process. So we wanted to take the basic idea from GRPO. Basically, what we wanted to do, or the clear... The basic idea was, okay, if an execution went well, we will not learn that much. The the real learning happens when we run the same exact query in the same exact conditions and we see a different behavior. So whenever we get different behavior, then we can learn what really worked and and differentiate it from what didn't work. So actually, we change the completed approach. So whenever we, somehow, we're likely to learn from an execution, what we do now is to run... To take the sub question, run it to multiple times, and, instead of, you know, taking just the the the different run over the single... Of the same sub question, we actually consider multiple executions. You will see an example. So let me try to explain now, but it will be clear later. So Imagine that you have a question about a company Microsoft for a fiscal year 2025. This is some question a. I run it three times and see how it goes. Then what happens is that I run the same exact question for the same exact company for a different year or the same exact question for a different company. Actually, the the real learning happens when I run this. I fix the question and I change the execution path. To change this execution path, I can just execute it randomly, I mean, many times, and the temperature to one will give me enough variability. Or I can change some condition, like the same question but different company, same question, different here, and see how it will behave. So that when I run the reflection step, it won't run on a single well done execution that teach me teach me nothing, but I can run from a pool of questions. And actually, what the system does apart from running all these questions multiple time or aggregating these sub questions before running the reflection is that also decides if a group deserves to be taken into account. If for a question, even with all these changes that the execution is always perfect, there is nothing to learn. Which means in reality that the engine already knows what to do with this. There is no lesson that will be necessary. It's better not to provide any lesson rather than providing a wrong lesson. That's exactly what happens now. So this idea taken from GRPO also that really taught us how to generate variances of the same execution so that we can learn better. Actually, this is the way which we generate a more accurate lesson. And this was the other big step in our learning process in this last, probably five months. And as I said, the greatest results that we wanted to obtain was in terms of capability of the system to adapt to specific needs of different type of users. We run this execution and we provide the same answer to different users to see if they appreciate or not, and then to see if we can learn from the fact that they appreciate a solution, an answer rather than another. In this case, we were not optimizing to match the, let's say, the execution or the retrieval part, but more the answering part. Actually, we've got a lot of the playbook, we will see a few examples later, that literally optimize the way which the answer is generated. We can see that each one... If we have an executive playbook, it will be rated higher by the executive and lower by the auditor compared with the more generic, the one who using a generic playbook. Actually, by using a specific playbook for a specific type of user, we can get to that the user will be more satisfied by the answer in the end, which is exactly what we wanted to obtain. Here is exactly an, an an example. You know? So for the same, question... Oh, sorry. From the same answer, we can say... We can have that executive says, okay. Every cell carries the full dollar figure tries to plus across check. So it gave a one of five as a rate. The same question for the... And the same answer were rated a four over five because it provides all the detail. Because of course, the executive doesn't need the details, The auditor, does instead. The opposite situation for the, other type of cases. You know? So no details, one sentence on, and one table in billions. That's all I need. Four over five. No. You say the feelings match, but in reality, I can't see it. So it was, narrated poorly by the, auditor. So this is, for the same question, we can get two different, let's say, playbook or I mean, entry in the playbook for different users and we can, of course, provide more suitable answer that will match their their needs. So with that, we, can see a a quick demo of my, web coded, live coded application. So as I said, you know, last time, I, I really, was ashamed that I didn't have an interface to to show during my talk. So this time, I decided to to went for, if I call it application, actually, I learned so much, you know, in testing these new experience because I'm not coding anymore, as you can imagine. But, you know, doing this with the support of Cloud Code was definitely, a good experience for me. So I won't run it because it will take a while, but I run it already, many, many, many, many times. So here is one of the questions that you have seen already. So in this case, I'm asking, over the last five years, did the commercial model change, etcetera, etcetera. And, this was asked to, for Alphabet Google. Okay. So the answer was provided, etcetera. And, actually, no. Let me show you something different because this is not very, very much. Okay. Yes. This one. Sorry. So identify the two fiscal years among the last five years with the lowest operating cash flow, etcetera, etcetera. So in this case, on Tesla. This was guided by, the playbook, you know, while I have this one that was with no guidance. So same question, you know, different behavior. Let's start with with the one with no guidance. So we answered to this question. Actually, this was the final answer, so that what the answer combined. So it's the finish, you know, okay, gaps, etcetera. So how the planner, define the the difference of questions? So here, we have, the the the question q one. So we have the literally three type of questions. So q one, it was asked at once with the four retrieval and three, iteration of the answer. So three iterations of retrieval and three iterations of answer. Then we have the same q two question, answer... I mean, ask it for different years. So q two for 2021, 2022, 2023, and 2024, and then a q three that, somehow depends on all the the previous one. So in this case, we don't need, any retrieval because the information is already available. We need to adjust to provide an answer. So the first thing that we do in general here is to categorize the the the questions. So here we can see that the q one is called structured figures or panel because it literally uses structured figures, which means a shorter data. Actually, here instead, these are Q2 belong to session text because it takes data from the fillings, so from 10 ks, etcetera. And the Q3 is instead just calculation. It takes all the results and just provided the final answer that you can see here because we have a table in particular, so we needed to compute the the results. And we have a specific tool for it just by the way. So here are the the answers for each of the questions. So here we have, you know, these are coming from Shraddar, as you can see, plus with the Edgard for cross check. Of course, this is completely decided by the system that decides to take the data from Shraddar and double check. Of course, it's structured by my, by prompt. Then we have the answers for, you know, the, each fiscal year, the employee head count for, 2021, the employee head count for 2022, etcetera, etcetera. And then the final the answer that had been evaluated by the system partial, but in reality it was quite accurate and complete. That's the plan execution. This is what happened. What should happen later is that as a user, we should rate and provide a description, why did you like it or not, and that's exactly what is used to customize the outcome to your needs if you like later. And then you can run these called batch curation. Batch because now instead of running once, it will generate more, more things. You know? So let me show you an example of the batch curation. So this is, for the identified two fiscal years. Okay. So, for example... So these are the the the pools. So alphabet Tesla. Here we go. Okay. So this is, an example of the different execution that I told you. So this case, you know, q one was fixed as a sub question and then was executed multiple times. So in this case, you know, we've seen that, here and here, the grad... Grader, failed the... I mean, considered the answer not sufficient or not sufficient. The retrieval not sufficient, so we went through two iterations. Here, the one was accurate. So it's a perfect example of saying, okay, what differs from these two? The same is here. Same Q2, and the different type of behavior. So here, the reflection went well, here not. Here we're speaking about the answer. So the reflection part. Here the grader and the reflection, everything went well, so nothing to complain about, the same here. And you know, then we, you know, it it measures different approaches. So from the strategy perspective, from the strategy agent, you know, it takes the execution of the the q one across the three iteration... The the the three actually execution where you see the, three runs. So you see that, for example, the average retrieval duration is the two dot 33, etcetera. So the delta, the two calls six six zero. No. Sorry. Six six, as a rerun, strategy calls, etcetera. So it takes a lot of statistics about the the execution and, actually, then decides if, this specific set of this specific batch deserves to be used as a... In the reflection part. And the same is for the, for the rest. It'll, you know, draw those... These, like, style guard attempts, iterations, repeated the tools, call, etcetera. And then it, will, generate some some proposals. So let me give you an example of a few proposal that we applied. Applied like this one. Okay. So here, we have an example of what has been applied. So here, we have an example of what has been applied because the application can come from from multiple batches. And also, for multiple batches can be aggregated again. So not only, between the same type of company, but between multiple companies and then we can generate, you know, multiple, how to say, rules for different, playbooks, etcetera. So here, you see it, and etcetera etcetera. You know? For example, here we have all add because the playbook was almost empty. I can show you some others in which we have some some other merge. No? All add, add. Okay. These are all add probably before. But in general, you know, we see that we have, for example, here, this has been dropped, not accepted, etcetera. So this is the the the entire process end to end. So we have the question, then from the question, we can decide to start a batch that will rerun the question, for multiple... I mean, multiple times. So we can decide how many times because I didn't show you, but you can decide, how many times you can rerun the, you know, the the batch here. So let me check an example. So here, we have... You see, we can decide to run it three times or five times. We can even decide to use the guidance during the execution to see, you know, if the guidance somehow happened or, or not in this, in this sense, etcetera. So this is one that has been applied. Let me see if I can show you how has been applied here. So this is another question, for Microsoft. The process is exactly the same. This failed all the time. This failed once, etcetera. So here, we decided what to take or what to, discard. This is a proposal, and then we have, you know, the the proposal to to apply. For example, in this case, we have two ads and one scape, for example. You know? So this is, the the entire process. Actually, this is a a fully automated. Now I made it step by step so that I can check every single step. And, of course, this was very useful for me to present it properly to you. The last thing that I would like to show you in the context... In the graph is how we can, you know, present these in the, in the graph. So, of course, again, the graph helps us quite a lot to to to keep track and to, and to verify every single step. In this case, what I wanted to see, was to see, what question... Sorry, what lessons were used to generate guidances and what lessons we generated back after. So in this case, you know, we we see that we add, these question that generated all these, you know, playbook entries. These are all playbook entries. For example, now here in the answer, reject, the answer if the post table conclusion does not match the tables displayed, the results, the action, and labels. Calculation answer is not acceptable when the reader will have to reconcile conflicting summary and table statement. This is absolutely a nice suggestion, you know, etcetera. And there are many others. And, I can, of course, see what was the effect of this, playbook. And, again, the graph is helping me to see where the the specific playbook content was used. And this is enormously useful for me because I can see if one playbook was applied to which company, etcetera, because I can, of course, expand it all to that, to the relevant company, etcetera. But the company is also mentioned in the, in the text here, like Apple. You know? So here, literally, we have all the information we needed to to get through that understanding of what was generated as a result of, these batch processing on a on a question. And, of course, what are the, you know, the the the questions that came later that were, you know, enjoying or were using this this playbook somehow to get to the answer. And we can check, of course, what was the quality of the answer, etcetera, etcetera. So this creates a really complete end to end mechanism to verify how the, the system is, is performing. Okay. So, I would like to leave some time for questions. So let me just jump in straight to the conclusion, and then we will have a bit of time for a few questions. Okay. So agent. Okay. So here was the the the full end to end approach. As I said, in the... In this specific presentation, I wanted to continue what we have done. So if in the previous talk, I presented it just quickly what was that, let's say, the the idea of creating this context graph and how to use it with a a bit of results, in this one, we focus the, extensively on that. You know, we went through large amount of campaigns. We learned a lot. We, created, you know, different tests to evaluate the quality under different circumstances for different purposes. As I said, we used different, even mechanisms to to to monitor the accuracy of the final answer. So let me... I tried it in this slide to to give you a bit of, you know, second ways so that if you would like to start doing it, you know, already, how to deal with the most complex part. So first of all, I mentioned many times, so I can summarize it with, the immediate Imprea. So I noticed how creating the specialization in many levels that we discussed today helps me a lot. It happened, to, I mean, to move from a React agent to a specific set of agents dedicated to two specific scopes. It happened when I decided to generate a playbook for specific steps. It happened when I decided to generate specific playbooks for specific users, for example. So this idea of dividing and supporting specific type wasn't very helpful. As I showed you the context graph, offered me exactly the flexibility I needed in terms of data structure. As I said, in this specific case, I wanted to match one to one my, let's say representation, my, let's say workflow with the data representation, but then the next step would be to use a more generic context graph. So that will be definitely something that I would like you to learn from. There is another important takeaway. That is, of course, I'm seeing again and again, jumping out ideas like loop engineering, etcetera, and they are, don't get me wrong, very powerful. But I noticed that often by changing the code, I can get immediately a result and it's almost for free. I am introducing the deterministic checks, the proper tools, adjusting a bit the prompt and of course, fixing some small things in the code gave me a huge improvement in the accuracy, for example, gave me less error or less iterations, for example. Whenever you see space for adjusting the prompt, do it. Don't wait to that, for example, in this case, a playbook will fix it for you. So the the basic principles remain valid. Really, the idea of the playbook is when you would like to provide a structured well organized system that should customize the outcome to the specific users. This is something that you cannot do with the code, but, you can do with the playbook. That's the moment at which you should use it. Don't pretend that the, the playbook will fix your prompt or will fix your code because this won't happen. The other thing that I found very useful by dividing all these was that I can use a different LLM. Definitely, this saved money for the company. Consider that, for example, now in production, you know, with the next version, we are testing, for human, GPT 5.4 Mini. So by dividing, we were able, you know, to really specialize the the results that we can use, very, very basic models. But, of course, in my idea, when I will finalize this, make it right application, for example, I prefer to use for the arbiter, I don't know, something very complicated like, Opus 5.5 or 5.8 or six, whatever would be available at a time. Because the arbiter would be called at the end of the process, and they needed to be very accurate and very reasoning oriented. But the rest doesn't need it. So using different models will allow you to use the the right model for the right, approach. Of course, keep always an eye on the, let's say, on the budget. You know, in in the application, there is a budget concept that I didn't show, but you will see in the code, you know, that keeps me to, I mean, constrained. Otherwise, I will spend an enormous amount of money. So the budget allows me to to talk at some point if I'm not able to get an answer or if I spend too much for retrieval, I don't want to spend too much also for the answer. So this concept of budget as a... As an idea of, you know, controlling and constraining the number of execution is something very helpful. And, of course, keep, an eye on your, LLM bill because it could, definitely skyrocket if you don't, take an eye on it. And, of course, there is no solution that fits all. You know, you need to make experiments as I did and to, you know, do it. So my suggestion is really to try it. In terms of learning, I had the confirmation that I needed. You know, Playbook creation per persona type of questions and tasks definitely works. I can say it comfortably because I notice the quality and the, let's say, the type of questions changes significantly because of this, adaptation. And, because I I see how the agent can adapt the work to their specific needs. And, the other really winning idea is that running, reflection and then creation on top of a well done execution doesn't help that much. You know? What we notice is that, the the biggest step was done when we compared the multiple execution, hopefully, in different conditions and hopefully with something that went wrong. So we literally, like kids, learn from mistakes, you know, and, these was the the the most relevant, you know, lessons that I learned, not that the system learned. So Learning from mistakes is exactly what we needed, and this is exactly what we use for this tool. With that, I really hope that this was useful for you to start entering this world of you know, agentic AI and how to customize data by learning. And I saved some time for questions. I think we we can't hear you, Adam. Oh, sorry. How about that? Is. that... Cool. That's cool. So, yeah, thank you very much. That was a lot of, a lot of information to to take on, but really, really valuable information there. And I've been keeping an eye on the q and a. I think you've answered most of the questions as we've, as we've gone through. If anybody's got any any questions, in the last couple of minutes, feel free to to put them into the q and a. I think one that really stood out to me was a question around, I guess, the the evaluation of, the the the questions and the answers. So how much can AI, like Codex or or Claude, for example, you know, understands and and evaluate the the answers and how much of that needs to be done on a human level? Well, in this case, specifically, I used entirely an, LLM as a judge approach. You know, so I I didn't evaluate the answer manually. I just did, in the early iterations. Let's say that, what I, used, is, is something that I do very often. So I use this LLM as a judge, as a a principle, and I constrain the judgment in a very, let's say, precise way. I don't say, okay, the answer is partial or not, but instead, I define criterias against which I ask the LLM to judge the final answer. And whenever it's possible, I ask more complex and advanced LLMs to make the evaluation. So in particular, in this case, I will using cloud, and in particular, Opus 5.5 as a very advanced LLM, model to evaluate the quality of the answer by, passing specific criterias for, for making the evaluation. So this was my suggestion. Of course, the human evaluation should be the best one. But of course, considering the time constraint and the people constraint I have, I needed to use some automatic system. But it was very helpful at least to get a glance of if the question was correct or not. Yeah. Definitely. Yeah. And another question, on... Along the same lines is, I suppose the... There are lots of of reasoning and and thinking models now, that seem to do the... This type of planning. Do you see that the the models will be able to take care of this this evaluate evaluation stage in in the long term, or do you think there... There's always gonna be a place there for, these land graph, type, agents? Well, yeah. I mean, definitely, these LLMs are getting better and better. You know? At the same time, I think that, they are also getting more expensive in some way. So, I I found very useful, as I said, you know, leveraging the different models to save money. And probably in the future, in particular, with this new idea of the loop engineering and the specialization of certain model around it, probably we will need less agents. That's, I mean, something that I can imagine in the future scenario. But still, I don't think that we should, you know, or we will ever land in a scenario where you can say, okay, this is what I want, do it. You know, I think that this will be quite far. And in any case, it will be super expensive, so I don't know if this would worth considering as an option. Yeah. Okay. And one one last question that I I will, I'll throw at you. And so what happens when we reach a point, that we have millions of, pieces of data, data points, different files? Is RAG... You know, could... Can we can we use RAG at that that point, you know, if agentic RAG, is is still the best option? yeah. I think that if we think about a rug as a generic concept, I would say absolutely yes. Something that, here hasn't been, investigated, deeply is the use of a more complex type of rack, you know, that is the graph rack that can be easily combined with, with this approach. So if we enter in a scenario in which we have a really huge amount of data, probably having a a more, let's say, better data model for chunks and entities and chunks and subchunks, for example, could help us more in that scenario. In this specific case, what I wanted to do was to take the complexity on the question side, but of course, considering a limited amount of chunks. If we needed to enter a system which really the number and the amount of data is so complex that, you know, even the current indexes won't help, well, we can have a later to have a different data model. And then solutions like Text2Cypher, for example, can help, you know, to dig into, this more complex model. I found this quite useful and is an area that will work for investigation. Excellent. Thank you. Yeah. And I absolutely learned a lot through this session. Also great to see, human action as well. That's the first time I think I've seen context traces displayed in in that in that format, which I think is very, very useful. But, yeah, it was definitely, you know, a commodity, for for me, you know, to have the opportunity to navigate and check the data, by the funding actions in, in... This was definitely, helpful. And, yeah, you know, the rest, you know, in terms of the workflows, etcetera, also helping me to to keep the things Yeah. Great. Perfect. Cool. I suppose with that, I'll, I'll bring this, this workshop to an end. Thank you, Alessandro, for your time. Really appreciate. Thank you. your your expertise very much. matter. If anybody has any follow-up questions, you'll get an email coming through to your, inbox in the next day or two. Absolutely. We'll in links to the, to the the the presentation to the code sample. You can, follow all of the the rest of the the training series and also catch up with the ones we've already, run so far in the series at nifshare.com/nodes and also find out more about, the conference. And with that, good morning, good afternoon, good evening. Thank you all for your time. Thank you.