Name: Stop AI Agent Hallucinations: 5 Techniques + Production Patterns
Uploaded: 2026-10-08T18:16:14.391Z
Duration: 1 h 43 min 28 s
Description: Stop AI Agent Hallucinations: 5 Techniques + Production Patterns

Transcript for "Stop AI Agent Hallucinations: 5 Techniques + Production Patterns": Hello, everybody. Good morning. Good evening. Good afternoon. And welcome to the road to notes. Second part of the... Oh, it's not third third workshop of the series, but we did the actual second second actual proper proper workshop because we did an intro to Neo4j and graph, and Gen AI last week with Martin. Today, I'm super excited to have, Elizabeth Fuentes Leone join me. She's a developer advocate and AI engineer at AWS. And, she has made, herself very known as making agents, turn agents failures like hallucinations and and and and token misuse, into... And fix... Fixing them. And she shows us, a couple of text... Techniques today in the in the workshop, how you can use a graph rack to ground answers, how to make, relationships between entities, how to use tools to cut down on your token spend maybe, and to find out what you can do to make your progress better, your agents give you better answers, and steering all of that into more correct, validated, grounded answers in the end. So you can work better with AI and Neo4j potentially. You will see, an in... A link that I will post in chat in in a second that you can, follow the... All the steps that Elizabeth will show you today. She is, gonna be walking through the workshop today. It's... It might be a little little quick to follow today, but but she'll explain to you everything you need to know, and then you can you can follow follow along if you like or at least watch it today, see her do do the exercise, and then afterwards, everything is gonna be available. So you can do it, at your own time as well in the in the next, you know, next step. The session is therefore, obviously, recorded, and you can watch, this video afterwards. I'll send you an email with the recording after this is done probably tomorrow, with the link as well again that that Elizabeth has prepared, and that is very, very concise and have everything in there. And, if you have any questions, if you have anything is unclear today, we'll try to answer those question. ABK Andreas from Neo four j is is in chat with us today as well. So we will have some some, experts here, during the workshop that can answer your questions. So please use the the q and a function that you see on the top right of your screen and put your questions there, and we'll see if we can answer them. I cannot promise that we can answer everything today. There's always the community forum and the community, to help you out with something afterwards. But, we'll try to get through as many, questions as as possible. Yeah. And with... Without further ado, Elizabeth, you you... Yeah. Is. she? Hello, Elizabeth. Yeah. Welcome. Hi. Thank you very much. much for for for joining us today, and thank you very much for for preparing the... Already in advance, preparing the content, preparing this this amazing workshop today for us. And, and, to guide us through, how to make your agents a little bit smarter, how to make your AI a little bit better, and, how to avoid stupid stupid answers like, you know, 15 guests into a ten ten person meeting room, or make up statistics that don't exist, or or quotes from people that you you know that they're not right, and still they come up. So I'm I'm I'm very excited about this. Thank you so much, Alexander. I'm glad to be here with you. And, you know, I feel like, I am like the black sheep because I am the only person who know... Who doesn't work for Neo four j, but I really love to deal with you guys. So that's why I'm here because I've seen that you can use using Neo four j to avoid a stupid question and to avoid agentic hallucination. So, let's. let's start. So, this watch all the information that I'm going to share with you in this, I... Two hours, I hope, little less, is here in this QR code that you can see here in the screen. So you are familiar with virtual studio from AWS. We usually will open their own accounts so everybody can use the service that we we show here in this virtual for free. But today, because I didn't know how many people are, are going to have in this in this, in this event, I'm not going to open the Bernanos account right now. But you can send me a link in Messenger. You can send me a LinkedIn request first, and I'm going to to accept all the request. And tomorrow, you can ask ask me about this workshop. So I'm going to open a workshop in studio tomorrow. So you can run everything that I want to show you here in the week. So please, today, only clone the repo, follow the steps, understand. I want to give you some tips and everything, and then in the weekend, I hope you have time, you can run the workshop by your own. So let's start. Okay. So today, we are going to talk about four four hallucinations that we are going to try to fix today. The first one is fabricated statistic. So sometimes when you use rack engines, the rack engine, it gets known. You know? Because when you do a a rack, a retrieve in a vector database, it only give you the key, means or the key, retrieve for for that vector stores. So you don't... You... The the agent doesn't doesn't see or it it can see all the information between the rack... That... Between the vector stores. So if you ask something related to how many, it's only give you the... That how many in the. The agent is not going to see everything inside that. Sometimes we have wrong term selections because you have this agent too much too much tools so that it can start to confuse in all the tool information. So it's going to, pick the wrong tool. And another problem that, is there is if you have a lot of tools, you have a lot of tokens that you are wasting for there. We are going to see that. And sometimes you give it to this agent, so this is the rule. The agent doesn't care about Because when you do this, when you talk with the agent to the system prompt, for the agent, the system prompt is like suggestion. So if the element is smart, probably it's going to say, well, you know, this system prompt is good, but I have a better idea. So system prompt is suggestion because LLM is probabilistic. So we need to put the rules in a deterministic way, and we probably can have some undetected failures. So undetected failures is when one LLM is able to answer what question. So for that, you can put several LLM, several agents that can help you to understand that failure, to detect the failure. You have, a question? sorry to? in... Yeah. Sorry to interrupt. Are you sharing a screen or are you Oh, sorry do you want. to share a screen? my god. I'm. sure. Thank you. Yeah. Thank you for telling me that. Oh, you know, I was. looking here. How long? Yeah. And I was waiting was not see the... if you... Yeah. Yeah. Yeah. I was. sharing problem. my screen. Yeah. You know, Yeah. I don't see the bottom here. Sorry. I was thinking that my screen was sharing myself. Wait. I'm looking at the, bottom, in, the top. top left top left where it says. screen. Yeah. Yeah. I already see. I already see. Yeah. Oh, Yeah. sorry. about that, guys. Sorry. I was talking here alone. Now you can see my screen. Sorry about that. Yep. Here the QR. Perfect. the QR you very much. is going to be in the... All the the slides. So it doesn't matter if I'm going to continue speaking here, you are going to follow the QR code in all the slides. So, well, this is is all the four, hallucinations, failures that we are going to try to fix today. And the solutions that I have for these failures are... The first one is the rough track using Neo four j, of course. That's why I'm here. Right? And for the tools problems, we are going to build a semantic tool search. So it's still the agency, all the tools that you have. It's a... The agent is going... Is only going to see that only three top relevant tools that the agent need to answer the question. For, validate, we are going to use a multi agent validation, a multi agent architecture. For the business rule, we are going to use narrow symbolic guardians, hooks to blocks everything. And sometimes, we want to have, like, a soft business rule. So for that, we are going to steer the area to give the right answer. And, we're we're in this workshop. We have six hands ons module, the the five, failures, and the last one is to put everything on production with less than five minutes using Amazon Metro engine core. First, to build all these engine, because we are going to build one, engine for all this case, we will use Strans agent. Strans agents is a framework that is completely, completely open source, and we, as AWS, we are the maintainer of this framework. Yes. It's from AWS. Yes. It's completely open source. That means it's free. Yeah. You can install that, and you can start to use a strand sending, and it's free. Yes. Again, it's from AWS, and it's free. So with strand, it's completely a model driver. You can use whatever model that you want. You can use locally if you want. You can be like, and we have few lines of code. And the difference is you can build a huge loop, something super powerful with two lines of code because a strand, it build all the nodes, it build all the agent loop behind the scene. So the only thing that you need to add is, like, little tools, the hooks, the steering inside this agent like a little legal pieces. And to put any known products up, sorry, because it's one SDK and any model, you can use any model provider. Amazon, Pedro, Entropi, Gemini,, or whatever you want. And for, to build the engines, this is the line. So you build the tool with the tool decorator, a simple Python code in this case, but you can use strands with TypeScript as well, and you build the agent like this, only one line. Engines, engine with the system prompt, and the tools. And it's like... It's usually the agent like that. You will use Amazon backdoor behind the scene. But if you want to add another model provider, you only need to add the model provider, and that's it. And AN core. What is AN core? So AN core is a service dedicated from a product against some production. So we have, like, another LEGO pieces of LEGO that you can use together or separately. But in this workshop, we are going to use runtime. Runtime is like the container that allows us to put our code there and to put this engine on production with a single line. And behind the scenes, runtime is going to build everything for us. It's going to build all the security, the scalability, everything. The only thing that you need to concern is about your agent, and runtime have observability built in. When I say observability built in, it's agentic observability about how many tokens, about how many span, what this agent is using these tools. Everything is there, and we are going to see that too. And we need to have short term memory built in. When I say short term memory is that you can have a conversation with this container between fifteen minutes up to eight hour. And if you want if you want to have more memory, you can use agent core memory. And, in the workshop that I want to open for you tomorrow, you can deploy agent core memory and gateway. Because if you want to build a semantic search, you, we engine call gateway. You don't need to build a semantic search. Gateway build that for you. I will show you that too. So let us start for the fun part. Let me share my screen, and this is the best thing because I'm going to show you some code. This code, you can see here. Let me go... Let me show you something first. So this is the URL that I'm sharing with you. So you... If... When you go to this URL, you will see a lot of content here. There is a bunch of content that you can see in my social media. So send me a link in your request, and I will share with you the the workshop tomorrow. And here, there is some AWS credits. Yeah. It's about 50 or, it's about 50 credits that you can ask, and I don't have, startup credits. Okay? A A lot of people send me LinkedIn request. I don't I don't work on a startup, man. I work with community, and we we don't have a lot of money to share with you with AWS credits. And this is how you can put these credits inside your AWS account. There is some Neo four j documentation. There is some blog that I love from Neo four j. There is a lot of resource, and and the... What? The... This this is the repo that I... This is the repo that you are going to see inside the virtual studio, and this is the repo that I'm going to follow right now. And here, this is the virtual. If you go to this link, this is the public website for this workshop where all the information that I, all the explanation for each techniques is there. So here is the five techniques I just tell you. So let's start for the first one. So what we are going to do in this repo, I'm going to run a utility notebook. So first, I'm going to build a vector rack. A normal vector rack, I have 300, hotels with a lot of information. So I want... Each hotel is going to be, a chunk for this vector stock. And then I'm going to, every time that I ask a question for this vector, I need to be. And then I'm going to receive the the top the top three chunks for this table, and then I'm going to compare it this agent with this... With a different tool that is going to be the same, but instead to us... To the vector rack, it's going to us to our graph rack. And I have this graph rack database here locally. I have a Neo four j running locally here in my computer. Yes. Let me show you here. This is my local store. You can see here this Neo four j running. This is all my hotels. They have, amenities, service, room, policy. Everything is inside this... In this new frontier. And you can see all the hotels here in the repo. Just right here, this is the first one. So here is the data. So here, there is 300 hotels. Each hotel, we have, the name, it tell, rate... Rating, rooms, a bunch of information that we have inside this storage. So let's start with the code. Here, is there not good that I want to run for this? Yeah. I'm, sometimes I know that it... No. Everybody love to... Let me put it the smallest smallest. No. Not everybody love the the logo, but when you are running a workshop and you want to explain, Jupyter notebook is the best way to explain step by step inside some applications. So here, I need to be sure that I have my WSL credential because I'm going to use Amazon Petro as a model provider to my embedding model to build the vector structure. So this is... This function just right here is the function that I'm going to use to, every time that I receive a text, I'm going to send that text... That query to this embedding function. So I'm going to give embedding, using that question. This is a bunch of text. Right now, it's it's not... Is is not important, and I have two different tools. One tool that is going to do the retrieve for my, vector storage, the search facts, and another that is going to do the query for my Neo four j. You can see here in the context for this tool that I have some, they have a field shop. So how this, tool can build a Cypher query? Because I'm going to use Cypher query to do the search in my rack... In Minecraft rack. So that's the thing. When you ask how many or other as as some number to a rack store, a normal vector storage, it can do that. It's going to do the retrieve. You are going to receive a few information that is going to be the the probability answer for your question, then the LMM is going to be the answer for how many they calculate... The mathematical calculation using that. The LLM is the one who is doing the mathematical calculation. But if you use a cyber query, the... And the LLM is building this this cyber query for you, you are going to receive this exact answers because cyber query, they are capable to build a branch, to do calculation. And the LLM is not going to salvage case, not going to build a mathematical calculation for you. Some LLMs, they are good to do mathematical calculation, but another one, they are not. And the thing in build a good agentic application in a good system, a good harness, what is the number that they have today, is if you have a good harness around your LLM, you are not going to need a huge LLM to give you a right answer because you can have a huge LLM, and you don't have a... If you don't have a good application around the LLM, you are going to receive a bunch of hallucination and stupid answers. You are going to waste talking as a crazy people, and it's not going to be in the web the best, the best application. Okay. So we have two tools, and this is something that, to connect. This is my, solution. This is, to calculate how many docs I have, and this is how I connect to my local Neo four j. Let me see here. Let me run this line. So I have two different tools. My files, because my directory store is locally, it have 300 documents, and Neo four gs, I have, less white because I know... Because in this hotel database, I have some duplicates number. And Neo four j, it can understand. It knows that I have some duplicate data. So it's... When I see that it's it's it's the... You know, it's match the the same note, so it doesn't build another note for a duplicate data. Now how I view, the... Their agent for this application? In this demo, I can use a Pedro as a model provider, but I'm going to use OpenAI. Why? Because I can. So I'm going to uncomment this. This is the same code that you have there in the sample. Let me uncomment this. I have here locally... I know that I have the, my OpenAI key. Let me put this here. And another thing that you need to do, you need to be sure if you are going to use another mobile provider, not only OpenAI, is you need to see the requirements because I already start that, sorry for the noise, and in a hotel room, and here. So I have my strands and... Yeah. I need to uncomment. Oh, I don't see my, OpenAI right here. I need to uncomment this because I I want to use OpenAI as a good provider. If you want to use Entropi or Lamad, Gemini, whatever, you need to be sure that you have the requirements for that libraries. So on the middle on this, it doesn't matter because this is this is going to... So here, what I have, I have my rack engine. I have the model. I can I use it in the open air because of the second one? The system prompt, and I have my tools. I have... And then system prompt is you are a travel agent that help people with hotel. Keep answers consist at the most two centers. The same agent, the only difference is the two. So let's start to use that. So we have, what is the average guest rating of hotel in Las Vegas? And how many hotel do you see? Because, you know, I need to know how many hotels. So let's run there. And inside the strands, I can see the... How many tokens. So this this implication here is going to give me how many tokens I I going to spend in this query because it have open telemetry built in. So I don't need to build anything beside that because I I have open telemetry. Build image side is, let me see. It's... I have everything. Built... Yeah. The telemetry built inside the frame. So for the five spectral store, it's the three hotels because the 20, it only give me the three best probability to answer the question. And it's it's... The LMM, it's it's doing the calculation by the young using the... Its knowledge about math, and it spend this amount of tokens in and out. And for Neo four j, it see... It it... It's give me the old average because Neo four j with the cyber cyber query, they can do that. You know? The other name, it's using the tools. It build the cyber query to calculate average, and it give me the number. And he told me, hey. I'm using five hotel for this. So the rock, it is... There... It's not going to see the five hotel. Never. So... Well, and it's giving all the names. And another thing that you can see here is when you use cyber query directory, you will spend less token. So this is not something only about hallucination. This is something about token's economic. So if you build a good harness, you're not only going to avoid hallucinations. You are going to avoid waste tokens as well. Let's continue with the next one. How many hotels have swimming pool, and how many hotel do you check? So this is our barrage, and this is how many. Let's round. So five. The number of hotel with swimming pool,. Yeah. Because I have a chat history here. It's not specific, so they don't have nothing. And, man, what... He told me nothing, and I have... Here I have this amount of hotel. I don't know, man. So... And for the other one is tell me about hotels in Antarctica. There is no hotels in Antarctica. So LLMs is because this question is going to be about the LLM. Is is this LLM is going to do the retrieve in the back to a store, and, yeah, it's going to receive a zero answer about that because, there is no hotel in Antarctica. There is no telling. And the LLM is going to use the answer that the retrieve it gave you, and it's going to fabricate an answer for me. When the Neo four j into the the the the Cypher query and start to do the query for this vector data. The data store is going to receive zero. Now crunch. So for the first one is, there is limited information. Come on. 10,000 tokens to tell me there is a limited information, there's nothing to say, nothing. And the other one, 3,000, there are no hotel listed in Antarctica. That's it. That's it. That's it. Talking economic. And another example, this is to see the, exam... The the Cypher query. And you can try it yourself, crawling, test it. You can test this locally using your Neo four j locally, or you can change the environments here and, put your Aurora DB data if you are using a Aurora DB Aurora DB data data store here. Now let's continue from the other one. So, I will have some question here. Nope. Nope. No question. Error, something went wrong with... Oh, sorry about the credits. And, Edith, probably, I don't have any more credits there because I I think I have a, 2,000... 200 credit. Probably, I don't have anymore. My piece, Edith, Send me a link in your request, and I can share with you credits and the virtual URL. Super. And this is not only for evidence. This is for everybody. No? Super. Thank you. Now I welcome to continue here. So now semantic tool selection. So let me see here. I blame it again. I want to run this, and I want to explain you what I'm going to do here, you know, because this is better if you see this in a, a visual way. So I know that I have this in some video. But it's my work choice I missed here. So if you follow the work choice studio, the next one is picking the rear... The right tool. So when the engine, it see all the tools. For example, in this example, in in this, this code, we'd have 40 overlapping tools. There is a a bunch of dummy tools with... I I try to do my best, okay, with these tools. So every time that you invoke the engine, the engine is going to send all the context for all these tools to your LLM. So the LLM is going to receive the tools information plus the system prompt plus the chat history. It's going to receive everything in each invocation. So if you know, the LMM, they have a contact window. And the contact window probably is is huge, but it's not infinite. So the engine can lose in that content window because every time it need to understand everything inside. It's going to process everything, each tool, its tokens, too, that you are paying for. And there is some, theory, this paper that is say that the engine, the contact window, it can lose in the. So you have a, and the engine is only going to remember the first part of the and the last part of the and it's going to lose the information down. So if the information that you need, the important information, is in the youth in the youth team, your agent is going to answer a bad a bad answer and probably is going to fabricate an answer for you without using the information that it possibly have. So we are going to build two different agents, one with all the tools and another one that is going to use the vector store for the tools. But in this example, we will use... Let me see if I have here the image. Not here. We will use the hooks the hooks. When I say hooks, the hooks is something, is a... Let me show you first in the code base better. The hooks is a little pieces of code that you can put between the invocation. So you can use a hooks before to invoke the LLM, after invoke the LLM, or before or after to invoke tools. In this example, we are going to use hook before the LLM invocation. So we can filter the tools that this engine is going to send to the LLM before the invocation in the code. Let me show you. So here is my hooks. So we have a before tool called event, and we we, we integrate the hook provider for the Strans library. This is something that Strans have built here. You only need to build the class and the code for this hook. And, we will use, Amazon bedroom. I think this model, I prefer this one. Wait. Let me see. Let me see this work. So let me run this first. And this is the... This is only... You know, because every time, I want to show you not only about hallucination. I want to show you how many tokens I spend in this, and I want to show you, what tools this agent is using because we need to compare it. So I I already have here. That's a one of two. Now let's build the engine with semantic tool search. So I build in this class. This is my my hook, my code for the hook. So I have here in this craziness, code, I have my my... I put in here the... I... The vector store for my tools. So the vector the vector store is the name of the tool and the context, the description for the each tool. So every time that I receive a question, I'm going to convert that question in a vector... In an embedding. I'm going to search in my vector store, and I'm going to receive the top k tools that I can use to answer the question in this engine. And I'm going to put those tools inside my agentic application. So let me show you in the in the engine because this is a lot of code, and you can review this code in the repo. So this is the thing. So I send all the tools to that huge code, and I'm going to receive the top k. And this is the question. I'm going to send all this question to test my hand, and this is, like, the ground tool. So I have the tool that I think that I know that is the tool... The right tool to answer this question. This is, Genuity test. This is a demo, and we want to know what's happened with this agent. Now this is our agent. So we have this agent. This is my agent. So this is... I want to use the the question there, and I'm going to use... I... I'm using a form. One thing here. Every time that I run this form, I create a new engine. So this engine, every time that I send the the question, it's going to be a new engine. The engine is not going to remember the conversation because every time I create a new agent. So I have the model. I have my system prompt. You are a travel engine that help the same. And I have the tools. I send me all the tools, and this is the tracker. So this is... It's wrong. What I do in here, I'm sending all the tools to this engine. This is my my normal agent that is wasting talking like a crazy. So in this, the first one, is spending this amount of... It's all... It's almost the same amount because, you know, it's the same amount of tools, and it doesn't have a chat history between the the ends. You know what happened with the accuracy? So because it received all the tools, you can see all the tools here. Oh, where are my tools? I know. Sharpie or semantic tools. No. This is to build the semantic search. That's another thing. Well, the tools are there. And... Wait. So I lose my... This is the other one. This one. So... Yes. It's not... We don't have a either. And this is another. What happened when... I I have almost the same agent, but the difference is that I added the tracer. This one is the one who is going to put the the semantic search in these tools. So what... Let's see what is happening. It's the same I'll... The only difference is that hook. And, yes, you know, we have less tokens. Yes. Come on. It's a lot. It's a lot less tokens. So every time that I invoke the same, here, yes, and send them, send them the question to the tool vector store, and it's going to give me only three tools. And every time, something here report. Every time that I regulate this agent, I pull in the tools inside the agent because agent... The strands area allows me to do that. I forget to explain that story. In the hook, you can see something that is calling register. So in a strands engine, it have the register. So we'll put the register here. With this line, if you worry, if we go to their documentation, we can see that using that... Let me go here so I can show you in the documentation. So using that, you can't... Oh, what has happened here? Again, well, let me show you another another website. Using that, every time that you use the tool register, you can swap the tools. So you can't register a new tool inside your end. So before to send all the context for this LOM, the code, the Python code, this hook is going... First, it's going to look for the best tool to answer that question. It's going to find the three names, the three tools names, and it's going to register the names inside the engine loop. So when you invoke that... When you invoke the LLM with the engine, you invoke the LLM, it's only going to send the three tools that the alien needs. Let me show you this. So to register, trans... So you can see that in the, documentation, this is... I think this is, Python. So tools, you can release there. This is the the the tool decorator, And... Well, here. So when you use this, you can reduce... You can add a new tool in each loop invocation. Now, you know, this is a super dummy agent because you don't have conversation history. So it... Yeah. Did you know this is like, oh, yeah. But, you know, every time that I build the engine, I put the tools, and that's it. But what happen when you want to use the same loop every time when you want to have a a conversation history, when you look... When you don't want... When you want to, they use the same agent again. So you can swap the tools. So you go the agent the first time, you do the retrieve for the tools, then for the next invocation, you send the next question. You are going to receive three different tools, so you delete you you delete the old tools for your engine loop, and you add the new tools. Yeah. The thing is the engine is going to remember... Is going to remember because it have in the context window the old tools, but it's not going to receive all the tools again. You're going to... It's only going to receive the tools that you need in this loop. So how this work? So we have the tool tracking. So all... This is, yeah, this is for... Yeah. We already do... Did that. I know this is a lot of information. If I have questions, please, tell me because I noticed it's it's like too much. Empty tool with search... With memory. So what we have? First, we are going to add... This is the file... The file session menu... Memory sessions. You can add this, like, a a little storage for all your conversation. I'm going to build a session ID and a session storage here in my computer, and I use in the hook and the session ID. So, yes. So, you understand the one question? Maybe, there is one one question. that that came in about, you know, how do you validate or how do you rank what you just said, token controlling, for anti hallucinations in AI agents? Do you think this is... At the moment, we are, I think, more saving saving tokens, right, because we are limiting the number of of tools, an agent can use. And GraphRack, as a as a tool would then avoid anticipations more? Or how how would you how would you think, that that, that sits in the the flow of things? The question is on the amount of tokens influencing the hallucination thing. If I have a lot. of tokens, a a lot of, yeah, you know, there is... There there you go. You say that if you just eat... You know, it's something that you can, that you can see when you have a a conversation with your assistant, with your cloud code, with your codex, that when you have a a large conversation with this assistant, with this LLM, this LLM is going to have more context, more and more content. It doesn't matter if they have a huge contact window. It's going to start like, it's going to be dummy. Because every time that you invoke the engine, you are sending everything to this engine, to this LLM. And the LLM need to process all the information first, have first before you answer your question. So the LLM probably can have a lot of information in the contact window, but if the information is a lot, the LLM is not going to remember everything there. So it's going to start to fabricate in, the answer for... I don't know. It's it's going to use one half of the information, and it's going to start to fabricate the answer using only one half of the information that we have and that that you suppose that we have all the information. So it's going to lose the information that you send it. So it's better to control the information that you send to this LLM. So this is context engineering. Using context engineering, you can control the amount of information that you send to the LLM and only send the information that it needs to give you the answer. Don't send everything for that. Makes it. sense. Thank you. And. another thing, you know, this demo is obsolete because I'm using a vector store to search the the right... The... To to, yeah, to calculate in a probability where in a vector search the right tools to to answer the question. But about two weeks ago, Jeff exists, and Jeff is a profitability airline. So when you build a vector store, you need to put money in the embedding model, please. First, building the vector store, and then in free time that you go that you go and do the retrieve for the vector research, you need to pay for the embedding as well. So using JEF, you can avoid that problem, and I have another another code for that. So... And only, Jeff, because yesterday, OpenAI then launched the... An OpenAI decision model. You see, I think... I I don't remember well, but I see here's in LUMA. So this is not an open decision model. It's a open decision API because what's happening? So when you are using a a decision model, the answer is not the same. So if... When you use LLM, the LLM is going to give you text. It's going to give you a conversation that you are going to chat with the LLM. But if you are using a a decision model, the decision model is going to give you probability. Probability that that thing is the right answer. So the API is different. So you can use the same LLM with OpenAI but with a different, API. So they launched this. The... I I don't think it's in GA. I think it's it's better. Using Luma, you can go and use it right away. They launched that yesterday. And I... And we, as I do, yes, we have another one that is the strand decider model. So you can use this strand decider model. It's completely open source. You can go to Hugging Face and download this model, and you see it locally completely free. That's the one. This is the same agent. You know, the same with all the rules, with all the code. And so this is the same with the... All the the 30 tools. This is my five vector. You know, we are going to receive probably the the same. I'm going to send the the question. It's going to search that in a vector store, and I'm going... It's going to calculate the billing, blah blah blah. So this is the age. I have the agent, the tools, the system prompt, and my hopes. So I... This is the question. It say at the hotel rooms. So it's using these hotel room rates, book tools and get hotels a minute to answer this question. That's the tool that is using for that, and it'll spend this amount of token, 1,000, almost 2,000, or almost 2,000 to the to the total. So using... I'm going to start... To do this random model because my... This computer is... I don't know. I don't have the the RAM for run that, but you can do it if you want. And this is the JEP, using JEP through OpenRoutes. You know, I know because JEP, you you can ask... Access to JEP, but you can use him through OpenRoutes. So I have a open... OpenRoutes API. It's super cheap. It's not so expensive. You can use it. And this is the same agent you can see. Agent model, prompt, and I have a Jeff Hook. So this Jeff Hook is using the, the open this... The the the vocation for this API, and this is a different API because I'm receiving a probability a probability. So you can see it's not... The difference is not a lot between the tokens. And I have the get hotel rooms, and it's using less... Using only one tool. It doesn't use all because I receive in here only three. I receive in three. And here with Jeff, it only give you one, and I have the question that I answer right away. So we open it. It's the same. I have a different hook. You know? I... You can see that I'm running almost the same agent, and the only difference is the hook. So because I invoke this hook this hook before to send all the information to the element, so I run this code. I received the the right tool. I put this tool inside my l l... My agent loop. I use the the tool in the l l... The agent loop to answer my question, and that's it. The difference between, conversation is not is not too much. And this is a, what's... Semantic search. I know it's a little... A lot, but, please, if you have a question, please ask the question, and you can follow everything by your own this week. I I lose the the window where I have the question. I don't know if I have. Everything is okay? Yes. We are okay. And Okay. We're good. the hard, work. I think... Yeah. Yeah. Hardware requirements, was was one, question. I don't know if you wanna go into that. No. You know, I don't have the answer, but let me let. me switch. the the the because this computer is the computer I used to travel, and it's super small. But I. want to share with you the block. This is the block about this new strand designer model. So the thing with this model, it's that they share all the data. So in that... In the in the whole interface, you can you can find all the data that the team used to train this model. So you can use that data to record a new model. And, for example, let's say that you have, I don't know, your business is related to bank. So you can retrain, do like, yeah, the training for this model using your business case. So you can have a model decider with a specific, decisions to bank. It's... And you can have that. You can have a small decider in your computer, so you don't need to pay. And another thing, I'm using OpenAI, and again, using Jeff, but every time that I send my tools to this model, I send them my data. So if you are using a local decider model, you don't need to share your data. So for bank, I think it's a good a good use case because I know some bank, they have some security problem. They don't want to share the data with OpenAI, for example. So let's continue on here. So now we have multi agent. So we have three different agent with three different models. And let me search my virtual studio. So let's come back to the explanation. Here. Here is... So this is the tree. So we have tree model. They, they catch convenient guesses because if you only have one agent, you know, one agent, they they they control everything. It's only always going to depend on the use case. So you can have three different agents that can cache the hallucination. Of course, you can have one. And the the next question probably is, why not using a LLM as a judge? Because when you use LLM as a gesture, it's only the LLM. So when you have a different engines, you you can have a different agents with the same tools, with the same agent loop, with different LLM. Or this... Can you... The second sample is that I use, like, three three different engines to conceal... Conciliate the answer. Bam. You can use, I don't know, LLM, a swap. They they share information between them, and they can have different tools to to to review the answers between them. In this one, it's a cons... Consortium is the work. So I have three different LLM inside three different engines with the same tools. So each LLM is going to process the answer in a different way. Sometimes, I go... You are going to receive the same answer, but Sultan, not. So let's try this. And, what else I can tell you, before too young? That's it. So let's continue to the to the code here. So it's the same. I have a notebook. We are almost there. I noticed too much I noticed too much information. So let me clear all the outputs. So I'm going to check that I have everything. So here, I'm going to build a graph, a graph multi agent application. Why I'm going to build a graph? It's not the same like a Neo four j graph storage. It's a graph agent. So I have a graph agent because I want I want to have, like, a a parallel invocation. So I want to both my agents in a parallel way. If I build a swamp, when you build a swamp, the agent share information between them. I don't want to share information between them. I don't want... I want that they go and search my answer in a parallel way. So at the end, I can I can have this consortium that's in? Yeah, I can have the average, and I can calculate the best answer for my question. So here, I'm using three different models. I have CloudHowCo, HiCo, NovaPro, and NovaLine. This is a, this is some model that we have inside Amazon. They are not the best. So Novaly is the the the more small. So the the thing is that the trade and the... These trade models are super different between them. And this is the one shot. You are a travel agent that help people with hotels, answer by naming exactly one hotel for the list, and the task is no ceilings. I... So I have my new my new four j locally. And, well, this is the the query and. What else? Yeah. My Spanish. What else? So, yes, I have this. And let me show you how, we can build it. And this... And this is the question. So where is my agent? Sorry. I forgot to show you my agent. So I use the strand, the multi agent. Oh, because I'm using Neo four j because Neo four j is one of my tools. So I want to use Neo four j to retrieve the answer for this question, and I can I can find my agent? This is to calculate the metrics. Oh, is this... So what... The the thing is that you understand how you can use the agents to give a consolidate consulate answer. So this is my answer using Nova. When I imported mobile model, I had this stolen collection, and the Nova lite, it give me the other is, here. So I go... Yeah. If I lose any company, yeah, it's a different... I have three different answers for this question because it's building... The the question is not so, so clear. So you are... What is the question? I forget my question. Here. Which hotel is the best for luxury travel? It's it's not a lot. Probably it's it's some... So LLM, it it seem that it's better to calculate, between the the CISA for the hotels. Others say if you have bigger rooms, it's, you know, it's only... It's going to depend of the probabilistic of this hotel to say what is the best luxury travel... For luxury travel. So it's like a hallucination because models disagree. So if you are... You have this travel assistant, and you have three different models, and the three three different models answer a different, different, answer, so probably you need to build a good... A better application. So... And the next one is which single hotel best match the color group? You know, this is a super... It's like when people start to answer a stupid question for the chat box and and see what this chat would answer. So, again, model is a way. You know, the question is super random. And, yeah, because I already see three different question between the agents. And this is the last one. Which hotel is best for a solo business trip? And, again, different answers because, these these questions are super random. It's like, to test my graph and genetic application. So the thing is sometimes you are building like a business case. For example, I don't know, when you have, like, this doctor application, you need to be super, super sure for the answer that this agent has given because this agent is giving some medical recommendation. I don't recommend to build that type of... Ask the agents, but people are selling that. So if you want to build something like that, you need to have, like, different agents. And probably that at the end, the the the the the answer, you can... At the end of this answer, you can use another LLA message, Josh. But the thing is, this is this is some information that you can have when you want to build some customer service because, Sonia, and another thing. So you can put inside these hooks, you can put some... Inside these agents, you can put some business room. That's the same... That that we are going to show right now. I want to show you right now. So the neurosymbolic demo is when you control the... Come on. I have everything here. When you control the answers in a hook so you can catch the answer that the LLM is give you, after the invocation of the LLM, or you can catch the question before to send... Before to involve the LLM, for example. Let's see. So I have my my, New York four gs vector store. I have my engine, my tool for strands, and this is the bookstore. So you can see here that this is a bunch of Python code. So what is this? This is write reservation. Because I'm, I don't have... This is locally. I don't have a a pick... A a database for my reservation. I want to book something. You know? I want I want my agent is capable to book something, so I build in this booking JSON. Wait a moment. I think I'm having problems with my search. Well... And I want to build this booking storage here in my computer. And I have a book hotel, and I have process payment, and I have configured booking. This is some tools that are inside of this class. And what else I have here? So the prompt knows nothing about each hotel, so it doesn't have nothing about the real cap... The real cap... The capacity. Everything lies inside my Neo four j. So in here, I have the tools. So I have the the same tools I told you there, and I'm going to use Anthropic as a model provider, as a, a more... Amazon Metro as a mobile provider, and I'm going to use Entropic through Amazon Metro. Now you are a travel agent that helps people with hotels. Keep answers consists, at the most, two centers. So let's see. I have my model, my tools, and my pro. Nothing else. I have all my three tools. So probably the the other name is going to do all the reservation is book, book it. But it's in. No one paid for that. And how many guests I can have in a single room? Well, no one know that because the AM doesn't have that information. See, the only thing that it can do is book anything. So there is one thing. You can put a rule in the in the in the gentle loop. So for example, you only can have mass guest per Room 10. And this is my mass guest my mass guest, room. So here, I I go into... This is to to do the query to my new Fourier, and this is my hook. This is the... This this is where I going to add my hook. So, what else? So I have the number there, and this is a a code that validate the reservation video. So that... And we can check how many guests is guess... Is mass guess. So... No. We can we can... We block. So if someone can and they want to have, I don't know, a room for eleven eleven, guesses, this engine is going to block the query because the engine, the LLM, this this modem, this, service, this hotel, it doesn't have room for more than 10 people. So let's run this. Now this is my agent, the model, and the only thing that I change here is the hook. The same, and I only add that that code here to my, agent. So run this. You see, I created the agent. I had some problems with Internet. It's taking so long. And now I'm going to send the question. What happened? Oh, yeah. Because, I connected with the graph. Probably having some Internet problem. I have some question. Meanwhile, these things work. No question. Are you following me? I know there's a lot of a lot of content. Probably, it's too much. Yeah. Your your Internet is a bit choppy at the moment. I think you were a little bit... We're losing a little bit of your, but here, I think slides are back. We can answer one question from Jorge. Two question from him. For misunderstanding rather than tool choice, do you resolve the user's mention to a graph node ID before querying or let the agent decide? What happens when a name matches several nodes? Will the names. match several notes? Wow. That's a, I don't have the answer. Andrea is there. Andreas Probably there. Andrea. Yeah. Maybe Andreas can can take that on. Yeah. Yeah. Because I'm I'm building... So with this, the thing. that I'm doing. with. the course, that I'm building, a a cyber query. So they. seem to have several matches for a single cyber query. Is... The problem is the... How would you build a node? If you have several matches for a single Cypher query, probably the problem is. that the the graph storage is... You know, they don't have the best there. Yep. Yep. And, you know, I... I'm having can... some troubles with with this. Let me see. I I I lose my my... Can you see my screen? I see my... Yeah. At the moment, no. I don't want to see it black. It's it's flickering a little bit in and out, Why? but, yeah, it's. out. Lisa. Now it's back. Yes. Yeah. Well, I, don't I, don't know. we me let me, take a moment. well, the... Yeah. While you while you get this sorted, there were a couple of questions. Is this recorded? People had to join late. Yes. We record this. Yes. I will share the link to the video and to the material that Elizabeth shared today with you all afterwards. So, and, Goldcast usually also provides a transcript. So, you can you can check that out, after we're done today. So you should get that probably tomorrow in your inbox, and then you can, you can, yeah, rewatch the video, re... Redo the exercise or properly do the exercises because, yeah, as... It's it's a lot of content we know, lot of lot of lot of things, in a in a very short time. But, yeah, that's why we're recording it. That's why you, you list... You can listen now to what Elizabeth says, and then you can do the exercise. You can recreate what, what she's sharing, in your time. I don't know. I'm trying to make some trouble here because, you know, that's not normal that it takes so long. But the thing is when I'm using the hook and I ask a question, for example, the question here is, book hotels, no word for yongo, and they did... We have some dates... Some dates here. We have two problems here. The first one is probably fake hotels doesn't assist. So, when I go to my Neo. four the moment, j... we we only have a black screen. Maybe do we wanna stop. sharing and reshare and see maybe that... If that helps anything? Yeah. Only about that, guys. Yes. Yes. I'm going to. do that. No problem. Maybe that helps. Take me a moment. I see my problem is my my charger is not, charging, okay. but, give me a. moment, Yeah. please. Alright. Give. me a moment. Yeah. I think that's. my problem. Yeah. Sorry, guys. I will come back in a moment. problem. Super. Well well, while Elizabeth is, is getting that that double check. So, yeah, Andrea has answered in chat, several no problems. A classic entity resolution challenge and the flip side of entity disambiguation. So, thank you for for answering that. ABK, Yeah. I I I... I'll take this moment to to promote, our other road to notes workshop sessions that are going to happen in, every week until until notes itself. Let me share the link in chat again. If you are not registered to the upcoming sessions, then please do so. We have interesting content coming along until, the November 5. And I see your... Yeah. The web is back. Screen is back. Sorry. You know? Laptop. is charging. That was the problem. That was the problem. You. know? You can see the answer here. Yeah. So book. the fake hotel was it was struggling. Yeah. Oh, Yeah. Sorry. Sorry, guys. yeah. No problem. whatever. No problem. Yeah. Whatever. So... Yes. And... Well, you can see here that the book hotel, he blocked for business rule. So hotel... Fake hotel nowhere first doesn't exist, and I will not... Unable to book that hotel. If you can add another business rule here, you know, because this data is in the past. So, you know, first, the the the hotel is is... Doesn't exist. And this... You you can't book a hotel in the past neither. So this is... Because this, this is something that is wrong locally. It doesn't have a tool capable to search for the dates. That's another thing. So there is another thing. It's guess... I wanna put the max cap... Capability of this. So a rule for max pH guesses. So I've probably changed here. It's max six. So I put 10. Max guesses is 10 or six. Let me see. Well, I'm having a problem with my own business rule. So, ah, yes, the thing is is is that it's looking for the Neo four j, and Adobe Plaza, it doesn't have a room for more than six guesses. So you can search that inside your... These are the the the Neo four j. So, yeah, it's... First, it's looking for the data storage, and it's retrieving and say, hey, man. I don't have... Yes. That hotel doesn't have that capacity for you. So here, configure before payment. Yes. It booking, blah blah blah. Your confirmation ID. Yeah. You can. And, well, there is another example that you can use. Now, what's happened when you want to have soft rule? So for example, if you have rooms with, you know, for this guy, if this guy want to rule a book for only eight guesses, you can say, hey, man. I can only get get eight. Say, hey. But you can book two rooms, one for six, another for two. So you don't need to block everything. You can steer the LLM, the agent, to book two different rooms. So you can do that here with this steering demo. So inside the strands agent... Let me clear everything. Inside the strands agent, you have that you have the hooks. And here, let me let me see. Let me look. So you you can build rules, and you can create steering tools... Steering rule with that. So let me check this, the steering tools. So if you go to the steering tools, here, you can see that there is something inside the strands air engine that allows you to build that. So you can guide the engine with a rule if the the the questions is have the right rules. You can proceed. You can steal the handle as well, or you can still steal in the action. So let's see. Is we have the same? We have the same. We have max person Room 10. Similar things. Hotel exist. You are... Is is the same. Some hooves. Split, you can split. This is the thing. If if the amount is max, 10 per rooms, so you can steer. So you can, you can split the amount in different rooms in different rooms. And this is something that you have, so you can block as well. And let's try this, and we must guess as a steering. This is the the rule to the steering. So let's see how we how we can put all this rule... All these steering rules inside the engine. So you build the code. This is... You can see this is only Python code with a bunch of if, and then with r is only code. So you can have a deterministic answer. You can put deterministic way to answer question inside a log entry loop. So the alien, it doesn't need to guess using the prompt. So remember, prompts are a suggestion. If you only guide your LLLM with a prompt, probably, your LLM is going to hallucinate and is not give you the right answer that you're looking for. Doesn't matter. You have a bunch of tool that the LLM can use to answer the question because the LMM is going always to take that as a suggestion. So now let's go back to the Python code. So... Yeah. Yeah. We are already on this. So you are a travel agent that helps people with hotel, book the whole party together with the food guest count. Do not assume any per room limit to yourself. If you receive guidance to split the booking, follow it exactly. Keep answers consist. I always, try to use this because this is a demo and sometimes it's the only name of my god. We start to generate a bunch of text. Imagine that I have to read everything here in this in this time. So book. This is the the request. We are... We want to book this, and let's from this. So let's see my agent. So we have the the the hooks. Code, again, it's only code. And first, blocking hook. It's the fixed. Yes. Call is canceled. Nothing is blocked. Nothing is booked. So we... Again, we only add the hooks, and we add the tools. I know this is a little confused, but the thing is, for example, I know that right now, not everybody is building engine as a code. You know? I I don't remember the last time I went and opened a newspaper notebook and start writing for library, blah blah blah. So everybody now is using assistance to build agentic application to this agent system. So the thing is, if you know how to prompt a good assistant, if you know the techniques, if you know the problems, the next time that you start to prompt an agent to your assistant, you now know this. You know that you can, put hooks inside a framework to block, to pull some business code inside the code, knowing the system prompt. And now you know that you can build steering rules to steer your engine to build a good agenda agenda system. Because the thing that I that I want that you remember here in these two hours of a lot of code that I show you here, probably you are already like after, is harness architecture exist. Right now, it's not about problem engineering. Problem engineering is completely obsolete because the problem is only to give, like, the personality to the engine right now, to the LLM. Don't use the problem to put rules. You know? Because if you do that, your engine is going to probably hallucinate. Let's continue. So the question was... I don't remember that. Something that the lady took in, to 16. So block 15 guests excess 10 rule. But it say, so, you know, we can steal this. So we block for fixed it, but all kids that... Oh, oh my god. All fixed guests are booking in... Into different rooms. So the agent decide the thing. So it... You you don't you don't have to block everything. So a student completes completes task because this is only related to the to the rooms. And let's run this. So we have the... Still there. So let me... What's the question? I think I'm using the same one. And let me check. Let me start by checking that the hotel exists before making... Okay. Yeah. Because that thing only only was about the amount of people in the room. Now we need to check first if the room... If the hotel exists, then I need to check if I can book that so I have. So, yeah, I'm using all my roles here, and I have two different role books. Oh, yeah, ma'am. It's... What's happened with my Internet again? Well, it's in a moment. Another thing. So these rules are ruling here in my local environment. So what happened, you know, if you want to... You are building this agent and you change the rules. If you have the rules inside your framework, inside your code, you need to redeploy the engine every time that you change the rules. So using this plugin, you can save the rules inside a bit, adapter storage, for example, like a DynamoDB. So you can save your rules in a storage, and every time that you improve the loop, you cannot retrieve your rules in that storage. So if you want to change the rules, the only thing that you need to change is in your storage. So you don't need to redeploy your engine again every time that you change the rules, so you can do that. There, and the engine on production is using that. So let's go. I don't know what happened here. Again, I have problem with my Internet. I hope for you works. If not, you can give me some complaints to link it in. And let's see. Now this is the best fun part of this, of this, workshop. Let me show you this one because this is anime. I like this. Yeah. We have some... I have some gift here. So you see this? This is how you can deploy this on production. Tomorrow, I'm going to share with you the, credentials to do this. So it's a... It's... But it's going to be a bare own account, so you are going to be able to deploy everything, all this in completely free, bare our AWS account. So in... Right now, I have everything ready, but using my Aura DB. So I have an Aura deploying with the same 300 hotels in my Aura because, it's better because you only have to put the URL and that's it. In the environment that you are going to use tomorrow, I have... I I deploy a container with Neo four j inside that container. Why? Because that's an AWS environment, control environment. So, you know, it's like a stupid that I ask you to put your AuraDB there. You can do it. You can do it. But in that environment, you will have the the 300 hotel ready for use in that container. So let's see. Let me close this. Well, what's happened with this Internet, Bettina? If you are looking for me, that's the that's the good thing. In this repo, you have two ways to deploy the architecture. So you have... If you are familiar with AWS, we have a library that it's Bototree. So Bototree, this one, Bototree is the API that we have to control the AWS service. But if you want to run this locally in your computer, in your AWS account, in the in the repo, there is some step that you, yeah, that you need to follow to put your AWS credentials in the environment, in this in this account. Because when you run Bototree, you need AWS credentials here in this computer. In the environment that I'm going to give you tomorrow, you don't need to do that because the environment already have everything configured for you. So the only thing that you need to do is run everything because this code here is going to build everything for you. But please not only run the Jupyter number. Understand what is there. It's going to build all the lambda functions. The lambda functions are code in the cloud. It's... In this case, it's Python code. So we are going to have several lambda function, and this Lambda function, they are going to be all tools. So we are going to have one Lambda function capable to do all the communication with my Neo four j. So this Lambda is going to receive the Cypher query for my agent, and it's going to do the retrieve in the Neo4j. It's going to receive the information, and it's going to send the information to my agent. Then I'm going to have another one that is going to validate the steering rule in a DynamoDB, and DynamoDB is not... No SQL storage. And we have another DynamoDB that is going to save the data for hotels and bookings and everything, and we are going to have our container for runtime. So when you follow the steps, when you run that bototry, it's going to build everything for you. I will... I'm going to show you in a moment that... What is building for you, and we have the AM core gateway. The AM core gateway, it's going to do the, like, the vector search for us for our Lambda. In this case, we don't have a bunch of Lambdas here. But with Amazon agent agent core gateway, you can, you can put there a JSON file with all your, tools, MCP, APIs, everything that you want to invoke in your in your run time, in your... Doesn't matter. It's in it's in this container. You can use whatever architecture, and you can impose this gateway through app API. So you send the... It's the same. You send the question to this API, and the API is going to give you the best ans... The best tool that you can... Or the best MCP or the best APIs, whatever, to answer the question in that engine. Yeah. It's a little complicated. And this is the architecture. So let me go to my AWS account. So once you deploy that... When you deploy that... Let me... Here. So you need to open the account. If you are running the AWS, workshop studio, inside the workshop studio here, because this is this is, public, you don't see that. But inside the workshop studio here, you are going to see the AWS credentials. So you are going to be able to copy and paste those credentials. If you want to run distributed node locally, you can copy and paste that in the CLI if you are familiar with that. But if you are not familiar with that, don't worry. You can go here, and you are going to see something that say console. Click the console, and you will go to this one. This is the AWS console. So here, I'm going to go... You need to go to front... Oh, sorry. It's AEM core. A n core. So here are all the little pieces, all the little the little logo pieces inside the a n core, and we have the, everything. So once you deploy your agent, you can go to the runtime, and you can see your agent. This is... Yeah. Because, yeah, this is this is super old. This is the agent that this thing deployed for you. So I have the AM, the run time, FRN. This step are in the virtual studio as well. So I have a bunch of things. You can deploy several endpoints for the same run time, and this is the observability. I go here. Because I already using this account, I don't see any messages here. But if if if you are using the virtual studio, probably, you are going to see on your... In your own account, you are going to see some messages here that say that you need to set up something at the first time. That something can appear there. You need to set up that. Because if you don't set up that, you are not going to be able to see all the observability here inside this runtime. It's at this stage. But before to see this, I'm going to show you my engine that I'm deploy... That just deploy here in this account. So let's come back to the to the code. So this is all the code. You can deploy this using portal tree, or you can deploy this using, architecture as a code. Yeah. It's not... It's, CDK. It's a cloud developer kit. It's similar to infrastructure as a code. You can use deploying that using infrastructure as a code. There is a CDK. If you are familiar with AWS ecosystem, CDK is like the telephone for AWS service, and all the steps are just right here in this with me and in the virtual studio as well. So you can follow the steps here if you are familiar with CDK. You can deploy all this crazy architecture I just showed you with a single line, and that single line is CDK deploy. This is the PIP install, and you can deploy everything with this... With CDK deploy. But it's up to you. I give you the two different ways. So this is for more specialist people, and this is for people that they are, they are more bigger. So let's close everything. Let's... I'm... I want to show you the end. So you are going to see the end inside this folder. So the booking end is the end that we are... That we... I show you in the older Jupyter notebook. So you can see that I have... That I need to, invoke. I need to import the Metro m core, and I have the gateway. I... To to deploy this engine, you first need to deploy the gateway. That is something you see there in the in the bottom three code. Now the most important thing here inside this code is this, the APP and deploy the corridor. So every time that you run this container, this run time is going to run for you all this code. So you can see here that every time that you run this, you have the engine. So the engine is here inside this code. And, what else? Well, I have all my tools, everything up upside. And another thing that you need to, that you need to see here is that, their requirements. In their requirement, there is a new library. This library is this one. So this is the library that allows us to send all the data to CloudWatch, to the observability dashboard that we are going to see in a moment. So this library is integrated with Strans, but if you want to, I don't know, deploy a long graph, you can do that too, and you don't need to use Amazon Redraw as a model provider. You can use your own model provider as well. So you can do it in the thing is you need to be sure that you are saving your API key in, in a security way. Don't put the API key in the engines. You know? Don't do that. And if you want to use another framework, you need to do some configuration. The... All the all the information is there in the AWS sample, but you need to send the data to CloudWatch. This library do that for you. It's easier to deploy agent using a strong run time, but you can do that as well with another frameworks. Now this is my agent. So I'll press first. I want to... Let... Let's see. Let's test the agent. I need to test the agent first. And let me close this. So I create this to test my agent. So this is the same. I have a a environment And, well, this is to to control that. And I have this is new. I, well, this is so different model. I don't need that. Where is my room temperature? Yeah. This is the. Okay. I think this is not the... Yeah. Agent no validation. I don't care that. What are the prune? This is the strands. What is my AM core? Yeah. This is the right one. So test the deployer AM core on run time. Let me see. Yeah. Because I deploy this agent in this computer, in this environment, I have the agent called runtime just right here. But if you, if you don't do that, you need to change that. You need to change that for this value that is here. So let's come back to the runtime. Yeah. I'm going to start because this is... I know this is a video. So you can go to your runtime, and this value here is your ARM. So you can copy this to test the engine. Copy this and paste that here. So when I run this, I have an error. Let me see a validation. Call it describe. Stack. Okay. I have a different stack there. Let me use this. Where is the... Let me close this. I don't care about that stuff. So where is the code for input in? I can input the in right away. Okay. Now I did the thing that I just show you because this stack is where this, this code is looking for my ARN. Superbly, I I deploy the same with a different name. Okay. I have my ARN, and I'm going to start in both end. So search for available hotels in Paris. When I do that, I I can see the data right away, but I already know this. I I guess the the answer. Okay. I have some errors here for matching your search criteria for this. Will you like to... Oh, I can see the error in my in my wrong time. Let me see if I... It is... Give me another error using this. And move the ground hotel, please. Let's see what has happened. But meantime, I can go to the, observability dashboard. Because I run this agent, I don't know, this morning, I can see all these spans, and I run this agent yesterday as as well. So I can go in, for example, I have three different traces inside the... This this when I, I run that. And I can go to the traces. So it is... There I have... In each traces, I have to expand. Let me see what is happening inside this span. Oh, it's it's... Let me put it small. And, yeah, you can see internal operation. This is as as a small invocation. You can see all the delays. And what else? I can show you here inside. You can see the JSON five, but there's no some... There's nothing there. Let me see if this this this stress have more information. No. Doesn't have more information. Let me close the... Let me see if, last night, I had more, a better conversation with this. I only have to expand. So I didn't look... I didn't deal, a little test. But, you know, when you start to test this engine, you are going to be able to demand of token that you spend in all the invocations. Let me see if I can have something here. Probably, I can see the error that I just received. No. Nothing yet. You know, this... When you invoke the engine, it take about an up to... Oh, I have I have the the data. You know, this is five minutes. Yeah. This is an invocation that I just did, and this give me an error. So I am... I using a. So because I'm using Amazon Metro as a mobile provider, I can see how many I spend in this invocation, the amount of tokens. But the amount of token doesn't matter if I'm using Amazon as a model provider because this amount is is the... Is is inside this... The framework. It's transcalculate that for me. But because I'm using I'm a... A model... Amazon better as a model provider, I know how many cost. So you can see... Let me see if I can go to this trace. And here is the span. Yeah. Here I have more data. So you can see what happened inside that. So invoke is transcending, execute the loop. You can see the delays, how many... The times, the seconds, everything is inside this. You can see the region, the pen telemetry. Well, there is a lot of information. You can see all the information between the invocation. Probably, you can see I'm using AuraDB. So let me see if I if I can see here the the invocation, how I can close this. I'll close this one. Yeah. This is this is super small. But if I put in this bigger, it's worse. So it's... You know, I have, all the tools, all the evocation that it did to give me the answer. I have a success. So execute a Cypher query, a graph database query language, a way in which case. And against the hotel knowledge, blah blah blah. So, yeah, this is, a success. So execute two. Well, that's it. This is all the information. And I signed the the virtual studio. You are... If you... However you have time, you are going to be able only because in the... In this virtual... I have this virtual. It's not the the the repo. This is my part of the repo. But in the public one, in the in the AWS sample, there is a way to put memory inside this engine. So inside the virtual studio, this is optional, but you can be an agent core memory, and you can add long term memory to this engine. So you can, when I say long term memory, agent core memory will build for you a vector storage for your memory. So it's build a a more personalization conversation for your agenting application. So it has two different type of memory, building with two different type of prong. So it just send all the conversation to that, virtual storage to that prong, and it's going to build, it's going to extract, some content inside that conversation, or... And it's going to save to save that data in the memory in the vector storage. So when you stop to... When you when you want to retrieve that conversation, you need to use a user ID and a and a session ID. So each user ID is going to have, like, ice... Isolate vector storage to manage the memory and the chat history for that, for that. And that's it. Oh my god. We have some questions. I know there is a lot. Let me see the chat. Hey, Elizabeth. Yes. That that was a lot. Whirlwind whirlwind tour through through through all the exercises. There were a couple of questions about the the the the repository and the and and the deck and where everything is. So I've shared again the link to your, to your overview page. I'll I'll share that again. Here's the QR code exactly. I'll share that again in the in the email that you get. If people are unable to redeem the AWS credits, as per instruction, you. Please please ask me ask me the credits for LinkedIn because, you know, this is a form that I have, and I have a limit amount. of credits. So, probably, the amount of credit that I have here, you know, is already gone, but I have more. Okay? So. please send me a a request and... Or whatever you... It's only about request. You can send me, I don't know, for, for Instagram or whatever, and I can share with you the the credit. Yep. Yep. Sounds good. Amazing. Sorry. I know that it's a lot of information, but the thing that I want you keep it here is not about AWS. You know? AWS is a cloud. So... And you probably know about Strand neither. The thing is that now you know how to control the harness, how to control the LLM in the framework, in the harness that now you know. that you can put hooks with code to control the invocation of the LME because you don't need to remove the LME every time. So if you put a hooks, you can control that. You know? And a hook is good as well to pro injection. So you can control pro injection in a hook, and you can control that. You can put business rule after the LMS. Let's suppose that I have my LNM with a specialization in AWS, and this LNM answer some question about Azure. Come on, man. This is about AWS. So you can control that after that. With hooks, you can control the amount of tools in vocation because sometimes the tool is like an infinite loop, querying and querying and querying. You can control that with a loop. So... With a hoop. Sorry. So you can put in a hooks maximum invocation per tools, so you can control that. And another thing, if you want to, implement agentic catching, put catching in your agents, you can put the catching in hooks as well. So before, after. So, you can put caching in tools in reasoning as well. I have a content. I have a block where I create a context graph, like a caching. So I send all my caching information to a Neo four j, so so you can retrieve that as well. And I know here the guys, they have a lot of information about agentic application using Neo four j. So now that when you start to see the new workshop that the Neo four j guys, I'm going to show you, you can think more about... Oh, and I can implement this amazing reasoning, graph in my engine using the thing that Ellie showed me today. Yeah. Exactly. And... Yeah. It's it's it's all, it's all universally, you know, applicable. You can you can choose however, you wanna you wanna build this. I have shared, because you mentioned this, agent memory and, yeah, and you shared also the context graph we have from our Neo four j team, a couple of of guides or a couple of blueprints you can use as well if if you're interested in that. I have shared that in chat. The Neo four j memory, service is a very, very new one that we just, launched a bit, ago, and the context graph build, tool has been around for a little bit. But I think the context graph build link is also in your handy workshop materials. The NAMS, the Neo4j Edge memory service is not in there, I think, but, or maybe it is. I think I've seen something. Yeah. It is already also there. Yeah. Everything is in there. So, yeah, amazing. Elizabeth's library Wiki page is is amazing. Everything is is on there. And I, know that Andre. Andres, that he... He is here with us. He have a a couple of amazing training in the learning AI. You know, I learned with him. I learned about Neo four gs how I can build a knowledge a knowledge graph with him saying the deep learning training that you have in the the learning website, and it's completely free. So you can go to deep learning AI and search for the Neo four j training there. They are completely completely free, and come on. You have address like a Yeah. Exactly. The deep learning courses are amazing. GraphAcademy courses are also available. We have something on knowledge graphs there as well. trainee. Currently, we are building more, more content out if you... If you're interested in that. And, yeah. Perfect. Thank you, Andreas. Share the link to the deep learning course. That's a that's a that's a great resource, a great, very interesting and very interactive, workshop as well. So, yeah, check that out. Have a look at that. Also check out, our next, upcoming road to nodes workshops. So if this for today was interesting, next week, we're gonna build, you're gonna build your first knowledge layer with with Zach and and and and Ryan. We're gonna do virtual graphs. We're gonna do, agents in the wild and, also agent memory, at the very end on the November 5. So if this is, like, interesting, then, you know, let us, let us know. But, also join the workshops and give, give them a a go. And, yeah, like we said, Fanny and others, if the AWS credit doesn't work for you, ping Elizabeth. She'll she'll she'll get you sorted, so that it works, for you and you can give this a spin on, on AWS yourself. Yeah. Thank you very much, everybody for watching. Thank you, Elizabeth, first and foremost, for bringing the content together. Amazing. You know, very, very, very rich, very, very thick. Very, Almost dense. very dense exactly. I know. But, but I I think it was great. I think that's that's great. You don't... You showed us today how it works. You showed us all the all the steps. And, because it's recorded, as as we said, well, you you can watch it again. You can you can follow, clone the repo, give it a star, and check it out. And, and, yeah. Thank you for, for for the presentation and for the for the demo time. Thank you, Andreas, for behind the scenes support. Thank everybody, obviously, for watching, for your questions, for participating, for sticking around. And, yeah. See you around, at the next row to notes. See you around at notes itself, or at another event, workshop, meetup, or anywhere else around the globe. And, yeah, take care everybody, and, see you soon. Bye bye. Thank you for having me. Like, a black chip for this workshop. The only. one. that. is. not. for me. or. for yet. not. No. No. Yeah. Yeah. That's definitely not the case. You are you... You're amazing. Thank you very much for for doing that, and thank you for for, you know, working that. That's... It's really great. We... We'll try to balance it out with with Neo four j community, members and Neo four j staff. So So you're not the black sheep. You're the you're the star of the of the roster. Well, I thank you so much. Yeah. Thank you, everybody. Bye bye.