Name: Actionable Knowledge With Context Graphs & Agent Memory
Uploaded: 2026-11-05T19:16:19.922Z
Duration: 2 h 7 min 28 s
Description: Actionable Knowledge With Context Graphs & Agent Memory

Transcript for "Actionable Knowledge With Context Graphs & Agent Memory": Hello, everybody. Good morning, good evening, good afternoon and welcome to the final road to notes workshop. It's been five amazing weeks with five amazing workshops and, dear colleague, Wil Wil Lyon, senior product manager at Neo4j is rounding us out basically for, the final session today with actionable knowledge with Neo four j agent memory is, the official title. And, yeah, I'm I'm looking, specifically looking forward forward to this one. This... They were all amazing, obviously. So, no. I I I don't have a favorite child here, but, but this one is very interesting in terms of, topics. And I talk to people at conferences and at events and oftentimes, like, yeah, this has been an issue. Like agents go haywire. They go all over the place. They they come up with the same strategy every time, but spend on a huge amount of tokens. And then you come back and say, well, you did this before. Right? I mean, this is this is... I told you how to do this. Why why are we doing it differently this time? And, and it is it is somewhat annoying. It is somewhat frustrating. And, Will has a has a way of of of navigating this and having having a a great tool, Neo4j Agent Memory Service, that will help your agents to become a little bit less dumb and a little bit less repetitive. And, and I think that's, that's all all all for the better. And I'm very excited to to see the workshop. It's it's gonna be a little bit of a mix, you know, obviously, some hands on exercises, some slides. You you'll be, you can follow along, as you as you like. You can always watch this back as a recording. So after the the workshop today is done, we'll send out an email with links to the session recording and also with some material that we will share with you today. So if you didn't catch something or if something was, wasn't, in your speech, then you can always look at... Look back at this, and we will we will, have that there. If you have any questions, anything that is, obviously, we have some time for break... Some breaks for questions, so it's encouraged. Please use the q and a function on your right hand side. You should see a button that says q and a. Click that button and, and type in a a question there. And then we can see it, as the moderators. I have... Besides Will, who's gonna do the workshop, I have Andreas and Jeremy from our developer relations team on on-site here as well, and they will help out with questions. And we will, answer some questions, obviously, in breaks during during them, and we can read them out. But, yeah, I think that is more or less it from my side. I I think, Will, I'm like to welcome you on stage as well. Thank you. Hi, Will. Welcome to this this final, road to notes workshop. One week out before the actual event. So notes is happening on the November 12. If you haven't registered for that, and I think everybody that has is watching this now should have registered automatically. But if you like this and if you're looking forward to it, please share the event with friends, family, colleagues, whoever should be interested in graphs and AI. So, please let everybody know how much you look forward to the event as well. But, yeah, Will, looking forward to agents' memory and graphs with you today. Awesome. Yeah. Hey, everyone. Thanks, thanks for kicking it off. Yeah. I I am very excited for Nodes. I I have been aware of of some things going on in the background. I I won't give any spoilers, but there's some, exciting things coming, for Node. So definitely looking forward to that. Cool. So I shared in the channel, the link to the slides here if they wanna load, maybe I'll share it one more time just so it's front and center to action. I think I spelled that right. What's What's going on with my slides, though? Come on. Let's refresh maybe. Like, the Internet has been having some problems today. Okay. Cool. So... Yeah. So like, like Alex said, this session, there's gonna be two hours, very hands on. We're gonna be taking a look at agent memory with Neo four j. We're going to kind of... As I was preparing this workshop, I I tried to... I thought of a few different approaches. And and I think since we have two hours, there's a lot we can cover. And so we're gonna kinda build up, starting with the same agent and and explore some concepts in agent memory, and kind of, see how our agent can get better. But then also, we're going to think of agent memory as something that we can build on top of. Right? And and this is where the actionable knowledge piece comes in. And and we'll talk more about this. But but think of think of memory as, like, something useful that we can, give to our agents so that our our agents perform, perform better, but also something that we can can build on top of. Cool. So all of the the info, that you need, there... There's a ton of links and and resources in the slides, so be sure to to pull that up. We're gonna work from, from a GitHub repository that we'll we'll take a look at in a minute. And please ask questions, in the chat. We'll we'll pause as we go through to make sure everyone, kind of follows along because the idea is for this to be as hands on and as, as interactive as possible. So, yeah, let's dive in. So, yeah. So like I said, we're we're gonna start with, with a running full stack app, and then build on top of that the the different memory concepts. And I think it's really useful as you're building agents and as you're exploring different options for, for adding things like memory and, and skill distillation, these sorts of things, that you have some way to measure, like, is my agent getting better? And so that's part of what we'll see today. We have, sort of, an evaluation framework that we're gonna start with. It's it's a bit simplistic, but, hopefully, you can, you can get the ideas as we kind of go along. So that's the that's the plan here. We're gonna start with with a basic agent, add some of these memory concepts like short term, long term, reasoning memory. We'll take a look at, skill distillation as well and then see how our, our eval performs, as we add those, add those layers. So I think a lot of folks are are familiar with graphs in Neo four j. If you're here, no worries. If you're not, we're we're gonna talk through, you know, the the graph data model, how graphs enable, agent memory here and and some of the tooling that we have built on top of Neo four j. But this is the the basic data model that we're going to work with, the basic structure of of the graph as we think about agent memory. Right? And and so we're thinking not just about the, like, the conversations and messages that you have with an agent. Right? Your... This is, typically, some sort of, like, chat interface, but there's also, tool calling. There's also, like, reasoning traces. Think of, like, procedural memory. We we call this reasoning... Excuse me. Reasoning memory. Other, other areas in literature refer to this as as, like, procedural memory, but not just the sort of extraction... Extracting entities and how they're connected from the chat messages. This is is certainly useful. But also looking at, okay, how are our agents actually, planning? What tools are they calling? How can we improve that process going forward? Okay. So we have, five modules here, and and this is all meant to be very, very hands on. And so we're going to start setting up our app. We're going to use, Neo four j Aura. And in a minute, I'm gonna share the the repo that we're gonna get started with. We're gonna have a basic agent and a basic chat interface, and then we'll take a look at, how we can start adding memory concepts and and see how our, eval changes. Right? And and and so that's the basic idea is we'll we'll identify some some failure mode, like, some shortcoming, in our agent starting from, like, a very naive assumption with with sort of a stateless agent and, and then see as we add the memory layers, like, what, what exactly are we measuring in in output? What exactly are we improving? Cool. So this is basically mapping to to each model. What we're gonna start with is is what does a a stateless agent look like just kind of to get things set up. Then we're going to add short term memory, which we can think of as the the conversations, the messages. And then we'll take a look at long term memory, which is essentially entities, preferences, things that are extracted out from those, from those messages. And one thing that is important to to think about is how can you, sort of connect the, like, memory, the graph entities then... That you've extracted, from the conversations? Can you ground that in your actual domain data? So we'll see how to do that. And then we'll take a look at, adding reasoning memory, to improve the planning, and procedural phase of our agents. And then finally, we'll take a look at how we can turn all of this this memory graph into actionable knowledge, which in this case are skills that we can then, then be some portable artifact that we can, share with other agents, as well. So we're gonna take a look at, two different tools built, on on top of Neo four j, the the database. One is the Neo four j agent memory package. This is an open source Python package. We're gonna start with this. Then we're going to, in later modules, use the free, Neo four j agent memory service, which is a hosted version, of Neo four j agent memory. Easier to get started with, maybe a bit less configurable, maybe. So so a question of, like, why would you use one versus the other? Well, the the Neo four j agent memory service is a a free service that that we operate. You can bring your own Neo four j database or or use, a temporary one with it. So it's really easy to get started. It serves hosted MCP servers, a hosted API. And it's useful for cases where maybe you're you're not able to run, arbitrary compute, yourself. So you can't run, say, like, a a a Python package. You you just need, like, a hosted MCP or a hosted API to plug into your agents to to add memory. But if you want, something maybe a little more configurable that that you can, that you can control and and build on top of and you wanna run, like, the the memory, Python piece alongside your agent, then, the open source Neo4j agent memory Python package is is, maybe a good good way to go. But we're gonna take a look at at at both of these, today just to to give an idea. Cool. So let's go ahead and set up our, our environment. So we have a GitHub repo that we're gonna start from. I really recommend opening this in, GitHub Codespace. GitHub Codespace is like a a hosted, containerized development environment. And there's a bunch of setup in there that that sets up the the dependencies and and kind of wires things together. So I would recommend going that route, just to to open in the code space, and we'll see what that looks like. If you don't wanna use code space, you can clone the repo and and work locally. But you may have some more issues with getting dependencies set up and and specific things with your environment. The the code space piece, I I went through last night and and was working, all great. So I would start, there. Cool. So here's the GitHub repo. I'll drop a link to this in the chat. And in the ReadMe, the ReadMe has the the steps we're going to follow. So we'll we'll take a look at that in a moment. But to set this up, just click on open and get up code space and create our code space. And this is going to spin up a containerized environment, for us. Yeah. Okay. I can... And can you zoom in a little bit, when you are doing that? Absolutely. Perfect. Thank you very much. Cool. And this will take a moment to spin up. I'm gonna say yes. I trust this code. Cool. And this will open to a, dot env file. So the first thing we need to do is to spin up a Neo four j Aura instance. We can use the the free tier, to get started. And then we'll also need, an OpenAI API key, which I will, share with us actually hidden in the the slide. So if you if you can find it in the slides, go ahead and grab that. Or if you have your own, feel free to use that. And then, the other interesting piece here is once we get this set up, you'll see we have modules one through five. And this is where the the code lives, the interesting bit in each of the modules. This is our agent, getting a bit more sophisticated as we go along. So that's where the the code lives. But first, let's get our Aura instance set up. So if you've used Aura before, Aura is Neo four j's hosted cloud service. There is a free tier, which is great for workshops like that. So I'm going to sign in. And when we sign in, we'll be asked if we want to, create a new instance. You'll have a slightly different view if it's your first time signing in to Aura. I already have an instance running, so I'm gonna have to create instance and free, and this is going to be my road to nodes. Cool. So it's gonna be a free instance. I think there's there's some limits on the, the size of the data that I can store into it, but, otherwise, it, it will, it will be there for me to, to use kind of indefinitely. And you'll notice here, that while this is spinning up, we can, download our credentials, and this will take a moment to to spin up. So we'll we'll pause to make sure everyone is caught up here. But this is... Let me find that text file with my credentials. Cool. So this is what, what that looks like. Don't... You don't wanna copy my credentials. You want, you wanna spin up your own instance. And what I'm gonna do here is just copy these credentials. And back in my code space, I'm going to replace these four lines. Cool. And so now I've got my repo pointed at my aura instance. And we'll go we'll go a bit slow here just to to make sure everyone is set up. Because I know... Especially if it's your first time signing in, it'll it'll take a moment. But, yeah, give a shout in the chat if you're if you're this far already or if you need another moment. Still creating an instance. Yeah. And it... And it'll take a while, shouldn't say a while. It'll take a moment or two, to to spin up. You'll you'll get kind of a a spinner here. Like, mine's still, still creating. Oh, there we go. And so now it's now it's ready. And it doesn't need to be, up and running yet. You get that that credential directory. Yeah. So the question, where to to get the username and URI? Yeah. So, so when you create the instance, so after you go through, I already have my my free tier, but you'll click, free and and then create instance. And then you'll get, a pop up, with the the password, and there's a button that says download credentials, and that's gonna download, a text file, like this. And that's where, you'll... You can just copy and paste the credentials from, the text file. Or if you have an instance already running, you can go here to, inspect, and then you can see the the details here, for that instance. You won't see the password. The password is is just shown that one time, when you create it. Cool. Yeah. And I see a question about the OpenAI API key, and Jeremy has found it hidden in the slides already. That's right. So, we take a look, slide 10. You can find it here in the speaker notes. This is one we can use for the the duration of the workshop. So we can copy this out of the speaker notes and put that here in our dot e n d file. And so this this has a a limited, budget on it. If we go beyond that, we might not get everything done in the workshop. So please please don't share that, and and let's just try to use it for the the workshop, but that should work for the the duration, today. Okay. Cool. So I've got my aura credentials here. I've got my OpenAI API key. The other thing that we're looking for here is the NAMS API key. This is the Neo4j agent memory service, which is the the hosted version of Neo4j agent memory. We will take a look at that in, I think, module module four. We we we get into that. So we'll skip that for now. Yeah. Cool. And I see if you folks are at the the stage of still creating an instance. Yep. All good. Just look for that text file that, that downloads because that will immediately have the, the credentials. Yeah. And the OpenAI key is in the speaker notes on slide 10. Cool. Yeah. And if you're if you're going through the initial, the initial onboarding, like, the the first time you sign up for, for Aura, there's some questions like what's your use case and and introduces some things. You can you can just kind of click through that, and then, and then create the free instance. Yeah. So question, what... What's AuraDB for? So... Yeah. So Aura... AuraDB is the hosted database service, for Neo four j. So the the reason we're using this is you get, like, a a hosted Neo four j instance, so so hosted database, and then we're going to use that to store our our agent memory. Cool. So, so as you're going through this, these are the the the pieces, the AuraDB, credentials, the OpenAI key. We'll just leave the... Leave the NAMS API key empty for now. That's that's one of the later modules. And then what we can do to make sure that, that we're good to go in the terminal in our code space is make check is the first command. So the way that we're going to, to go through the different modules here and and to run pieces of our app and whatnot is we're going, to use the the make file, commands. You can, you can see all the the commands in here. You can also run, like, make help, and it will tell us, like, these are all of the different scenarios that we're going to, that we're going to use. But, make check is... The first thing we're gonna run, and this is just going to double check that, we have set our credentials, correctly, and it's just going to ping our AuraDB instance that we're able to, to reach it and that our OpenAI key is set. So run make check, and if you've got these green checks, then, then you should be good to go. And once you've got all green checks, the way that we're going to start our apps or our full stack agent is with this make dev command. And then the first argument is gonna be the module that we want to run, because we have, just different implementations of the agent. And so we wanna specify, which one we want to run. So I'm gonna say make dev module equals one. And this is going to to start up two things. One, it's going to start our our our agent, which is a Pydantic AI agent, and then it's also going to start a web app, on the front end. And then if you're in the, the code space, it's going to expose those ports so you can, so you can open those in your browser. So for example, if we go here, we want the front end and open the URL. And so now this is my, my agent, but available, kind of globally from, from this URL. So that URL will be will be a little bit different. Sometimes it it it opens automatically, after the make command. But if it doesn't open automatically in your browser, you can find it here under ports. And then you want the support bot, which is the front end, and then this one here. You either copy the URL and and open that in a new tab, or this will open a a new tab in your browser. Cool. So code spaces, it's... Takes a little bit of getting used to, but it... It's helpful for these sorts of, like, hosted workshops, I I think. Well, maybe you guys can tell me. But... Okay. So... Cool. So let's talk a little bit about, about what we have here, and and then we'll look at the code, and then we'll make sure that everyone is, is up and running. So, yeah, so the app here is, like, a very simple support chat agent for, Acme business, and I think was the the name of the business. And, as very simple architecture here, there's a, SQLite database, which we're gonna think of as, like, our our, you know, business records, our our our database that contains, you know, accounts, the the product catalog, and also business policies. And we built some some tools on top of that that we're exposing to our agents. So our agent is able to call these tools to look up things in this SQLite database. But this could be this could be another Neo four j database. This could be, APIs to different services like our CRM, our, you know, multiple relational databases. Right? And we have some tools that we give to the agent that's able to interact with that. So that's just kind of an abstraction, over, over our sort of business records. And then we're using Neo four j as our database where we're storing, memory. And and so this memory, these are gonna be learnings that we are, extracting from interactions, with our agent. Right? And and so this is gonna be conversations and entities preferences extracted from the conversations. But, also, we're going to capture reasoning traces, how our agent went through this planning process, the the tools that are called, and and, that is all going to be captured in Neo four j. So that's why we, why we created that AuraDB instance and why we want it to be private to each of us is because that's where, our memory for our application is is going to live. Right. So, yeah, so exactly exactly what, what I was just talking through is we have these these business tools that, we give to our support agents that's able to then interact with our our sort of system of record. And our our bot is... Support bot is going to be reading and writing from Neo four j, in this case, using the Neo four j agent memory package, to do that. Cool. So if you look in, in the module one directory, you can see the code for, for our agent. It... It's very simple. It's a Pydantic AI agent. We like Pydantic AI. It's a Python based agent framework. We use it for some things internally at at at Neo four j, but there are lots of other agent frameworks out there. I'm curious what's, what other folks are using. Eve, which is Vercel's TypeScript agent is is interesting. That one is is, I think, a a popular sort of newer, agent framework. There are also agent frameworks from, from, like, Google and, and AWS has strands, and and these sorts of things. So lots of lots of options. Has anyone used PyDantic before? Let us let us know in the in the chat. I like, Pydanta because I I think it has some fairly, like, you know, simple abstractions that that transfer to a lot of the different agent frameworks. Cool. And then there's, there's also a, like, front end piece that, that's the same across all of the the implementations. And then I mentioned the the business tools. These are the specific tools that we give the agents. We can, like, look up an account. It can, you know, search the product catalog and fetch a policy, and and then these these are just, like, parameterized Cypher queries that... Or, sorry, parameterized SQL queries in this case to to fetch, data from our SQLite database. But, again, that could be calling another API or or something like that. Cool. So this is, a code snippet for our agent. Let's... Instead of going through that, let's look at the actual code here because it's... The interesting bet is just a couple of lines here. So I'm gonna go... This is now in my code space. I'm gonna go to module one, and there's, really two files that are interesting here. One is the agent dot py. And, the interesting bit here, we have the, the instructions. So this is, like, the the prompt. Like, hey. You're a support bot for Acme Cloud SaaS company. You know, use use tools to look up data. Don't make things up. We have a chat service because this is... It's gonna be like a like a turn based interaction, right, where every turn is a user message and an agent message. And there's some other other pieces in here that we'll take into account in the later modules that will be interesting. But the basic interesting bit is here. We create, the agents. We need to say what model, are we using. We're using, one of the OpenAI models. I forget which the the default is, but it's set in the the dot e n v. We're using one of the the cheaper models, in this case so that we don't, unnecessarily burn through, burn through our tokens. But... And you want some sort of, like, tool calling, model for your agent typically because that's the typical thing that is... Your agent is doing in, the agent's loop. There's this this concept of, like, a a React agent or there's some, like, reasoning and and action loop that your agent goes through. Essentially, on on each turn, it's deciding, hey. Do I have the ability to answer the question or perform the task that the user sent me? Or given all of these tools, maybe I should call a tool and and, maybe I need the result of that tool call to complete the task or answer the question that was given. That's the the reasoning phase. And the tools here, we pass in when we create the agents. Right? So here we're defining, the tools in another file, which we'll take a look at in a second. And then for the... Actually, executing, the agents turn. Where is that? That's right here. So in, in the turn, there's there's some boilerplate around here, but this is the interesting bit is, hey. We have a message. We have our instructions. We have our tools. Let's, let's run. Let's send this to the agent and, and see what the result is. And we put that in in some sort of loop. So that's our very simple agent to to start off with. The the tools, by the way, are are here. You can see the tools are things like, look up accounts, search the product catalog, fetch the policy, and these are just, these are just functions to call SQLite, to execute some parameterized, SQL query. Cool. So let's take a look at our agent. So here's our, the web app that we have for this, and here we have kind of some, some chat interface. And then the the bit on the right, this is just for us to kind of see what our memory graph looks like and and some traces of tool calling. So I can say, like, I have a billing question. It says, hey. Give me your email address. And I I have some fictitious customers here, so I can say, Maya dot Chin. Let's go with that. I have a billing question. My email is myajen@northwind.example. And we can see here that, okay. Like, you'd... You looked me up. You found my my pro plan that I have a refund limit. And I can see here the the tools that were called. There is no no memory rights because we haven't enabled memory yet, so that that makes sense. And then we executed the lookup account tool, and we got back some information from looking up the account. We did that by looking up, by email address. So... Okay. Fine. And we can say something like, I was charged twice on my latest bill. Please refund the excess. Something like that. Right? And our agent is saying, hey. Provide your email address so I can look up your account history. And, like, what? I just told you my my email is right here. Why do you need my email again? Well, our agent is stateless. If we look at the the code and and what we're passing, on that agent turn, we're only giving it the the messages and the tools. We need actually to make sure we're we're passing in the actual, in this case, short term memory or the the conversation history, so that the agent understands, that we're, in the middle of a a longer conversation that is just one sort of interaction. Right? And this is, this is some commands we can run to to kind of see this in the, go back to our terminal here. Right. And I wanna create... I wanna leave my agent running. So I want to open a new terminal. And the command that I can run is make s one dash baseline. So that's, scenario one. So you can see make s one dash baseline. And this is just gonna run essentially something very similar to to what we just saw in the chat. It was like, hey. Across, multiple, chat messages, like, there's no there's no state, to the agent. And I can also run our eval now. So we have a very simple eval. I can say, now make eval, module equals one. And so, like I was saying earlier, it's really helpful to have some, some eval as we're implementing, our agents so that we can actually have some measurement as as we incrementally add some of these features. Like, hey. Are we actually improving, what we're trying, what we're trying to solve here? And we pass zero out of four. We can also run the eval from, from the websites, And we can see here that, yep, none of these passed. So, let's fix our stateless agents, and actually add, first this idea of short term memory. Oh, and this this is what we're running in our eval just to give you give you some idea. And and, typically, right, we would have more than just, like, four basic things that we're that we're checking, but we're essentially, like, hey. Can we do a a named lookup, across, you know, multiple, multiple messages? And then we're looking at, the second one is can you save a preference? And then the the last two are looking at the procedures. Can you follow a policy? And once we've had a successful interaction with the agents, can we, can we go back and sort of explain... Can the agent can the agent explain, what was the policy that was applied? What were the tools that were used? What is the rationale for, in this case, either granting or refusing, refusing a refund. And so to add conversation history initially, we're going to use the Neo four j agent memory package, that we we talked about, at the beginning here. The object agent memory, Python package has three basic abstractions for memory, short term, long term, and reasoning memory. So short term memory, we could think of as conversations and and messages. Long term, these are the the entities and preferences extracted out of of those messages. And then reasoning memory are looking at the the tasks, the steps, the the tool calls, the reasoning phase, that your agent goes through. But these three memory types, these are all stored in one connected graph. So the the green piece here, which is what we're gonna start with, this is short term memory. Right? So these are your conversations. Long term memory, these are entities that are extracted out, of those messages, and then, how those entities are connected. Right? So a person might, you know, work at an organization or, or, you know, a customer places an order, an order contains a product like these these sorts of, relationships. And then reasoning memory, we can model, connected to the message. This is the... Like, what kicks off the the task or the the question that triggers the reasoning phase that the agent goes through. And for each reasoning trace, there's one or more steps. A step, can result in a tool call, which then calls, like, one of our specific tools, and then we can capture all of this information in in reasoning memory, both like the, the rationale for, the agent, the, you know, tokens that it used, the the time that it took to call these tools. And we can then build on top of that information to help agents as they, again, go through this reasoning phase sort of like, here's what we did before. Here's what worked. Here's what didn't, because our reasoning trace is also going to record, was this a successful interaction? We could also score this with some human feedback, that we can use to then build on top of. So that's the the general idea of the new agent memory package. There's, like I said, a a Python package that we're going to use with these these three abstractions. Essentially, what we do is we create a a memory client, and then we can add messages. Those messages can go through an entity extraction process or not. We can we can manually add entities, and then there's retrieval pieces built on that so we can fetch the most relevant context given, given the task. Cool. Yeah. I think I think we kind of said that. And, oh, and and one thing to point out is there's a lot of... I think this is the right page. Yeah. There's a lot of integrations that exist in the Neo4j agent memory package with various other agent frameworks. So we're gonna use Pydantic AI. So we published a specific Pydantic AI integration. All all of these agent frameworks have some concept of memory often. And so, so some of the the integration piece here is just wiring up the, abstractions and that the API that we have in new to agent memory with the, the agent framework. So this talks about some of the different, agent framework integrations, but I I wanna say we've covered, most of the Python agent frameworks. And then, for the the hosted memory service, we also have these ones plus also TypeScript and and Java integrations as well. Cool. So what what does this look like? We'll look at the the code and switch to module two in a second, but, there's a few configuration options when we, when we set up Neo four j agent memory, the back end piece. So so currently, the the Python package, can be, Bolt's back end, which is I want to talk to my Neo four j database directly, or it can be the Neo four j agent memory service. Just be a a client of the hosted service. So because this case, initially, we're talking directly to our Neo four j instance. The back end is gonna be Bolt. Bolt is the, the wire protocol that we use to talk to to Neo four j. And then we need to grab our credentials. This is why we we pasted those in the dot env file. And then for the extraction piece, that's why we, we need the LLM, and also the embedding model, because we use vectors and and text embeddings to, to better find the most relevant context using semantic search and and these sorts of things. And then the extraction piece, there's... This is the process of extracting entities and and preferences out of the messages, and there are a few different options here. Initially, we're we're not gonna do any any any extraction, so extractor type is none here. The other options are, like, we can use LLM only, which is which is accurate but expensive, or we can use, smaller models that are that are built in as well and and kind of a pipeline. We'll see what that looks like, in a minute. Cool. So let's switch to module two here, and we'll see what our memory graph looks like. So, the idea here is that instead of, instead of just having a stateless agent, we're going to build upon, keep track of our conversations using that, conversation has a message and and sort of the next message, chain, in the graph. And we do that by sending a message to the, short term dot add message, and we wanna, you know, keep track of this as a user or assistant message, and and so we wanna be able to capture both the the user message and then the reply, from the agent. And then the next piece is, okay. Well, what do we what do we actually feedback to, to the agent on on each turn, you know, recent history, like, yeah. Sure. We can we can give the agent all of the messages. And this is fine for small, not long running conversations. But as we have some, you know, very large, conversation, we don't want to blow up our context window. We often want to, you know, make sure that we're giving some some summary, some compaction, or or maybe the the most relevant. And and so there are a few options here. In this case, what we're gonna do is we're gonna take the most recent messages, and then we're also going to do a semantic search to find the the the relevant messages, as well. So if if we've stored, if we've told the the chat agent our email address and and we know we need that, again, to look up for a tool, we'll we'll be able to do some semantic search to find, okay, this is the email address you told me, that, that sort of thing. Okay. Cool. So let's switch back to our code here. And what I wanna do is stop the instance or stop the process that was running in my first terminal. So I ran... I had run, make dev module equals one. I'm gonna switch now to make dev module equals two. And then I might need to restart, my web app. It's gonna be the same the same web app, but now it's connected to, hopefully, to our new agent. I got a connection refused. What happened there? Was it running now? Yes. It's running now. Okay. Cool. So let's see. Now we can try, try... Let's try that piece again. I have a billing question, and we should be able to see the tool calls here. Like, nope. There were no tool calls. Hey. We still need to know your email. And I'll say my email is Maya at... Was it Northwind dot example? Is that what it was? Or example... Yeah. Northwind dot example. And now, we see... Yep. You did my lookup. You found my information. What's my question? I was double charged on my last invoice. Please refund me. Whatever our whatever our question is. And, you know, we have, like, pretty limited tools here, but, hey, it it looked up our account. It was like, hey. I don't see any any issue. Like, hey. You know, that's kind of all I can do given given the tools that I have. But we can see here the graph that was built up. Right? So we have our conversation node. We have the the message nodes connected to the conversation and and kind of this chain, of messages. And so this is what we're feeding back into the agents, the conversation history plus the... Any of the the most relevant messages, for our agent. This this kind of enables, simple stateful agent. And what's interesting here too is is we can also, you know, this is good not just for, you know, a single chat, but we can do this across conversations as well. So now if we run our evaluations, let's start a new eval. Eval is running. And we'll see what happens. We can also run, run our eval from the command line. Alright. But let's let's run our two scenarios. So our two scenarios that that we can run are, s two scenario two return and scenario two restart. Make run s two return. Does it make... What was the command again? Oh, no make no run. Just make s two. Cool. So these are going to, to run our scenarios, showing that, hey. We can send a message, and then start a new conversation, and we still have that, available to our agent. Cool. So let's pause here and maybe see. Do we have any questions that we missed? I think I see Jeremy in the chat, hopefully, I think you with folks. you answered most and Jeremy answered a couple of other questions. Maybe we can go... Before, repeating maybe the steps to start module two, there was one question that could be interesting for everybody. And I think, Jeremy answered it in the Q and A section. But I think it's maybe interesting. Mark was asking, if the agent was looking at a Neo4j graph instead of an SQLite database, could the memory layer be part of the same graph or is it better to use two different instances or or general? How... Is it is it a good idea to combine agent memory plus actual records in in the same graph database? Or should they be two separate instances? Yeah. Yeah. That's a really good question. So so, essentially, like, hey. Can I can I have my domain data, alongside my my memory data? Yeah. And there's just the... Like, yes. Like, certainly, that that... That's an option. And and we're gonna see in in the next section, how we we can actually, like, make sure that we're joining those... Joining our extracted memories to our domain data, even in this this, like, SQL system. So that's certainly an an an option. You you can certainly, certainly do that. One thing to think about is is just how... As you're extracting entities, how do you go through that in a resolution process to to say, like, oh, this customer that was mentioned in the chat is the same as the, you know, customer one, two, three. that already exist in in my database, and and we'll see some options for that. That's kind of kind of the the thing to think about. And so that... That's certainly the case, an option, as well. I think, in general, you know, folks have lots of different systems and APIs that they're drawing data from. And and so, it's useful to see how this works if you have some some sort of like tool abstraction over fetching data from my operational database or or calling other APIs and and that sort of thing. And and, yeah, those those tools could be an abstraction over, yeah, the same you know, data database using, you know, parameterized Cypher queries or or something like that. So, yeah, it's pretty flexible. Cool. Amazing. Yeah. I think this was it. But maybe, if you could quickly run one more time over the steps that needs, to happen before, continue. I'm about to start the module two. Yeah. So, so you wanna have, like, two terminals open here. And and the make commands are what, what we're using to run each, sort of module of the, of our agent implementation. And so I have one running here. So I'm going to just control c to stop that one. And the the command that I ran was make dev module equals two, since that was what we were looking at. And then that will start our module two agents, which in this case, we can see, like, now has the memory implementation pulling in, the Neo4j agent memory, memory client and memory settings. So we're configuring, configuring our agent. We saw this this code in the the slides just passing in, our connections to Neo four j and the, the embedding options, and, we want no extraction here. And then we ran our eval. Did our eval... Oh, we we stopped it, so it wasn't running. But what we what we should see is that, the first piece of our eval completed, like, yes, we can have, conversations across, we can have some conversation state across turns. So nothing nothing too, like, earth shattering there that that may seem like kind of obvious. Like, yes, of course, we wanna pass in previous messages. This is just introducing something to, to build upon. Okay. Cool. So let's, let's take a look at long term memory. So messages record, like, like, what was said, and the, you know, sort of like the chain of messages across in, like, multiple conversations. But the the more interesting piece is now what can we build upon that. So things like, extracting entities, extracting preferences out of the, the messages. And this is what our long term memory component is, is all about. So I mentioned earlier in Neo4j agent memory, there are a few different extraction options. One is we can do a pure LLM where every message gets sent to an LLM, and we extract entities, from that. And that can be, like, quite slow and and quite expensive. And so there's a flexible entity extraction pipeline in the Neo4j agent memory, tooling that can, use... First, we use the the spaCy package, but this is a a statistical NLP, Python package for things like named entity resolution and, this sort of thing. The the next phase in the pipeline is using a small, model. I like the the gliner model. This is a a small model that you can run on CPU. You can run it locally. That that is trained for entity extraction. One thing that's nice here is we can also use a ontology with Gliner, which we we can't do with spaCy. So, anyway, the... This this pipeline is is tunable and and and configurable. We can also set kind of a threshold for when things get sent to the LLM versus just GLINER. We can do just GLINER if we want to, to just do all of our memory extraction locally. That's, that's quite flexible. And, again, just a reminder, like, this yellow piece now, this is what we're gonna build. We're going to be extracting entities, out of our messages. And we're gonna do this using an ontology, and we'll we'll compare, like, the the default ontology to one custom for our domain. But, essentially, like, the... Think of the ontology as, in this case, like, how we're how we're defining our domain. So ontology is, in this case, is gonna be some domain specific data model. Like, these are the entities. These are how the entities are connected. This this is what they mean to to my business and and and that sort of thing. And, Gynecology here is is quite simple. We have, like, customers and and products and and orders, like, that sort of thing because we're just a... We're a generic, ecommerce company, I guess, or a SaaS company, whatever whatever Acme is. And, and... Yeah. So the ontology piece, there's a there's a default ontology with Neo four j agent memory, which I think we have some slides on this in a second, which is the the poll model, people, organization, location, entity. I'm I'm forgetting exactly what what all what all of those are. But, I think that that... That's useful for, for getting started. But the the challenge is that essentially covers everything. Right? And when we're talking about memory, like, one one important piece is being able to tell the the memory system, like, what is actually important for you to remember. Right? You you don't necessarily want to remember everything. You wanna remember things that are that are relevant to whatever the agent is supposed to be doing, whatever is relevant to the business. And and so I think it's important really, as you're adding memory to think of what is the ontology, what is the the domain specific, sort of data model that that I want to to remember? Alright. Thanks, Jeremy. Pull pull plus o. Person, organization, location, event, object. Yeah. So that's the the default, model, which which is everything. Right? And and, again, that can be super useful for for getting started and and super useful graph data model for, this comes from, I I think, the poll model, which is is used typically in, like, investigations or something like that. Cool. So, the ontology functionality, we, have a few different ways of working with ontologies in the Neo4j agent memory, tooling. In this case, we're gonna load from a a YAML file that we've already defined our ontology, which is essentially, like, what are the types, how are they, how are they connected, what are the the node labels, and what are the relationships that I want to extract. And there's some... Then additional configuration. In this case, we want to enable the abstraction or the, extraction. So extract entities is true, and we're gonna use the LLM, extractor. Okay. Cool. So should we look at the code snippets? So we're now... We're gonna switch to module three. Here's... You can see here's the default, our generic pull model, and we don't we don't need to define this. This is what, what ships built in. Here we can see, like, our specific ontology for our Acme. Right? So we have customers that work at a location, customers subscribe to plans, customers use products. We also have incidents and and these sorts of things. And the interesting bits here, we saw, like, some code snippets of this, but we can see here... Like, here we're adding some additional pieces to the extraction. There's also an enrichment piece, to Neo4j agent memory. This is useful if we want to fetch some additional information about, about an entity that we've extracted. And and this is meant to be configurable, I think, by by default. There's Wikimedia and Diffbot are the supported, enrichment options, but you can also implement, the enrichment interface. So if you wanted to say, like, fetch some data from some other, third party system, we'll also see, though, that piece that we talked about earlier is when we extract an entity, maybe by by name or by email, we wanna be able to, match that to our domain data. And so we'll see how we can, also, in this case, fetch more information about our, our engines we've extracted from, in this case, our SQLite database using those those tools, like the account lookup tool and, and that sort of thing. Cool. So there's a a little more wrappers here around creating the memory tools that we're going to give to our agent for, for searching memory, saving preferences. These are, wrappers around the, the abstractions in the Neo j j j j memory API for, in this case, the long term memory. And we're passing those tools, to our agent, giving some additional instructions on how to use these, these memory tools that we're now, giving to our agent. Cool. So let's do make dev module equals three, and this is going to start our module three agent. And if we go to the web app and give that a restart, It'll take a minute to spin up. And and by the way, we can we can inspect, like, the the memory graph that is being built up. If we look at all stored memory, we should have, yeah, our... Here's the the conversations that we had previously. But we can also go to our Neo four j instance. If we go to... Here's the instance I created. This is my aura free road to nodes instance. I can go to query or or bloom, and I'm a very similar view. I don't I don't know how useful this is, going to be since we have that that graph view and the the web app that we're looking at, but this is where our data actually lives. And so we can see here... Right now, we just have conversations and messages. We can see here, though, these are the, constraints that are, that are... If the agent memory Python package added to the database, so these are, like, other labels. We don't have we don't have any reasoning traces yet. We don't have any tools or tool calls, but we we will be using those later. Okay. Cool. So let's see what this looks like. We can run the extraction for the s three scenario three default piece. So I'm gonna switch to the other terminal here to allow paste. Yes. And so this is going to, I think, run, run some conversations and, just give us an example of what it would look like. Right, so we would have people, organization, and then some events this person works at and and knows Acme Cloud. And and, like, that's not, like, super accurate. Right? I mean, that's useful. But, again, it's not specific to our to our business. Whereas if we looked at what would that any extraction look like, if we were using a ontology specific to the business, specific to our Acme SaaS piece. Right? Well, now we get... Okay. We know that this is not just a person, but it... It's a customer. We we know the plan that they're on, the product, the the organization. Right, and our relationships are much more, much more relevant to the business. Right? So, this is... I I think the the message and the conversation that's used for this is like a billing dispute. Right? And so we treat that treat that as an incident, some billing disputes, and, you know, this is affecting the the customer, and so on. So, anyway, that gives us an idea, like, oh, okay. Like, using, using an ontology that maps to the domain, that maps to what we're trying to, what we're actually trying to solve with our agents. This is gonna give us much better results in our memory system than using essentially the the default, which is, basically everything. Cool. The hydration piece is quite interesting, as well. We'll take a look at at that in, in a second. But, basically, what what we have is, okay, you... You've extracted some, you know, like, information, from the message. Like, okay. The... This is a customer. This is the customer's name. But then it may be useful to to then go to our, our billing information, our CRM, in this case, which SQLite is the the standard for that, and actually, fetch some of that data and and we'll be load that into the memory graph, as well. And so we'll see how we can do that here, using the memory dot long term add entity piece. So, for any entity that we've extracted, we can, go back to, in this case, SQLite to get more information. But these are, think of, like, any tools that that we could define to go back and fetch information about our, our user from, say, a CRM. And and behind the scenes, this is doing, doing a merge. Right? And so the extraction piece creates the entity, and and then we're add... Essentially adding additional information, from our, from our SQLite database. Okay. Cool. So we switched to, module three. So module three is running. We took a look at the the memory tools. Let's go in and see how this looks like in our memory graph as we're, as we're building up the memory graph. So, let's see. We can say something like, I am Maya Chin. I, have a problem with my bill. I was charged twice on my last invoice or, you know, something like this. And we can see here that this is gonna go through the tool calling phase. So we should we should fetch information about, both the account. And so what's interesting is, right, we didn't have to provide our email address that comes from the short term memory piece. And we then, called some other tools. Right? So search product catalog, and this search memory tool. So I see, I see a question in the chat about, like, how how does this memory actually get utilized. Right? Like, so far, we're we're talking about, we're talking about just constructing the memory. We saw how we, how we are building the memory. We saw how we pass, like, the short term messages, back into the context, essentially. But what we've done also is define these memory tools. So in this case, the search memory tool and given that, as another tool to our agents. Our agent is able to, to actually search the messages. We can see here exactly, like, what was, what came back from this query. And this this was a semantic search, and so we... You can see here... Do we show... I don't know if we show the the vectors. I don't think we show the vectors in the chat here. But if we look in the database, if I look at what are all of the the entities that we extracted, like, okay. Here's, here's Maya. We're using a, the ontology. So this is not just a a person, but it's... We know that this is, this is a customer. And we make use of embeddings here on, embedding the the name and the, like, additional information that we fetched. So we hydrated this data, from the SQLite database. We saw that how we went back to fetch additional data after we extracted the entity from the message. So now we have available in our memory graph, like, the preferences, the, the plan that this person subscribes to, the organization they work at. Right? Like, now all that's available in, the the memory graph that we've, like, surfaced up essentially from... Think of, like like, our CRM system. But, anyway, what I was getting at is we use embedding. So we create, a embedding property. This is, is using, like, a vector index so that when we, for example, when our agent calls the search memory tool, it's passing in, okay. What is the query? Well, double charged on my last invoice is is the query. That's what the customer is complaining about. And we're doing a semantic search, to find, okay, like, where where is, you know, relevant information about being double charged for memory and using the the vector index functionality. So, anyway, that's, to answer, Vereen's question, like, how are we actually, like, utilizing memory? That's, that's one piece that we're giving, tools for agents to go search the memory, and we're also doing things like injecting the conversation history into, into the context. Cool. Yeah. And I see a question too from, from Mark Henry. Is the poll or custom ontology loaded for? Yes. So in, in this example, we're just using the LLM, extraction. And, I think I think that makes sense for doing this in in these code space environments. The the glider model, you know, uses more CPU and and memory. And, so here, we're just using LLM and passing the ontology to the LLM. But, yeah, that... That's one thing that's quite nice about GLINNER is is we can, give it a, an ontology and and so that the entities and relationships that are extracted from Giner also map to, to the ontology, that we give it. Yeah. Glynr is quite flexible. There are a lot of, fine tunes of of Glynr. I I think the the version we use in Neo four j agent memory also has functionality for excluding PII. Sometimes you don't necessarily wanna store PII, in your memory. So there are lots of options for different Gliner models fine tuned on, on different pieces. Cool. Let's run... Oh, yeah. Let's run our, eval for module three. So we'll go here, eval, and start our new eval. And so this will take a second to run here. But, yeah, the the idea here is if we go back and look at what we're actually trying to measure, with our eval is the named lookup across, across turns, the, unique ticket phrase. Yeah. That's one that we didn't, that we didn't exercise in the chat. But, essentially, like, if we're, saving a specific code word or something like that for a for a support agent, sometimes they use that to, let's say, like, reference this, this ticket later on. You'll you'll give, like, that code word. Or the other piece, I think, that we're checking in the eval is saving a contact preference. So if I say, hey. I wanna be contacted by email or by phone. Can we save that and and tie that to the customer? Cool. And so so we've passed three out of four here. So the name and email we passed, the verification phrase, which we can we can demonstrate in the chat here, and then the, sign up date for the customer, and that comes from, the hydration piece, right, where we've actually, like, fetched that data from, from the database. Cool. So if I do a new conversation, I can say something like, remember my, contact preference to be contacted by phone. I misspelled phone. We'll see if that actually works. Yep. My email address is Maya@Northwind.example. Cool. And we can see the... Our memory graph kind of growing here. Let's look at just the current conversation. Cool. And we can see here, like, here's the, tool calls, that were made. So first, we fetched info about the customer, and then we saved the preference to be contacted by phone. Cool. Okay. I think that is what we wanted to cover for module three. Any questions we should address before we move on to module four, which is all about, reasoning memory? Cool. Amazing. I think you you covered the custom ontology question, which I think is interesting. There was one question came in a little bit earlier is, when building a recommendation system powered by a knowledge graph, should we also consider incorporating agent memory? So directly from the start out when you think, okay, you've built the knowledge graph, you're going for for that. Should should an agent memory be already incorporated there? I think, I think possibly, like, there are many knowledge graph use cases that, that don't necessarily, require agent memory. I I think, ultimately, if we think of, like, agent use cases for for memory, like, the problem that we're trying to solve, is fundamentally, like like, making our agents, more effective, right, for... And we're seeing this as why we're we're trying to, like, work through these these simple evals. But also things like introducing determinism or, token efficiency, time efficiency. Like, these, I think, are are challenges that agent memory can, can help address. And so if if you have a a case where, yes, I want to be able to to to introduce some determinism on my agents. So the ones... Once I have, you know, trained an agent to have a a successful, maybe, like, processing a refund request or or that sort of thing, how can I make sure that my agents will use that, that procedure going forward? That can be a a good use case. Or I want to, you know, make sure that my agents are operating at their their most efficient use of of tokens and time, that can be a a good. case to to introduce agent memory. But... Yeah. So I I I think the answer is... Yeah. It kind of depends on, on the the challenges that you're you're trying to solve as you're, as you're starting your project. Cool. And then maybe one more question. Does the ontology also needs to be stored in Neo four j, or how does the ontology get get into the graph? Maybe two questions here, but, where where where does the ontology live? Yeah. Yeah. There... There's a few options. Right? In in this case, what we're doing is we've stored it as the simple YAML option. So, so this is what Neo four gs agent memory uses. It's quite simple. Right? You have essentially, like, your your types and your and your labels and and some additional, like, descriptive text. When when we look at the Neo four j agent memory service, the hosted service, and I think we're gonna get to in the next module here, it uses this... The same format to, like, store the ontology, and and then, ultimately, that's stored in, in the graph. Excuse me. The... There is a graph ontology specification that, that we're going to move to instead of this this sort of simple YAML format, which is one that that Neo four j uses, internally as as well. And then, you know, this is the the concept of, like, just describing the the data model, but then you also wanna think about, you know, how can I enforce this at the the database level, which we don't we don't currently do here because there are also cases where maybe you want the agents to evolve the ontology? And and so we'll see in the the NAMS example, the new agent memory service, we can actually, we can actually, like, have the agent suggest new types and labels that we might want to add based on the type of conversation that that we're seeing. So maybe, like, an evolving ontology. But, yes, you you also wanna think about how could I maybe, generate, database constraints. And, Neo4j has this concept of a graph type, which is, like strictly defining like, okay, this is the shape of data that I'm allowed to store, in the database. Yeah. So so lots lots of different options and and different, different ontology tooling. We're we're gonna talk a lot more, I think, at, note some of the notes talks next week about, how to work with ontologies both in Neo four j and, in some of the additional, ontology management, tooling, and and some things like that. So if you're interested, usually in ontologies, yeah, check out some of the the sessions, in notes next week. There'll be a lot a lot more detail there. Definitely. Alright. I think, that's, what we have for now, and, let's continue. Great. So we've got, thirty minutes left. I think we have two more modules. So let's get cooking. Okay. So we've looked at, short term memory, which are conversations and messages. We've looked at long term memory, which are entities and preferences extracted from those conversations and messages. The next type of memory we're gonna look at is reasoning memory. And reasoning memory, this is all about keeping track of what the agent did, how it, sort of, went through the reasoning phase, its planning phase. And then the way that we we use that is the next time, the agent goes to do a similar task or or even a different agent, we have these execution traces, these reasoning traces of what worked well, what didn't work well, how many tokens, you know, this execution plan cost versus another. And we can then allow the agents to use that during, during this next reasoning phase to be more, more efficient going forward. So the the data model that we use in Neo4j agent memory, essentially looks at, reasoning traces. A trace is... Single trace is tied to a message, which is gonna give us, like, the the task. And then there can be one or more reasoning steps and and tool calls connected to that trace. And, also, like, the the the tool calls and our reasoning steps are going to potentially interact with, entities, and that's why we we add this touched relationship, so that we can have some, you know, auditing and and explainability as well for each task. Like, okay. These are the tool calls that we use, but we also referenced, these entities in our in our memory graph. And so the the model looks like this. We have, a message. And that message, again, that... That's connected to, the conversation and and entities extracted out of of the messages in in this piece. But then we have a reasoning trace, in one or more reasoning steps, and that step may may touch entities and and we keep track of the the tool calls and and metadata about each of the tool calls. And so we look at the the API in the Neo4j agent memory package. How do we record traces? Well, there's this, this third abstraction is the reasoning memory piece, and, we start the trace. And then as we go through, go through the turn, we wanna keep track of the, the messages that this triggered. Right? So we need to link the trace to the message once we've created it. And then, different agent frameworks have different ways of exposing the the reasoning phase or phase and the the tool calls and and steps. And so, essentially, when you think about how to to fetch those, this is what it looks like for, in this case, for Podentic AI to fetch the, the steps and tool calls. And, also, we want to be able to fetch things like... And, again, this is gonna depend on the agent frameworks that... How how they expose us, but things like the the tokens used and, then, ultimately, whether or not this was a successful, interaction or not. And we can also, assign some, human feedback. Right? Like, some some sort of score is going to be, important. Okay. And then that's how we record it, but then, like, how do we how do we use this? Well, we create a tool that we then can give to our agents to use during, during the reasoning phase. And, again, we use similarity search. So so vector search, like, find, find some task similarity, and that's based on the the message. Right? The message that triggers the reasoning, trace is what we're doing a vector search on to find similar, similar, like, execution plans, essentially. So if I say process a refund for, some customer, that's what we're going to, to do the vector search on. And and we can set some threshold, that that can be, you know, configurable. We don't wanna give every, every reasoning trace that, you know, maybe like the the failed ones, that's maybe not so helpful. So we may want to to rate those by the... By score or some some sort of option there, to think about. Cool. And that's what that looks like. Here is, fetching the, the trace with steps, once we've done that. So let's switch to module four. So I'm gonna go stop my previous agent that was running module three. I'm gonna say make dev module four and run that, and that will take a moment to get started. And then the commands that we can run are things like, scenario four refund. Now let's run that and see, how we're gonna process refund. So make scenario four, refund. We can see here we're using the Acme Cloud specific ontology. And I think we're gonna run through, some agent messages for processing a refund and recording, those traces and then, looking at the the actual tool calls that were made to do that. And while that's running, we can go back to our our agent here. And, again, just for for the purposes of seeing, like, how the the graph grows, we can say, like, I have billing issue. I'm just charged twice. My last invoice, please refund. And so we're going to store short term memory. So we'll see the the message graph layout here. This is nice. We know from memory that, yep, we know who you are. We know that this is Maya. And we can see that there was an existing refund, it looks like. And so now we can see this is the reasoning memory piece that was initiated by this message. So this is the trace that has, the step. This is the first step, which resulted in a tool call, which was an instance of this tool. If we go back to our Neo four j database, we can, maybe see all of the nodes. We can trace this this backwards. Right? So there was a refund issued. I think these are from, these are from the messages that we're running in the command line. So this is our... Let me get some more space here. Right. So to see what this looks like, here's our conversation. And Customer asked for refunding our duplicate charge that, you know, resulted in these entities being extracted. So the customer and the product. Right? So compute is is some product name. And then that started this reasoning trace. And we can see here the the number of tool calls. I don't think we're capturing the tokens in this case, but we are capturing, like, the number of tool calls, which could be a a good proxy. And so here's... This was the the step that, in this case, we... Because this aligned with the policy, we just went ahead and issued the refund by calling, this tool, which is issue refund. Cool. So now let's run our eval again. And so last time, we were... We passed three out of four in our eval. Right? We were we were able to keep track of the customer email, the, customer preferences, the, like, saved, issue, like, keyword for the for the, billing issue. And then the fourth one that that we weren't able to pass was, the, there's our our first one. Right. So did the customer... What name and email did the customer use? Yep. That pass. The exact verification phrase? Yep. That pass. Sign up date. Right. This one came from, the long term memory extraction piece and hydration, and then the refusal quote. And this one, this is more of a, auditing piece, and so this is like an explainability piece. We we ran this, essentially saying, you know, hey. I want a refund, and based on the policy, the user is not eligible for the refund. And then in a later conversation, we said, hey. What was the the rationale for refusing, this refund? And then by tracing the reasoning phase, the reasoning traces and and the rationale for that, we're able to go back and some explainable answer of, oh, in this case, you were not eligible, based on the policy. And that one, did we run... Is that this one? We have that in the slides. Cool. Oh, yeah. That was this one. Scenario four failure to issue a refund, which didn't actually run, in the command line, but that ran, as part of the eval. Cool. Well, we have about, twenty minutes left, and we have, one more module. And I wanna introduce the Neo4j agent memory service, which maybe we... We've saved the the best thing, for last here. But, yeah, this is this is explaining what we were talking about. It's like, why did you issue the refund or why didn't you issue the refund? That's the the explainability piece. Okay. So the last piece here, which it says is gonna take twenty minutes. Actually, we're we're right on schedule, and that's this concept of skill distillation. And so this is where we're gonna switch from the, Python package to introducing the Neo four j agent memory service. So if you if you wanna, sign up and play around with this, memory.neo4jlabs.com. That's the the URL. This is a free service. You can can sign in and get started. If you wanna try that out while I'm I'm talking through that, go ahead. But we're gonna go go through it and see what it looks like, in a minute. And this is gonna be module five in the in the repo. So so far, we've seen, like, how we can use agent memory to, like, improve our agent. Like, okay. It's able to, to be, a bit more efficient. It's able to, you know, keep track of user preferences and and these sorts of things. And and that's fine. Like, we we can give our agents, tools to explore and and read memory. That's great. That sort of is is, you know, record of of what happened. How do we tell our agent what to do? This is where skills come in. And so that's what we're gonna look at in this last module is how can we create and govern agent skills from a memory graph. So if we think of, like, two kinds of knowledge here, like the the descriptive knowledge, like facts, context recall, that's what we... We've done so far. But how can we, like, structure knowledge so an agent can act? And this is where skills come in, skills, agent skills. This is a, standard that I think was anthropic that created this and and then opened it up. And I think the spec for skills is now part of the Agentic AI Foundation. So this is a a well established standard that most, agents support. And one of the the fundamentals of agent skills is this idea of progressive disclosure. So we have kind of, like, some metadata, that describes, like, roughly what the skill is. Then we have, some, you know, more in-depth information in a markdown, bodies. Think of the metadata as, like, the the front matter in a markdown file and then the body has more information about, like, the core procedure essentially for, for what the skill is and then even more information like references and code. So it's not just a single markdown file. This is like a like a bundle, that we can can give our agents. And this is this is... Idea of progressive disclosure is nice because it means that we don't have to inject, like, all of the skill content into the the context window, we can just inject that that metadata. So there's some little information that the agent can then fetch more. It knows like, oh, this is the skill for issuing a customer refund, and I and I can fetch more information about that. Typically, like, we we think of skills as some, like, pros, some, like, markdown file. But, there's some some challenges in working with skills just as prose. And so for example, you know, how do we update? How do we govern a skill? If it... If it's just a bunch of text, that can be a bit of a challenge. But, some folks at, Neo four j on our research team, Zach and Jim published a paper recently, about how can you model skills as a graph and what benefits do you get from that. So if you think of a skill as like a a typed execution graph with, with steps, sort of like the addressable unit of a skill, well, then you can do, like, some very interesting things. You, you don't kind of have to, like, reread and and, like, reinterpret the the skill. You have, like, these steps that can be, addressed specifically. So if the agent maybe wants to, update and maintain the skill as as things change, You're not just appending text. You can actually edit the the skill steps directly, since that's broken down in a graph. They... In the paper, they looked at some of the the benefits, using skills bench, which is a a benchmark, for measuring how successful, skills are and and found that this, agent instruction protocol, actually, like, helps quite quite substantially for, skills successfully executing. Cool. And so in the the agent memory tooling, we took this idea of modeling skills as a graph and added a feature that we call skill distillation, which allows you to construct skills, that are grounded in agent memory. So grounded in your memory graph and in the conversations, the entities, the reasoning traces. And create the skill, but then also govern the skill so that as those, underlying instances change... We have a slide about that. I don't think we have a slide about that. As those underlying instances, of your memory change, as those entities change, as maybe the the tool call results are, are observed to be different from what's in the skill. We can flag the skill as stale, and then that needs to be, to be redistilled. Cool. So the last piece in the workshop is looking at, this feature, the skill distillation feature in the hosted Neo four j agent memory service. So we have just a few minutes left. So let's just go ahead and jump in. I shared the link, earlier. This is the Neo four j agent memory service. This is my previous workspace. So the the idea here, should we create a new... Let's create a new database. This is my... We run out of capacity. Cool. Well, hopefully, lots of folks are trying this, and we we ran out of capacity. That's fine. I can find an existing one, and we'll just take a look at, what this looks like. Do I have an existing one? Yes. Let me switch users here. Here we go. And I think I have one of these running. This one, I think, has some data in it. Cool. Oh, or, actually, I ran this one earlier today preparing for the workshop. Cool. So the idea here, we have, the the... Lots of similar functionality to what we saw in the agent memory Python package. So we have these short term, long term reasoning memory abstractions. And so I've I've, if if you run the the module five, make command, it will load this and and then go through kind of the entity extraction piece. Right? And so we've extracted out the, the entities. We can observe our, reasoning memory pieces as well similar to what we saw in the other, in our other example using the Python package. We also have the concept of ontologies. So someone was asking, you know, how can I, how can I represent the ontology? There there are lots of different ways of representing ontology or or a data model in in different tooling. Right? Well, I can import, say, for example, a a turtle file, which is, come from the RDF world of representing an ontology or, some of the other data modeling tools that that we might use, with Neo four j. There's also a a catalog of, preexisting, ontologies. And, what is nice about this service is because it's a, because this is a hosted service, we don't... There we go. I was looking for the docs. We, we have both a REST API and, hosted MCP that maps to the short term, long term, and, reasoning memory, abstraction so that our agents can configure all of this. We can give our agent these tools, point it at the hosted MCP and, very similar to the functionality we saw in the first four modules. But what we haven't seen yet is this idea of skill distillation. And so what we can do... Let's go ahead and see what data we have loaded. So we have, we have a few conversations loaded here asking about the billing API, processing a refund, and if we've had a successful interaction, with our our agents. So once an agent has gone through the reasoning steps and figured out how to... What, what tools do I call? First, do I look up the policy? Do I look up the customer first? How do I verify, the issues in the policy? Do I go back and see how how many refunds the customers had previously? All these sorts of things. The first few times an agent does this, it needs to figure out... Needs to go through the the reasoning phase to understand what tools to call in what order to accomplish this. But once we've had, you know, one or more successful interactions where the agents, the agent has done this, we can distill a skill. So So I'm gonna do this from the, from the UI, select the initial scope, which can be a conversation, or we could do this for the whole workspace. And then this is going to start the process of looking at the memory graph, so the reasoning traces, but also the the entities, and create a skill that we can then, we can then review. And so this is, the process for handling a duplicate, duplicate charge for a refund. And it's quite simple. First, we verify, that the user has the pro account by calling the lookup account tool. Then we fetch the refund policy, to verify that the refund amounts, are correct. And then we call the issue refund tool. Now you can imagine, like, this one is quite simple. We could have something, like, much more complex. The... Actually, the patient journey example I was gonna show that has a much more complex process, that involves calling, like, many more, tools. But we then have the skill bundle, that's, in markdown format that we can... Here, we we can download it, but we also, we can also fetch this dynamically through the API, and this becomes sort of a skill registry, essentially, for our agent. Now we have just a couple minutes left, so I won't go through this. But I will... Do we have a few slides on this piece? Yes. So the the piece that I won't show, but that I think is is really interesting. If you, if the... Any of the memory that the skill is grounded in changes... So for example, if we were to, lose the, one of the tools that we... That is a requirement for the skill or if there'd be some contradiction, say, like, the the policy changes, then that would... Skill would be flagged as stale, and we would need to redistil that. Cool. So you can play around with that, on your own. I would encourage folks to to sign up for the Neo4j agent memory service and, play around with that. There is a, where am I? Here we go. There is a, feedback form that would really appreciate, folks filling this out. This is, an active involvement, piece that we're that we're working on. So appreciate any, feedback that folks have on helping us shape the the development of the agent memory service. And with that, I'll just point out that in, in the slides, the last few few slides are just links to a bunch of different resources. A lot of the the docs, for agent memory, the, the different, yeah, resources there. So feel free to to explore that stuff on your own. The repo that... Everything is, listed, in there. I'll drop this link in the chat one more time if folks are just joining and didn't see it. Oh, it's gonna be a very large... Okay. There we go. There's the link to the slides. Yeah. Cool. And with that, I think we are about at time. I don't know if we can take any, any last minute questions. I see one from, from Alfonso. Yeah. This was this was a lot to, a lot to set up initially. So I'm sorry you weren't able to to follow along exactly, and that is... That's on me. I I was, late in getting everything, built for the workshop and, didn't have a chance, to send out the the prework email that, that Alex usually sends out for the other workshops. So sorry about that. But, yeah, hopefully, you have the resources still to, to play around with things in the future. Yeah. Absolutely. Hope... Hopefully hopefully, this is, like I said in the beginning, we have, recorded it. We will share the recording with you all. So even if you couldn't do it live with Will at the moment, you can always, follow it up, at another time. I know we are out of time, and so I I think everybody... For everybody that has to leave now, thank you for attending. Thank you for, for your question. Thank you for working along with Will, and thank you for your questions. And, yeah, hope to see you next Thursday for notes, one week away. Again, as we said, lots of interesting content about ontologies, about agent memory, about graphs, obviously, about all the topics that are interesting, for people at the moment from the Neo four j engineering team, from Neo four j community members, but also from, from customers and, and, enterprise users. So it'll be a very, very interesting day. We have an interesting keynote by by, fireside chat between, Emil, our CEO, and Andrew Ng from deep learning. So so very, very interesting context. Thank you very much to Andreas and Jeremy for a whirlwind, activity in the back on answering, all the questions. Amazing support. Thank you very much to you. And, yeah, if if you have to drop, please please drop. Obviously, no no harm in that. But, I'll... I think we have a bit bit of time, Will, right, for maybe answering one or two more questions that that came in. I think that was, was interesting. So, For sure. Yeah. one was... Yeah. Cool. One was, from from Oscar. Can you give us the main argument for using Neo4j or graph system to store these agent memories conversation instead of using a relational database such as Postgres, for example? Is there one, you know, culminating advantage? Yeah. I think, it's interesting to compare to, right, like, memory in a relational database versus a graph database because I think the the first hurdle that that folks often get over when they're talking about memory is just using a database at all versus, markdown, which is kind of, like, the the default, like, memory that that a lot of agents, will use is like, okay. I'll just create a bunch of markdown files. And so, like, I think once you once you move into these, like, multi agent or or multi user environments, like, where you you need to have a database that can support, you know, multiple agents reading and writing from them. Like, that's that's certainly the case of moving to a database from markdown files. But on the question of, like, why should I use, like, a a graph database instead of a relational database for memory specifically, I mean, I think to to cut to, like, one example of things you can do, with a graph is you can build on top of your memory. So we saw, like, the the interesting pieces of recall and and things like this for finding, like, the relevant context. And, the the graph is a very flexible model for, this, like, entity, the hydration pieces as we were adding additional, information there. But, like, the real interesting bit to me is the the thing we went through at the end, which is skill distillation. Right? Where we're building on top of our memory. It's not just something that we're, you know, fetching and and injecting into the context of our agent, but we're actually, like, creating these skills, modeling them as a graph, and grounding the skill steps in the observed memory so that as those change, we're able to flag, like, which specific piece of our skill, is no longer relevant and and redistill that. And then our memory becomes something we can can build on top of. Right? And and I think this is something we can do in a graph, that we can't do in, say, like, a relational database or or other, like, storage structure. Cool. Super. And then maybe to to close it out, because... And because we talked about this on the... Undergoing meta episode on, Tuesday. We we talked about Jeff and decision models and and graphs. How how would something like Jeff work with with something like agent memory? Yeah, that's a great one. There's two things that I think are really interesting for agent memory that we didn't talk about today. One is using decision models, and the other is using graph algorithms, like like GDS in in Neo4j. I'll I'll save the the graph algorithm, discussion for another day, but but just talk about the decision model piece. Yeah. I I think this is something that we're gonna look at, look at adding these features to to Neo four j directly and and also to our our agent memory tooling is you can, you can much more efficiently decide if, say, a memory is worth saving or not. Or, right, like, we talked about, JEV as being like a like a classification model or a decision model that is very, very inexpensive and very, very fast. And so whereas before, we might, want to say something like, to the LLM, you know, does this preference or that's been expressed or is this, memory that we've observed, is this worth saving? Is this currently accurate? Does this, conflict with another memory that we've saved? Right? Like, all all of these sorts of things that we might want to do with an LLM to, like, help curate the memory graph, that, in practice just ends up being too expensive and too slow. This is like the, the dreaming process that runs, like, overnight or something because it's... It can be quite quite slow. But instead, if we have, like, a very quick and efficient decision model, could we do those in in near real time? Can we can we use that for curating our, our existing memory, in real time if the decision models are are much faster and and and cheaper. So lots of, exciting things, I think, that we can can explore there. Definitely. Very exciting. Sounds sounds very, very exciting for sure. Yeah. I think, that's that's it. Thank you very much for, for sticking around. Thank you for your questions again. And as I said in the beginning, material to the stack from Will plus the recording will be shared with you all. Look into your inbox for that tomorrow. And, yeah. With that, have a good rest of the day, rest of the week, and see you at notes in one week time. Thank you, everybody, and, bye bye. Cheers, folks.