Name: Hands-On Lab: Building Self-Grounding Agents with Microsoft Agent Context Provider with George Bittencourt, Microsoft & Ryan Knight, Neo4j
Uploaded: 2026-03-11T19:16:11.262Z
Duration: 1 h 58 min 44 s
Description: Hands-On Lab: Building Self-Grounding Agents with Microsoft Agent Context Provider with George Bittencourt, Microsoft & Ryan Knight, Neo4j

Transcript for "Hands-On Lab: Building Self-Grounding Agents with Microsoft Agent Context Provider with George Bittencourt, Microsoft & Ryan Knight, Neo4j": Alright. Just in time or on time, actually. I'm very happy to welcome you all to Road to Notes. It's the second Road to dot a I, I wanna say. Second part of our workshop series leading up to notes.i happening on the April 15. This time, the session is called hands on lab, building self grounding agents with Microsoft agent context provider. And in this in this hands on session today, we will be using Microsoft agent provider with Neo four j to build agents that automatically receive graph context and that that help you, you know, improve your your retrieval, improve your your code, and get you more accurate explainable results in the end. So I'm I'm very excited to have you all join today and particularly excited, obviously, very well warm warm welcome to to the two speakers who will be conducting this workshop today. And it's Ryan Knight, is a senior partner architect at Neo four j, and George Witten Kut, who is partner solution architect at Microsoft. So these two will be actually conducting the workshop today. Before I let them take it on, obviously, the session is recorded, so you will get will be getting a link afterwards with the video so you can watch it anytime you want. So something was too quick or something you want to watch again, then, obviously, no problem. You can look at this video recording afterwards. There's also a little document, handy document with a couple of scripts or with a couple of information about the workshop today. We'll share that as well. Please don't be shy and put your questions in the q and a tab. Don't use chat. You you should see on the right hand side of your of of this video, hopefully right hand side, but, you know, it should be there. You should see three chat buttons, and one says q and a. And if you have a question or comment or something, rather questions, put them in q and a and send it in so we will have some time to answer the questions. If you have generic comments or, you know, some feedback or something, you know, and wonder says well, just wanna say hi, and I see a couple of people doing that now from India, from United States, from Mexico, and and beyond. So well, welcome to all of you. You can use the chat function, but for questions, please use the q and a because otherwise, we might get you know, we might miss your question. Yeah. With that, I think we can we can kick it off. George and Ryan, you need to click on the green button on the bottom right that says go on stage, and that would pop you over to where I am. Here go. Hi, Ryan. Hello? Hello. Perfect. Hey, George. Do wanna start by introducing yourself first? Sure. Thanks, Ryan. So my name is George. I'm a partner solution architect here at Microsoft, and my focus is AI at Microsoft currently. And I like to say thanks for Nelto Nelforj for the opportunity to come to this event here to present about Microsoft agent framework. And let's see. You're gonna love the workshop that we created for you. Alright. Ryan Knight. I'm a senior partner architect? at Neo four j. My role is on architecting the integrations between Neo four j and our cloud providers. Primarily focused on Microsoft. And so we're working on building out, you know, combined Gen AI solutions between Neo four j and Microsoft and how those two platforms can be leveraged together. And what we'll be going over today is exactly one of those primary integrations, which is the Neo four j context provider for the Microsoft agent framework. First, I wanted to kind of go through the the setup instructions. Let's see. I don't know. Did they get posted in the chat? I don't see them in the chat. Let me drop the link in the chat. Yeah. Sorry. Alright. So that if you guys follow that link, there's, there's that'll take you to the the the hands on workshop and the labs. As well, there's some an API key that we're gonna use for running the these labs. So the one requirement that you will need is you need your own GitHub account because everything is gonna be run-in, GitHub DevSpaces. So let me share my screen and kinda walk through that. So let's see here. Window. Alright. So to set up your development environment, there's there's a couple of steps. The first is you create a a Neo four j sandbox. You go to sandbox@neo4j.com, sign up for a free account, and then you select the project, and you scroll down, and you choose the recommendations dataset. So I'm already logged into sandbox in Neo four j. What you would do is you'd scroll down here. There's this recommendations dataset. So be sure to use this one and not the movies dataset because the labs all depend on this recommendations dataset. I select that, and I do create and download credentials. And then you need to be sure to save those credentials and then use those credentials for the the remaining labs. Alright. So there it is. So, yep, I got the credentials, and I've downloaded them. So then after that, the next step is to launch GitHub Codespaces. And when you launch, it's gonna ask for an open API key. In that doc I just shared, there's an open API key there that you can use. Just ask that you be respectful so everyone has a chance to run the labs. And then use the URI and username and password from that sandbox instance that you just created. And so if I go here, open new code space, create new code space. And so I already have my fields populated, but it's in here. The open API key, the URI username and password will be filled out, and then you click create code space, and it sets up the code space. So now the one thing to be cautioned of is when you open up the code space, it takes probably five to ten minutes to provision everything. And what you'll see is it it starts up here, and it looks like the terminal started. Everything is set up. Well, actually, in the background, it's still running and processing everything. It's running, like, post processing setup steps. So you need to give it five, ten minutes before you do anything because it'll populate an environment file, create all the the setup instructions and everything. And so give it a minute. And then after you do setup, there's there you'll see a a a markdown file in the main window. You need to be sure to follow the instructions there. Primary thing is is this step. Right? So you you need to verify your setup just to be sure everything works, and then you need to load the embeddings and the create indexes. So after it populates, you'll see here in the terminal, you'll be able to enter a command to the terminal. So in the terminal, be sure you run this setup command to load the embeddings. So give it a minute to load. What we're gonna do is we're gonna have George give us an intro to the Microsoft agent framework and context providers, and then I'll be on the messages kind of answering questions trying to help people out as much as I can. Yeah. Ryan, So before before we hand it over to to George, could you recap one more time the the the initial setup process? It was a little couple of quick, and people need to need to chart a couple of windows. I think that would be helpful to to get everybody on the. same same pace so that we don't lose lose people. Thanks maybe let's do it one more time. Alex. Yeah. So first step, create the Neo four j sandbox by going to sandbox in Neo four j. And in there, you'll see this a a recommendations dataset. So because I've already done it, I don't I I I won't see it in here. But you scroll down in here, you'll see a recommendations dataset. So do that one. Yeah. And then click create to launch the sandbox, and then download your credentials. And then the second step is to launch the GitHub Codespace. So you just click on open up put in GitHub Codespaces, and it's going to launch the Codespace for you. And the key is is when you launch it to fill in all your credentials. So I already have mine populated, but you'll see where you can enter your credentials in here. So you'll need to go in and and take from that document the API key, the URI, the username. And then when you launch, you'll see in here so it says finishing up. So it's going to take a while to spin up, but you'll notice in here that there the setup file has appeared. And so be sure to go through this setup file, verify your setup, and then load the embeddings. So be sure to do those steps, and you'll be able to enter those commands in the terminal after this post create command. So yep. And cool. Alright, George. Turn it over to you. Okay. Thanks, Ryan. So let me share my screen here. Can you see my screen, Brian? Is. it. okay? Okay. So let me skip these two slides here. And this is the agenda for this quick introduction about Microsoft agent framework. We're gonna I'm gonna provide this introduction for you for the next twenty, twenty five minutes, but we're gonna talk about the Microsoft AI stack, how you can create agents using Microsoft agent framework, what is to call in an agent extensibility, orchestration, multi agent patterns, observability evaluation and memory. Okay. So inside Microsoft, we have basically two ways of creating agents. In the software as a service offering, we have Microsoft Copilot Studio. Okay? And in the platform as a service offering, we have Microsoft Foundry agent service. And in the infrastructure as a service offering, we have a lot of virtual machines, a lot of orchestrators like Microsoft Azure Kubernetes or Azure Container Apps. The difference between them is that as you go to the right side here as a software as a service let me enable this slider. As you go to the to the right side in software as service offering, we have Copal Studio. So you can very easily and very fast create agents using Copal Studio. And on the other hand, in the right side sorry. In the left side, which is infrastructure as a service, you have more control, visibility, customization. And in the middle, with the platform as a service offering, you are exactly in the middle. We have some control, some visibility, but it is also easy to create and and we can and you you can create agents very quickly. Okay? So this is another chart to compare the three offerings. So, basically, Copilot two d is a fully managed, offering where you can which which you you can use to create agent. Right? And in the infrastructure as a service, you will have full control. So you can decide the orchestrator you're gonna use, like Azure Container Apps, Azure Apps Service, or Azure Kubernetes Services. You can select the virtual machine. You can select the framework that you're gonna use for your agents. Right? And the goal of this talk here is to talk about about Microsoft agent framework. And another item that I want to highlight from this chart here is the pricing. So when you use Copilot Studio, you pay per message and and and when you go with Microsoft Foundry agent in service, you pay per token. And if you use Microsoft agent framework, you're gonna pay per token plus the compute that you have allocated for your agent. But anyway, doesn't matter the offering you choose for your agent. You need to provide your agent with a model with you need to provide instructions on how you want your agent to behave, on how you want to to to to act. Right? And also words. As you can see here in this different, screen here, this is Copale Studio where we have all these three, informations here, the model, the instruction tools, the same for Microsoft Foundry agent service, and the same for this code here in sharp, which is creating the agent using some entry kernel. Okay? So a very important concept that you need to understand is that most use case today, we can solve using workflows. We call them agent workflows because we already have for most use cases this predefined code path, and we just and we just need to go and add some some agent at some specific place. Right? That's why we called in agent workflow because we have this workflow each which is deterministic. Right? And we simply go and add an agent at some very specific points to use the power of the models. Another way to orchestrate is to use agent orchestration. In this case here, we don't have a predefined code path and we want to let the agent we want to let the model the agent to decide is the next step for that task. So it's gonna use the model reasoning power and the tools that we have defined with the instructions that we have defined to orchestrate this whole process. Right? So which I would say that most probably 80% of the use cases, you can solve using an agent workflow. And the goal for this talk here is to show you the extension that, Neo four j created for Microsoft agent framework. So Microsoft agent framework is this new framework that Microsoft created. It is issue in public preview, but we believe I believe that soon it will become general available so everyone would be able to use it. But Neo four j is ahead and they already created this context provider that we're gonna explain to you soon. But, anyway, Microsoft Azure framework, it is being developed as an open source framework. It already has support for Python and c sharp. So and I believe that they soon will add support probably for Java. And it is the it is the the when and Microsoft created agent framework as a result of two other frameworks we have because we also have some kernel, which is which supports Python, c sharp, and Java, and Autogen. So what Microsoft did was to was to take the best from these two frameworks and create Microsoft agent framework. So if you are already using any of any these frameworks, you wanna see a lot coming from from them. And Microsoft agent framework provides this unified SDK. So now we don't have two SDKs. We have a single one. Right? And it provides some very simple agent primitives so you can very easily go and create agents. And we have some libraries and these are restrictions. They abstract all the details behind the models. Doesn't matter if the models is from OpenAI or on. And you can use oh, sorry. And and you can run your agent locally and and inside Microsoft Foundry agent service because both frameworks, they they have this sherry runtime. So you can get the agent you created in agent service and run-in your in your local computer or or in a virtual machine if you want to move from a PaaS to a a a infrastructure as a service offering. Right? And as you can see here, we have different ways of running, as I said, using Azure Container Apps, Azure Kubernetes Service, Bedrock or Vertex. In addition to that, we have this huge ecosystem because Microsoft Azure framework, as I said, is being developed as open source but it has this open ecosystem. We support MCP, H2A, and OpenAPI. So we have already the issue to the one that we're going to use during this workshop today. So it is extremely accessible. It all agent framework, it also supports a lot of different agent orchestration patterns like sequential, which is this the simple one. Right? So we basically define a set of agents, and they're gonna be executing sequence. But we also have concurrent where we have a a different agents executing parallel for the same task. Hand off where I where I delegate a task for one agent, that agent will delegate the next agent, so it will keep handing the the the task to one agent to another. We have group chat where we have, several in this case here, we have a worker and a and a and a reviewer. The worker creates something, the reviewer review. So this this is another workstation pattern we have. We have the one and we have workflows as I just said to you. So in this slide here, you can see how Microsoft Azure framework is. We have support for several AI services. So all models that are available in our Microsoft Foundry model catalog is available for using Microsoft Azure framework. We also have support for local models using founder local or or Yammer, for instance. We have support for several memory services. And we have support for as I said, you can very easily integrate that Microsoft Agent Framework with Copilot Studio or Foundry Agent Service. And OpenAPI, MCP. We have this declarative way of creating agents. We use an IAML file. And we also support OpenTelemetry, which is the de facto standard for telemetry today. But before we talk more about agents, we need to understand what is what makes what turns a model into an agent. Right? So it all started when OpenAI, they added a capability for models to generate JSON as the output of a of a request. Right? And and the idea is that we provide our call, our request, our prompt with the tools that we have available in our application. We need to provide them in a JSON in a very specific format using JSON. Right? In the model, we decided based on on its reasoning power and on the information they have in the context. Right? What is the best tool to accomplish the goal that was asked? So as you can see here, the developer defined it a tool called GetWater. GetWater. Sorry. And and the user submit a question. And how they decide in this case here to equip this tool, GetWater for this argument, which is in this case here. And now the application will receive this information, this JSON text. Right? We will extract the name of the function and the parameters from that JSON. We will execute that code in in the application, and we will send the output again again for for the model. And now the model have now the model has all the information that it needs to generate in an answer like this one here. So that's basically the the whole idea of an agent. Right? There is some discussion if an agent without tools if it is really an agent. In my opinion, it's not because we need you need really a tool to make to allow the agent to interact with the external word. And an agent can have more than one tool. In this case, here we have three. But imagine we can can plan our agent so we can provide more tools to then get weather, get activities, and during the date. So all these tools all these tools together will help the agent to execute its function. And we also can use MSCP. So in this case here, I'm creating an agent. And in this case and I'm pointing to a m e c p server. In this case here, I'm using the HTTP protocol. And automatically, this agent will go to that m e c p server. We will bring we will add to the prompt all the tools that are available in that m e c p server, and we'll do all the interactions that is needed with that MCP server in order to allow the agent to execute its phone call. As I said, OpenTelemetry is the factory standard for telemetry, right? And Microsoft Agent Framework, by the way, it generates telemetry using OpenTelemetry. Traces, metrics and logs are automatically generated. You just need to go and very easily enable that. In this case here, this line configure, hotel providers will enable the generation of OpenTelemetry data. Okay? And you can send for any destination. We have, for instance, Microsoft Aspire, which is a local development tool that you can use to receive this open telemetry data, but you can also send it to Azure application sites. In this case here, we are, you can see here in this case we are seeing all that data that was generated by Hopi telemetry in Azure application sites so you can go and monitor your agents running. Very important subject that in agent development is how can we make sure that output that is generated by agent is correct because the output is non deterministic, right? So if I ask the same question twice, I will have two different answers. They will be probably correct, but the way that they are written will be different. So how can I make sure that both are are behaving correctly? So one way is using what we call a human grading. So we can let a person come and evaluate the answer that was generated and evaluate if that answer is correct or not. However, that process doesn't scale. Right? And in order to scale that process, can create an automated evaluations. So in this case here, you can see some evaluators. So Microsoft agent framework has some evaluators built in in the framework. So one of one of evaluators is groundness. So groundness is will measure if the answer that was generated by the model is grounded on the documents that I added to my prompt. Right? But we have much more evaluators than these five ones here. We have evaluators to measure if the tools that I added to my prompt, they are being used correctly. There are a lot of evaluators. So one thing that I want to highlight is that today, the products that generate a lot of value uses evaluations since the start, since the beginning. Another component that we have inside Microsoft Agent Framework is agent middleware. So imagine that when you execute an agent, a lot of steps are being executed by Microsoft Agent Framework. So you can come and add what we call a midware. So you can inspect and we can change any message that is being passed during this pipeline here. So we can add a midware in agent context, in a chat context, or a function evocation context. And for each invocation, we're gonna receive some information for that invocation, and you can use that information to change the request or maybe to cancel the request. As you can see here, I have a middleware that's basically measuring the time that it is taking to execute agent a call. So when I create this, when I added this middleware in my agent, it will measure how long each call is being, is is taking to make a a call to my model. Okay? But I can do any other, thing here. I can for instance, if there is some specific words in my prompt, I want to stop the processing of the call. So I could use that to maybe do some filtering about what is being sent to my model, for instance. So agent midware is very powerful for that. But we also have memory in inside Microsoft agent framework. We have session memory, chat history, and dynamic memory. Okay? Session session memory is the memory related to that conversation that I'm having with my agent right now. Chat history, it remembers conversation that I had before. In dynamic memory is when I extract facts from my conversation, it starts somewhere. So, for instance, if I say that I like I like orange, I can have a component that will suck that information from my conversation and will add and will create that as a memory. And later, when I start to interact that agent, it it can come and add that to my to my prompt automatically. Another thing is knowledge retrieval. So this is where RAC comes, retrieval augmented generation. Right? And in this case here, we are adding more information. We are going to a knowledge store and are bringing data from that knowledge store and we are adding to our prompt that we are sending to the model. And we are doing that in hope that that information will help the model to generate the best answer. And as you can see here, this is a nodule retrieval for C. Coli and it is using what we call a context provider that Ryan is going to show us soon. Right? And this context provider here, it basically goes and do a select in my SQL like database and add all that information to my prompt. Right? And I can very easily create an agent here and point to that context provider. And if you want to learn more about Microsoft agent framework, we had six series about agent development using Microsoft agent framework. They are available on YouTube. I'm gonna drop the link in our chats later. So you if you want to learn more, this is a great, great, asset for you. Ryan, that's my Thanks, George. Really appreciate it. Yeah. If people have questions, you know, drop them in the chat, and we'll try to get to them as possible. George, one question everyone was asking for a a link to your slides. So if you could create, like, a publicly accessible link, that'd be great to share with everyone. So Sure. k. I'm going to walk through kind of go through the the next setup steps to kind of give you a preview of. the lab. I'm gonna do that. Can can you hear me, Yeah. Brian? I can hear you now, George. Alright. Okay. Give me let's see. Where's the screen share button window? Share. Alright. So my GitHub Codespace launch, it does take five minutes. It's it is pretty slow to load everything. You'll notice in the the file loads very first is the setup dot markdown, and it has two two scripts you need to run. One is just a test environment just to be sure your credentials work. Drop that in. So all that does is that's just checking your credentials to be sure that the credentials you added work properly. And okay. And then the other is setup script, and this adds the embeddings to the database. So I have precomputed embeddings that I add to the database that we're gonna use in later labs. And then there's labs here that just walk through the concepts that George just talked about at a high level. So the first lab is how you create a an agent with the Microsoft agent framework and walks through an example of that. It uses a hard coded movie dataset. So the very first lab doesn't connect to Neo four j. It's just a hard coded dataset and, you know, just a basic tool to get the movie information, and then you create and run the agent. The one critical thing is is you need to hit select kernel. So this is gonna select the the Python kernel to run this notebook in. So I select it. I go to Python environments and then Python three twelve. So now I see that I've selected the kernel Python three twelve, and I can hit run all. And this will run through it. And because I have because I've added the open a AI key in here, it's actually going to call the OpenAI. Everything is set up and configured in your environment file. And this lab is all structured around GPT five mini. So it's going to use the GPT five mini and actually call that for your agent. And then yeah. So the next one is yep. So it's yeah. Recall that it was able to use our our kind of fake, movie tool to get some information about movies. So next is the context provider, which George talked about. It's just a simple context provider. And the general idea with context providers, and we'll go into these more, is that you can add hooks in before an agent runs and after an agent runs. And this just shows an example of a simple context provider that you can build for the Microsoft agent framework. You should be able to run that. And then, again, I need to select the kernel and run that. Connecting to kernel. Yep. And then it should go out there and run the kernel. In terms of so people are asking for the slides. So in the the the workshop material, the lesson content is all on you know, you can walk through it here. Down at the bottom, there's also a link to these slides. And so these are the slides that we'll use that I'll go through today. And so George caught covered kind of what is a context provider, some of the basics, and then I'm going to go into what like, the Neo four j context provider specifically. So let me pull up this slide. So what George talked about is how, like, a context provider can do a what you're doing is, basically, you're adding hooks into the agent life cycle so that before that agent gets called, you can call your context provider to to add, context to the agent calls, or you can add a hook in afterwards to see what the agent returned. And so the idea is that I can add context before the agent is called to inject it into agent to give the ally more context to prevent hallucinations, and then I can extract information out afterwards to store it as memory. And Neo four j provides two context providers. One is the agent framework Neo four j, and this is retrieves knowledge from Neo four j indexes. It uses graph frag underneath the covers, which we'll go over a little bit, and injects this context from the graph into the agent, which is then used for every LLM call. And the other is agent memory, and that hooks in after the agent call and takes all the responses from the agent and stores them in Neo four j as agent memory. And so this module we're first going to talk about is the agent framework and the graph frag. And then on the next module, we'll go through agent memory. So the the Neo four j context provider is all it's basically a wrapper around the GraphRag Python library that provides the tools for calling and retrieving data from the knowledge graph for this graph frag. It also contains the tools for building a pipeline and building the knowledge graph from unstructured text. So there's something called the simple knowledge graph pipeline. And what you can do is you can feed it something like a PDF or articles, and it's going to split that article. It's gonna do chunking, entity extraction, find all the relevant entities to store in your graph, creating embeddings from the chunkings, and then store those out. So the simple knowledge graph pipeline helps create the the knowledge graph, and then the the the same library also helps on the retrieval side. So the Neo four j context provider, what it does underneath the covers is it's going to connect to the Neo four j database, and it's gonna search an index, which is configurable. And that index can be either vector, full text, or hybrid, and we'll go through each of those types later on. And then based on the search results it gets, it's going to inject those results into every call because it hooks into the the agent framework before running call. Then underneath the covers, it'll delegate all the searches out to that Python graphite library. So the Neo four j context provider supports three search modes, and these three search modes are base the same three search modes that the GraphRAG library supports underneath the covers. And this is kind of the foundation of GraphRAG is these search modes. The first is vector search, which is similarity search. So this is where you create embeddings. So if I were to take unstructured text like a PDF, news articles, blog posts, the way you create an the way you ingest those is you do what's called chunking. I want to just take small sections like paragraphs. There's different checkings. Like, you know, I can do paragraphs. I can do length based, you know, like, fixed size chunk and things like that. Take each of those chunks, create embeddings, which is creating a a vector of, basically, a mathematical representation of the text. And then I can do, similarity search. So I can use those same embeddings to search for the meaning of those vectors. It also supports full text, which is b n 25 ranking on tokenized text. This is full text search based on Lucene against the graph, or I can combine both of those. And those runs both and combines those scores the scores of both of those searches and basically filters to the top. So if you know, depending on, you know, which search result returns better results, vector or full text, those will be bubbled up to the top. And so then the power of GraphQL comes in where I can do a graph enhanced search off of these to enrich it with knowledge from the graph. So the idea with graph enhanced search is I'm going to do that initial search of doing the vector, the full text, or the hybrid search that I've configured. So it's gonna run that index search. It's gonna go and find you know? So if I've taken, like, a PDF file, I've chunked it, I've embedded it, and now I you know, the search does a semantic similarity search. I'm gonna find, you know, users asking for a particular piece of text. I can find that chunk of text in the graph doing a similarity search. And then once I found that chunk of text, what I wanna do is I wanna traverse out on the graph and find the related information to that chunk. So that chunk of text is going to have things like related entities. So if the chunk of text, for example, was a critic's review of a movie, that that chunk of text entity could talk about the related actors, the the movie titles, the directors. And so when I return the chunk of text about that movie re rating, I also wanna return all the related entities from that chunk of text. And so how I do that is after that initial search, I do a graph traversal. And what that's going to do is that's gonna execute a Cypher query from each result to follow the connections. And so you feed you basically, what you do is you take these vector searches, you combine them with these Cypher queries, and it's gonna provide a very rich context to return all the related entities, the relationships, the metadata alongside the matched text. So without graph enrichment, I get just a very, like, isolated I get just the, you know, the text entity of, oh, here's the the, you know, movie review, but I don't know what the context of that movie review is. With graph enrichment, I now have all of the structured knowledge from the graph. And this is what really separates out GraphRAG from standard vector RAG. Standard vector RAG is just going to return the chunks, where GraphRAG returns all that graph enrichment. And so how this works with the the Neo four j context provider and the Microsoft agent framework is the agent be has a a hook that says before run. And so the agent says before run, call the context provider. So that call that's gonna call out to the Neo four j context provider. It's gonna filter out the messages to user assistant roles. So I'm only I'm not going to do, like, the the system messages or anything like that. I'm only going to be doing search against the user and assistant messages. I'm going to take the most recent messages, and I'm gonna concatenate those into a search query. And then that search query goes out against the Neo four j index. It could be vector, full text, hybrid. And then I could do the graph enrichment from there, and then I format the results and inject all that context back into the agent. And then the agent sends that to the LLM so that it has all that rich context alongside the user's question. And so you can configure your context provider to have different index types and whether or not it does the graph enhanced enrichment. So I can do a vector. And if I've set a retrieval query, it's going to go out and do a vector Cypher retriever, which means it's gonna do the vector similarity search. And then after it does the vector similarity search, it's going to take out and execute out a Cypher query to return related text. Same with full text. I can either, you know, choose to be, you know, Cypher retrieval or not or hybrid, so all the same things. Key parameters, you know, you you can configure how many messages are used to build the search query. You can prepare prepend text to the the the prompt. I can configure the embedder that's used to create the embedding of the semantic similarities, embedding of the user's question so that it does the semantic similarity search, etcetera. And then I can also configure multiple providers. So if I want to search out against you know, movie plots, you know, as the vector search, and then I also want to do a full text search, I can do movie plots full text, and so I can combine providers and stack them. And then let's see. Yeah. Go into vector search. Let me just see if there's any questions and other so Cypher Query pre or post vector search. So the Cypher Query is post vector search. So I'll actually show an example of that and exactly how that works and kinda go through a detailed example of that. Yeah. What are some different ways to traverse graph for enrichment? How does hybrid search work? For example, reciprocal rank fusion. Yes. So alright. So I will I will cover I have a whole slide deck on how the graph enrichment works, and I'll go through that. Yes. And the hybrid search is that reciprocal rank fusion exactly. So alright. Let me talk a little bit about vector search, and then we'll go into the graph enriched retrieval. So vector search is the idea of you take your text. So if I've taken, like, a blog post or a blog article, and I wanna take a chunk of text and create a numerical representation of that, and then I can use that numerical representation to find similar text to it. So it provides similar meanings to to produce similar vectors. So time travel adventure is going to match up to journey through time because the vectors are close together. Time travel adventure and cooking recipes, those vectors are far apart, So those will not match. So this enables semantic search and finding content by meaning, not by keywords. So when the user asks, find me movies about time travel, what the GraphQL library actually does is you've configured an embedder underneath the covers. And now that has to be the same embedding model that was used to initially, embed the the context in the index. So once I've embedded all of the, say, the the blog post article or or, like, movie reviews, that same embedder, then I configure with the Graphrite library to run embedding on the user's questions. And then the Neo4j vector index is going to do a similar search to compare against the stored plot embeddings. That's gonna return movies ranked by cosine similarity. The results will be injected back in as context before the LLM call call automatically. So one critical thing that and this is a common problem I see is the embeddings mismatch. So the same model that's used to create embeddings in the Neo four j vector search needs to be the same one that's configured when you run the retrieval and the the search of the user's queries, and it supports a lot of different embeddings models. So the basic way you do it is I you just create an embedding, like OpenAI embeddings and, you know, which model to use, and it's going to use the OpenAPI key from the environment. And this will that I can use this embedder to configure the the Graphrag library when it does the retrieval against that vector index to use this embedding for embedding the user's queries. I won't go through this. Configure the provider. The general idea is that the ident I have configured the agent. I sorry. I configured the provider with my vector search index information, and then I'm going to pass that to set that on the agent. So then before every run, it's going to do the vector search. So what will happen is the provider embeds the query, searches the index, injects the results before the alarm call response. So no tool calls are needed. It's all automatic. Right. So as an example, if I had this query, like, find me movies about time travel, what it's going to do is it's going to the the agent framework is gonna say provider dot before run, and so that's gonna automatically call my provider. My provider will then take that user's query. It's going to do a embed that query, do a vector index search, and, you know, find the top five results because I said, just give me the top five. And the results are injected as context messages, And then the LLM generates the responses, granted an action movie data. Because I've, you know, found the action movie data, so instead of, you know the LLM could potentially hallucinate. I'm it might not have you know, most LMs have a limited cutoff knowledge, so it's not gonna have the most recent movies, for example. So if, you know, I wanted to ask questions about, you know, recent movie releases, those kinds of things, all that automatically gets injected in. So without the context provider, the agent's just gonna answer from training data. It might know well known movies, but it doesn't know specific movies from your dataset. You know? So if you have some specific movie reviews, recent movie reviews, ratings, things like that. It could provide in that more recent update information and ground the information. And so this is kind of one of the first examples of truly grounding your agent. So I want to walk through the the lab for this. Alright. I let me I was just gonna glance the questions here. So does the embeddings procedure encompass only the chunk document text, or can it add nodes? Great. So I didn't explain that. Great. Let me go back to hear the the context provider overview. So what the the the the simple knowledge graph pipeline does this sentence is actually out of order. What what the knowledge graph pipeline does is it builds that that knowledge graph pipeline for GraphQL for you. First thing it does is it automates splitting. So that's that's the chunking. And so you feed it like maybe you have an article where a movie critic reviewed a movie that's, you know, like, say, 20 paragraphs long, it's going to chunk it or split it. You know, each paragraph will be a different chunk. It will create the embeddings for those chunks. So this is actually where AdWords should say automate splitting, embedding. And then after it's created those embeddings, it can say take those same news art those same paragraphs, extract out the entities of those, like, movie reviews, and tie them into those embeddings. And so that's the entity extraction. So, you know, if that that article talked about a recent release of a movie, then it would, you know, find the the the, you know, maybe the name of the movie critic, the movie titles, all the entities in that paragraph in as part of the entity extraction and tie that into the embedding. Just a little bit more on the entity extraction. How that works is you're basically taking that chunk of text, you're feeding it to the LLM, and you're just saying you're giving it a prompt to help it kinda tell it what kind of entities you're looking for, and then it returns, like, a structured output of all the related entities. And then this simple knowledge graph pipeline basically takes that information that the LLM has extracted and ties them into those embeddings and to create that knowledge graph of embeddings related to the knowledge graph. Multiple vector indexes for different corpuses of data. Let's see here. Ah, yes. Yes. So if you have different multiple vector indexes, you yes. You would need to have, yeah. Because you you would have to probably have, like, a a supervisor agent. George, you can you can weigh in here if you have any thoughts. So if I want to search, like, multiple different data sources and I want the agent to kinda select different ones based on, you know, which index needs to be searched, I could do, like, a a routing pattern. Mean, could you talk about some of the different, like, workflows you have, George? So you could do, like, a supervisor pattern and have the supervisor kind of decide which agent it goes to based on the the the type of data that it's it's searching for. So something like that. Alright. Sorry, Brian. I I I. didn't. understand the you. on the spot there, George. Apologize. My my correction a little bit is a little bit old today. Just just wondering. The question is, say, like, I have multiple agents that have different context providers for different datasets, and I want to route between the different agents with the different context providers. What would the the, like, the Microsoft agent framework workflow be? Would that be you it has, like, a supervisor agent that could work out between them? Yeah. Yeah. You that's one approach that you could take. You could create the supervisor that we route for different agent, context provider for the same agent. Right? And then the the all context providers could add some information into the context. Yep. Alright. Let me show bring in the vector provider. Okay. No. Close others. Right. So this is this is why it's critical to add the the embedding system. Ryan, one. one one because I think it was a again, a little bit quick where where you how you ended up here. Maybe. could we could we catch up everybody after, you know, code spaces is loaded and Sure. so maybe. that everybody everybody can continue doing the the live part as well with the hands on part. Yeah. Good call out. Thanks. Yeah. So right. So if you're just catching up, in the link to the the lesson content, there's a section on setting up your development environment. We already went through we created the Neo four j sandbox instance, and then I've launched the GitHub code space. So the code space is basically like a cloud development environment that's been launched. And then what I've done is, this is structured actually out of order. You should first verify your setup and then load your embeddings. And so what I've already done is I've launched this code space. So what this is is it looks like Versus code, but it's essentially a it's a GitHub code space in the cloud. It's running a a development environment in the cloud. If I go here and what I did is I, you know, I ran the the setup instructions. Critically, I ran the setup pie and set up all the movie embeddings. And then we went through lab one, lab two. So here under the if you look under here under the labs folder, there's the vector provider. The other part of this to to realize is when you launch that code space, when I created a new code space, I gave it an OpenAI API key. That OpenAI API key has been set here in the environment variable. I have the OpenAI key here. So now when I make the when I run these labs, it's actually going to call out to OpenAI and, you know, run the LM model calls. Right? What we talked about, you load the Neo four j settings. This is getting how you configured how to create the Neo four j. I'm going to create an embedder. I just do get embedding, which uses the default, which is a text embedding three small. I'm going to create the context provider and with all the configuration. And so what I'm saying here, the index name is set to right. The Neo four j settings, vector index name that comes from let me show you where that was. Neo four j vector index name is movie plots. So when we ran this setup, ran it loaded the movie plot embeddings from a CSV file. In a normal, like, you know, like, a production data pipeline, what would happen is these embeddings would be created as the data is being loaded. So I just I'm simulating, you know, having already preprocessed it through an embedding model and loaded it. It created the vector index on the movie plot embedding. I created the the full text index on movie plot, and then I waited for those indexes to come online. So I've created those two indexes. And then in the here in the vector provider, when I create my context provider, I'm using those indexes. So here for vector, I'm gonna say index name is vector index name for my environment variable. It's gonna load this movie plots environment index. The index type is vector. I'm going to configure the embedder, which is this default embedding from before. And this is key that I use the same embedding that was used to create the embeddings for the movie plots. And then I'm going to use a context prompt. I use to find semantically relevant movie plots from the knowledge graph to answer questions about the movie. So I create that context provider. And then in the agent, all I have to do is I say context provider is equal to provider. So that's wiring in this previous provider that I created, and then I'm going to run the agent. Let's let's actually run this. Find me movies or time travel. So it takes a minute to run this. There we go. So this has gone out to the Neo four j. So what it did is it took this, this query, find me movies about time travel. It used the the configured embedding to to embed that for semantic similarity search, and it was able to do the vector search to find movies that are related to time travel. Anything else, Alex, you're done? No. I I just want to say maybe you could if if you could show a quote, maybe if you could zoom in once or twice, I think that is is probably gonna be helpful. But otherwise, all good. I think that was a great recap. Thank you. Just can does does the code show up, or is it pretty smart? Should I increase my font? Yeah. Yeah. This is much better now. Yeah. This is great. I think, yeah, that should be helpful. And then for everybody watching, there's a little Zoom icon top button right of of the of the screen so you can even zoom in some more, but I think that this is helpful, Ryan. Thank you. Yeah. So just just to recap, so if you can see the before, it's just creating this provider. I create the provider and then wire it into the agent as a context provider. So George, would you add anything else about context providers there? Yes. I would like to add. I don't know if it is it is clear for everyone that the difference because we could, for instance, use use a tool to add information to the context. Right? But in that case, the model will decide to to use it in a a a adding knowledge using a tool from context provider. It that is that using context provider, Microsoft agent framework framework will always call the context provider to come and add or extract information from the prompt. So we have the guarantee from Microsoft agent framework that he gonna do that. I know if this is clear for everyone the difference between adding context using a tool or adding context using a context provider. I think let me what are you gonna experience? You know what? I have sorry. Let me find something really quickly here. Oh, here. Yeah. So we've we've been talking about, context providers. So tools the way I think about tools, kind of to add to what George is saying, tools are reactive. The agent must recognize the tool is relevant and choose to call it. So the agent may or may not realize that tool is relevant to the question. And so the difference is is these tools, there's no guarantee they'll be called, whereas with a context provider, because it's basically this before run, you're guaranteed that it's going to run. And so they you're guaranteed to inject that relevant knowledge before every LM call. Right? So they have the hook before run, which is before the LM invocation or after. Right. I won't we went through most of this. Yeah. Yeah. So I'd say the difference is the tools are reactive where context providers run automatically. So that's kind of the the big differences between the two. And there there's our pros and cons to both of them because you could also think about you know, sometimes you wanna have, like, a wider suite of tools, and you don't want the context provider to be called every single time. Like, you might want to have the agent have the freedom to decide which context to inject. So there's you know, it it is a design decision between, you know, tools and context providers. Can a context provider also be used to strictly select a tool? No. They're really separate. So the agent tool calls are are really from oh, that's an interesting question. I guess a context provider I don't know, George. Could the context provider help drive the tool selection? I I don't I don't think so. Well, I would say that it might help drive the selection of the tool, Ryan, because you are basically adding information into the context. Right? So bay so if you add some information to the context, it might somehow make the model decide to not call a tool or call a tool. So it all depends on information that we added, I would say. That's true. Because yeah. Because so maybe that because I've added the context, then it doesn't. It realizes it doesn't need the tool because it's found everything from the context. So No. Yeah. alright. Alright. Let's go through graph enriched retrieval. So what we talked about so far is, just index search, which is just gonna return your text ranked by relevance. And the idea is I wanted to add graph enriched retrieval to add a Cypher traversal step after that index search. So the provider finds the matching movies. So, you know, we've been talking you know, maybe I have a, like, a large set of unstructured text that has, like, movie plots that have been coded and embedded. I find the matching movie, and then from that matching movie, I wanna watch walk the graph to collect genres, cast, and directors for each match. And so the idea is it's a two step process. The first step is the query is matched against the index. Right? You know? And it can be any one of these, vector, full text, or hybrid. It returns the matched nodes ranked by relevance with a similarity score. And then what will happen is you pass in a retrieval query that you write, and that retrieval query is going to go out. And for each matched node or, you know, chunk of text that matches, that provider is going to run your Cypher retrieval query. And that's gonna traverse outward from the matched node to follow the relationships. So, you know, in the movie example, it's gonna collect related genres, actors, and directors. That's going to return a very enriched context, combine that original text with the structured metadata. So you can think of this matched node as an anchor. So I'm going to anchor off of that matched node and then traverse outward from there. And this works with all search modes. So all search modes, you can add in a retrieval query and and traverse out from the initial index search. So the the key to all this is this notion of retrieval query, and this is the Cypher query used to enhance the search. And it what happens is because the context provider is configured when you configure the context provider, you configure it with the index, and you configure it with the retrieval query. So that context provider will automatically do this graph enriched search to match that Cypher query. And that's gonna transform those raw search hits into enriched context before sending them back to the LLM. So how these retrieval queries get written is you use a node and a score. So that that index search returns the the related node, the re they and the score of that node. And then my retrieval query must return at least, like, the text in the score column, so the node dot text in the score. And then I can do optional match for relationships that might not exist if I wanna be sure that I want to, you know, if I want to, you know, search for relationships off that node, but I I want to always return the node regardless of if there's related entities, I do optional match. And then I can do order by score descending to maintain the relevance ranking. And then I can use the Cypher slicing zero five to control context size, and I'll show an example of that. So this is an example of a retrieval query. So what's happening is that Neo four j is after it's on that index search, it's it's handing me the notes. So I'm gonna match on that that node that was found in the index search. I'm going to find all the related genres. And so then with those node scores, I'm going to collect all the genres as a list of genres, and then I'm going to match all the persons that acted in those movies related to those nodes. And then with that node so now I've collected, you know, the genres, the persons. Now with that node, I I take all that information. I'm gonna collect it as a list of actors. And this is the I only wanna do five actors because if a movie, say, had, like, you know, several 100 actors, I only wanna return five of those. Also know that it's an optional match because that node might not have the related entities of the per people that acted in it. And the same thing with the optional match on the directed. And so then I've I've gathered all this rich context and graph enrichment from the the graph. I'm going to then return that as, like, the node dot plot is the text, the score, the title, the genre, the actors, the directors. So I've I've now returned this very enriched graph structure. So in this example, right, starting from each batch to movie node, it's traversing out from genres to find the genre names, five actor names, and the director names. And then it's returning all that plus, like, plot text, similarity score, and movie title. So this goes beyond traditional Rag where you just return the, you know, relevant movie review. This is going to return all the rich entity and knowledge graph information around that text. Why option match? I already talked about this. You know, keep every match that are missing relationships, return empty lists. And, generally, you wanna use that option match for any relationship that might not exist. And then the same as what we've been doing before, it's just a configuration when you configure that context provider passing in that retrieval query, and everything else stays the same. And it's going to use that retrieval query to automatically run the Cypherware retrieval underneath the hood. So let's let's take a look at that workshop next. So, yeah, I'm just going to some questions here. Yeah. For graphic retrieval, who writes the Cypher query? So so, yeah, so you write the Cypher query. Now I you know, obviously, you know, for coding, you know, you can use an LM to help you write the Cypher query upfront, but then you write that Cypher query and configure it as as part of your context provider. So let me show an example of this. Graph enrich search. I'm gonna select my kernel, and I'll just fire this off, and we'll walk through this. Yeah. Again, I've configured my LM provider. I'm gonna create the embedder just like before. The difference is is now I'm going to define a retrieval query. So, yeah, to answer kind of going back to that question, yes, you you need to add that retrieval query yourself because the, you know, the agent and the LM don't understand, you know, you know, how to inject that retrieval query in. So that's that's something that you need to, you know, manually configure. And then the the difference is is, right, with that retrieval query, I now add that to the context provider. So I just have another parameter that adds in a retrieval query. And so now when I do the vector search, it's going to do the vector search. And then from the vector search, it's gonna do the do the additional graph enrich to get that additional rich context. And then it will use that context provider in the agent. And so here in the agent, I just say context provider's provider and wire in that provider. And then so now what are some good science fiction movies and who stars in them? So according to the provided graph rich context so you notice here, if I go back to let me just show you kind of as a comparison. Right. Here, know, I gave the name of the title and a brief description. Here, I'm now returning the name of the movie, the genres, the stars, the directors, plot note from the graph. And so now you can see this re the the results that are returned from this query are much have a have a lot all the entities and are enriched, and so a lot more rich information is being returned. Yeah. So before I jump to the next context provider so, yeah, so I guess let me just say something this. So we're not gonna do the full text search or the hybrid search. There's two more context providers. You guys can go through those on your own in the labs. All this lab material will be available. The only thing that's not gonna be available is an open a AI API key, but everything else is available publicly. So let's see. Without without yeah. What's why without any graphite yeah. Why would it be returned if you didn't do the graphite enrichment? I saw a question. So it's it's exactly this. So if I were to do a plain vector search, on movie plots, right, what we returned is a young man discovers he can travel through time and tries to change the past. So I've done the vector search. I've returned the, you know, the movie plot. The challenge is is this movie plot, like, with traditional rag, right, I found a related movie plot. Now what do I do with it? How do I, you know, how do I tie this back into the information about that movie plot? So when I do the graph enrichment, it takes this. It's matched that node, but that node in the knowledge graph is tied into the entities of that graph. And so it's going to also return, in addition to the text, young man can travel through time, it's gonna return, you know, the title, the genres, the actors, the directors. And so it's going to kind of, you know, really enhance the what the results that are returned. No vector either no vector either, just the query. I I'm not quite sure what that question meant, but how much is being driven by oh, how much is being driven by just the query? Gotcha. So yeah. So it it's it kinda goes back to how much is the query versus how much is the the retrieval query versus the search. So it goes back to there's basically, what's happening under beneath the covers is it's doing two steps underneath the covers. It's doing the index search, and that's all automatic. And then the second step, the graph traversal, is what's being driven by the query. K. So so there's a a question. No index with the name movie plots found. So what you need to do is you need to come in here. Yeah. If if you can't find the index with the name movie plots, you need to if you on your screen, you can toggle the panel to open up the terminal window. So there's a terminal here. And in here, you need to run the this command Python Gen AI math context provider setup. You know, I'll drop this in here just to be sure, make it clear. So I'll drop that in the chat. And so if if you're getting an error where it can't find the movie plots index, you need to run the setup, and that setup will create the vector index and the full text indexes. Cool. Alright. Let's let's get into memory next then. I guess, George, just to kind of to recap a little bit, tell us a little bit about what Microsoft agent framework has for memory kind of built in. Does it what what are some of the built in capabilities of the Microsoft agent framework for memory? Well, it can it's persistent memory using in in memory. I mean, in the you you you know, because some information we don't want to persist. Right? So when you create an agent, you can create a session. That session will persist during during only the execution of the agent. Right? But you can offload that memory to any destination. So Cosmos DB, PostgreSQLite. So there are some connectors available already for that. So you can use them to integrate with these tools if you want. Very nice. So but as I said, Microsoft Azure framework is extremely extensible. Right? So if you want I don't know. I already have this database in my infrastructure, and I want to store the memory in that in a column in that table. So you can come and provide your own implementation of a a protocol, right, in Python, and then Microsoft Azure framework can go and persist and re and recover the memory from that column. Okay. Thanks, George. Yeah. And so then this the the Neo four j agent memory context provider built on that to add that additional rich. So it it goes beyond just trying just the standard memory to kind of a very rich, graph enriched structured memory, and we'll walk through that next. Alright. So agent memory memory types. So what most, like, most agent frameworks store, I'd call, just, like, a very basic memory, which is your short term memory, which is just capturing, like, you know, your history. It might do some summarization. So the idea is we wanna go beyond just, you know, a basic memory to add a very graph enriched memory structure. Right. So the the reason for this is, you know, agents forget. Right? You know, every LLM call is stateless. And so you you know, this is part of the reason why we inject context, and this is the reason we wanna save, you know, past conversations is because these LLM calls are stateless. So the agents themselves need to manage the memory and inject in, like, what was discussed the last session, like, what are your preferences over time, what worked, what failed. And so the idea is that there's three categories of information that we want to accumulate. I wanna accumulate what was said, what were all the messages sent, what were the responses, the follow ups. And I wanna persist all that information. That's a lot of rich and important context I wanna preserve across the agent sessions. I want to capture, like, what was learned, the facts and the preferences extracted from conversation into structured knowledge. So, you know, as you're talking to the agent, you're learning a lot. You're you're digesting a lot of information, and that agent's learning a lot about you and your preferences, and you want all that rich information captured. And then the third critical piece of memory that needs to be captured is what was tried. So what steps did the agent go through, and what tool calls did it make, and what worked and what failed, and capturing all that information. So the agent memory context provider is built on the Neo four agent memory library. Let me and this is an open source library. You know, I should drop it. I'll put these in the chat in just a minute. But Neo four j agent memory is a open source library, and you can use it as a stand alone agent memory library for, you know, any agent framework. What this provides is the the the agent memory context provider wires into the Microsoft agent framework and ties it all together. So, yeah, it wraps the memory into context providers and callable tools. So it has three types of memory. The first is just storing your conversation as linked message notes with embeddings. It's gonna store messages with the role. So when you chat, you were when your agent does chat, it has, you know, different roles, user assistant system, and the content. It stores the vector embeddings for each of those messages, the conversation group by session ID. So I'm just storing all the basic information that's been, you know, flowing in and out of the agent. And this will allow me to do semantic search over past messages. So we talked about vector search and, you know, graph enriched vector search so I can do the same kind of graph rag over my memory. So this is the so you can kinda think of agent memory as building on the graph rag and really kind of being able to leverage that that rich graph rag library to then do rich search across past messages. And this is one of the the key reasons we do, you know, semantic search is so that I can, you know, find related messages and conversations. Right. And then that's where I can maintain conversation context across the sessions. Then there's long term memory. And so what I wanna do is right. There's short term memory, which is just capturing my conversations. I then wanted to take those those chat conversations, and I wanna do almost like I did with GraphRAG. Right? GraphRAG did entity extraction out of unstructured text to to extract out related entities. So I if you know, MoviePlots is to extract out, you know, related movies, titles. Well, what I do with the long term memory is I want to extract out the, what's called POLY plus O data. So, it's kind of what the reason it does this is most conversations have a fairly standard entity model. And so the long term memory uses this standard entity model to extract out people's organizations, locations, and events plus objects. That's the poll e is this I should have put the acronym here, but, yep, people, organization, location, events. I'm gonna extract out, facts, like subject predicate object triples, and then preferences. And how this all works is when you configure the agent memory with long term memory, this extraction happens automatically, and the agent will build this knowledge graph about each user over time. And so what it so by this is by default, and it's, you know, configurable, and you can configure how this all works underneath the covers. But by default, what happens is it it extracts out the short term memory, saves it, and then the back ground async process, it's gonna process that short term memory and enrich the long term memory off of that. So then the third type of memory I wanted to talk about, to give some background on the third type of memory, is just kind of a refresher on agents. You know, I mean, in agent frameworks, in general, all agents have this common pattern of what they do is they they reason about their environment, they act about the environment, and then they, you know, observe those those actions. Reasoning is, you know, you you the user input, the tool results, conversation history. It's the current state. You know? And it could be the context provider that's injected some context. And then act as the agent said, you know, given my current state, I wanna execute an action. I wanna call a tool. I wanna call a query. I wanna call an API, and then it's gonna examine those results. So this is agents are are not just a single, like, you know, call response. Agents have a a rich loop of calling, you know, tools, interactions, deciding all these things. So the third type of memory we use is, what I wanna do is I wanna store these traces, this dot react loop. So I wanna store all these traces of the past agent behavior for learning. I wanna store what it captures, the tech, you know, this description of the outcome, the reasoning steps it took, the tool calls with parameters, the results, the timing information. I wanna capture all this rich, context that the agent did, and I'm what I'm trying to do is I'm trying to capture all the decisions it made. And this is really valuable for a lot of reasons. One is I can do an evaluation. You know, if an agent goes off the rails, it doesn't answer appropriately, it doesn't have the right context, I can go back and I can look at this reasoning to see, okay, what happened? Where did the agent go wrong? I can also use this as learnings to teach the agent how to better respond to questions in the future. And then I can also have the agent use the same reasoning memory to make so it doesn't have to go if it does a complex decision tree, instead of having to to iterate down that context that complex decision tree every time, it can say, oh, I've already reasoned about this. I just wanna do that same thing. So you can learn from what worked and what failed. Sorry. I meant to go to the next page. So the three memory types, if we compare them, there's the short term, which is just your memory and conversation chains. Then we have the long term, which is all the entities and the facts and the preferences, and that ties back into your short term. So, you know, we talked about graph and rate spend. The same kind of concept is you know, all these are connected together so that from the short term, it connects out to the long term, and the long term connects out to the to the reasoning. And so you can look at the entire graph around the conversation and what it learned. So all these are right. They all live in the same Neo four g databases graph structures. It's all stored in the same graph structure. How it works is, right, the conversations are linked to messages. Right? Linked message notes store the short term memory. Those messages are then, you know, tied to entities, which are the extracted or mentioned entities. And then we'll go over entity extraction shortly. And then related entities connect to each other forming this knowledge graph. And then this is and then that's tied into the reasoning trace. And then everything connects back to the session where it occurred. Because so be what's really powerful about this is now that everything is connected, I can traverse the graph and find related information, or I can also do things like, hey. I wanna know where did the agent talk about this specific person, or, you know, where was there a conversation about this particular movie, or what were the conversations about this director? And so I can then, you know, search for entities, and I can go back and I can find new related messages. Or, you know, I can say, well, gee, you know, the the agent is, you know, always making tool callings when someone discusses, you know, a particular movie inception. I wanna understand why is it calling, you know, this particular API every time it calls the movie inception. So I could go to the you know, that entity for inception, I can trace it back to the conversation and then look at the traces. And so, you know, you reverse the graph in a lot of directions, find and answer questions from a lot of different directions. So based on this agent memory, then there's the Microsoft, agent framework integration, context it's a Neo four j agent memory context provider for the agent framework. That's a mouthful. And the idea is, it has several components to to build this context provider. The first is the memory client. So when you first start up the for when you wanna create the context provider, you first create a memory client. That's gonna connect to Neo four j and expose all the memory types. And then what I wanna do is I wanna wrap that in Neo four j Microsoft memory to provide a unified interface that works with the Microsoft agent framework agents. And then I wanna hook that into the context provider so I can implement before and after for automatic memory. And this is key because the context provider is automatically hooked into this after run. What happens is so I've created a context provider, the agent framework. After an agent con agent conversation, it's going to call my context provider, and it's going to say, hey. I just ran this conversation. Here are the results. Here's the entire conversation history. Here's the session ID. Here's all the information, the traces. It's gonna give me all this rich information, and I'm gonna take that as the context provider, and I'm going to store that in my knowledge graph. So it's gonna and then I can also, generate a callable tools for explicit memory operation, which I'll show. So if I wanna see exactly what happened, I can, you know, make explicit calls as well. So the idea is this right. The the way to think about these two different kind of context providers is the first context providers were retrievers. They're retrieving static knowledge out of the graph. So we prebuilt a movie data set, so it's retrieving out that prebuilt data set. The agent memory is dynamic and gross from the conversations. So this is memory versus static. So the static content would be movie data document chunks. This is all the fixed information, whereas the agent memory is going to be your messages, your preferences, and your facts. So both use context providers but with different goals. Alright. Is memory stored as a graph data as well? Yep. I think we answered that one. Let me look at this one. But you want to store the memory on a different graph. Right? How about your graph as a database that you don't want to absurd anything into? Yeah. I mean, I could imagine you have, like, a production graph database. You know, maybe you have a large, like, movie dataset, and and you don't wanna store the agent memory in that movie dataset. I definitely recommend, you know, especially for, like, a production database, you would have, like, a separate Neo four j instance for the for the agent memory in particular. And that's also valuable. I mean, that's valuable for a number of reasons. It it makes, you know, production sandboxing easier. I mean, especially as, you know, your agent memory gets quite large, it's it's easier to kind of traverse, you know, agent murph agent memory versus, you know, your static, you know, like, movie content. So alright. Let me walk through the example here. So we skipped lab five and six. You can go back and do that on your own. I'm setting memory provider. Alright. Similar setup from before. Configure memory settings. So the memory settings are basically the this this the connection into, Neo four j. And I'll walk through entity extraction in a little bit. Entity extraction uses a it's a multistep pipeline for doing the entity extraction. I'll do that in the next lesson. So I've connected Neo four j. I'm going to do some, setup of the extraction. One of the key pieces of the extraction is that memory extraction actually uses an LM, so it's going to asynchronously in the background call out to an LM to ask the LM to do the entity extraction. I then connect my memory client and configure it. I and I can configure. Right? Do I want short term, long term? Do I want reasoning? Do I wanna extract entities? And I can configure this. Right. So then I I configured it. I create the Microsoft memory, which is from the memory client. It's this this wrapper around the Neo four j agent memory. I get the client, and then I create a, right, client as an agent. And so and then I'm setting that context provider. Let me run this. Let me see. See. Start. Run multi terms conversation agent tree query. So I'm gonna run these three queries through it, and you what we'll see is that, normally, if we were to do these queries without a agent memory, each one of these would be sent to the agent in a stateless fashion so the agent would have no memory or recollection of what was previously discussed. So what we'll see is, alright. You I really enjoyed sci fi movies, especially movies about time travel. Grace, since you like sci fi, here's some tailored picks. And then the next question is, what did I say my favorite genre was? You said your favorite genre is sci fi, especially movies about time travel. And so this question was actually asked what what's what's powerful about this is you didn't have to configure your agent to remember the conversation history. You didn't have to try to store and match the conversation history yourself. All this was provided for you by just injecting that agent memory context provider underneath the covers. And then next question, can you recommend something I might like? Again, the agent doesn't have to, you know, figure out how to store the preferences. It hasn't done anything. All it's done is because you've configured that agent memory, it knows that you love sci fi time travel and space. So and then I can inspect. So I I said there's explicit, like, memory tools that you can use. So I can actually call, like and say, alright. What were the messages? What entities did you find? What preferences did you find? It didn't find any entities yet. So alright. Is I'll pause there for just a minute. Let everyone catch up. George, would you add anything else, or is there? You're you're we can't hear you, George. Internet problems always occur in the middle of the most important meetings. So very familiar. Sorry about that, George. It always happens. The most critical times, you always get Internet issues. Cool. So we went through all this. Brian, can you hear, me? okay. Yeah. Sorry? Good. Good. Oh, sure. Oh, thanks. Because I was trying to say something, but I believe I had a issue with my sorry. Well, so what I want to add to this great explanation you provided, Ryan, is regarding the context limit that we have. Right? Because as we keep adding more and more information into the context, at some point, you're gonna hit the the model context limit. And that's why it's really important to select the correct information that we have to prompt and that's why it is really important to have a good index and why graph database are really good in this process. Right? Because you can go and traverse the graph to find the correct information. Right? And the other thing that Quizi is using is what we call computation. So once we once we are very close to the context limit, you can go and ask for the model to compact the information you already have. So it's gonna create a summary of the conversation. So in that case, you can decrease again your context limit. You can add more and more information. I don't know if you would. talk that's that, a good, point. but. That's that's a our I I didn't. have that, so that's actually a really good point. Yeah. And so the old way of doing this is you'd have to match that context manually. And so, you know, remember building agents, you know, so, you know, a year ago where you'd you'd you'd store the conversation history just in, a basic list, and then you just truncate off, you know, like, the the the oldest, you know, messages. And so you'd say, alright. I only wanna keep, you know, the top 20 messages and then truncate the older ones. And you'd hope that you'd had the most relevant information because you have that context on it. So the the agent memory kind of really helps solve that problem of only injecting what's really relevant to the conversation. Because even if you have just ten, twenty conversations, it still could be you know, a lot of those cannot even be relevant. So by using this, you can you you're helping to manage the context. Yeah. Something I've heard a lot about is, you know, context rot. And people often talk about context rot of your context often gets polluted with a lot of information that's not relevant for that agent, and that makes it harder for the LLM to to make, you know, you know, high quality responses. And so if I have a you know, like, you know, even if I could store, you know, like, a really large context of conversation history in the agent, a lot of that's not gonna be relevant. It's gonna confuse the agent. I'm not gonna get as relevant if responses back. So Yeah. And yeah. Exactly. And there is a discussion where you should add information in the beginning, in the middle, in the end of the context. Right? So based on the based on the position where the information is, it might generate a different answer. Yeah. Yeah. Exactly. Yeah. Because the agent, like, if it's a like yeah. It'll sometimes it'll just read, like, the very first thing. Says, oh, I found the information at the very start of the context, and it won't ever, you know, continue reading the rest of that context. It could skip some really important parts that are, like, at the end of the context because it thinks it found the answer. So yeah. Alright. Let me go to the next one. Entity extraction. Let me look at a couple of the questions that come in. Mary did not store any entities preferences. Interesting. You know, everyone has my email. If if people wanna email me I don't know. I I hope I don't regret this, but I I'm happy to answer questions over email. If you have a problem with the labs and you can't get something to work, let me, let me know. Is it safe to say that for graph based agent memory knowledge graph building and retrieval happens in real time? Yes. It happens asynchronously in the background. Well, I I should clarify. Short term is as as, built synchronously as part of the the agent loop, but then the the longer term memory is built over time. So if the context is stored as memory in Neo4j, what is the easiest way to identify that my context is too big? Ah, that's a good question. I don't have a good answer for that one. I mean, I I guess, you know, you can control the retrieval patterns of the memory. You can control it, and you could you could do some you know, we'll talk a little bit more about this, but you can control the the amount of information that's returned and and, you know, kind of decide how much of that, you know, long term memory that's actually being returned and kind of tailor that to be sure that you're not returning too much information. Alright. Let's see. Entity extraction. So we talked about how you have short term memory, and that short term memory is extracted out to long term memory. So let's talk about how that long term memory, entity extraction happens. So the idea is it's very similar to building a a a graph frag pipeline. Right? A graph frag pipeline, you take unstructured text, and I wanna extract information out of unstructured text. I wanna create, you know, embeddings. I wanna do entity extraction, and I wanna build a rich knowledge graph out of that. This is with memory, it's very much the same thing. I wanna store the entities, the preferences, and the facts extracted from those conversations. And I wanna store those in a way that I can then do things like, you know, semantic search, vector search over that those memories to find, you know, what's relevant to the context. There's an extraction pipeline that processes conversation messages to identify entities and store them as Neo4j nodes. There's a multistage approach that combines fast pattern matching with model based extraction and LM fallback. So the idea is that I want to, you know, if if you wanna avoid calling the LM for every, you know, entity extraction, there's other ways and other tools that can do it locally. And then the results are merged to maximize convergence and accuracy. So each stage has different strengths, and they run-in sequence. So the first stage is is spaCy. So if you've if you've I I'm trying to figure out how to explain this. Sorry. There's a whole whole branch of natural language processing called named entity recognition. Named entity recognition is it's the it's actually came, you know, long before even, like, LMs and everything. So this whole, like, NLP, you know, natural language processing tools, allow you to do, entity recognition without using a a full LLM. So spaCy actually just uses statistical models trained on labeled text to recognize, you know, common entities. So we talked about poly, person organization, etcetera. So it's basically just trained on these kind of standard entities, and then it can, you know, quickly extract these because it's just a statistical model that's run locally. Blinder is, like, a very lightweight model. It's a little bit more accurate than spaCy. And so you can think of these as, like, the accuracy goes up, but there's also the call the cost of calling them goes up or as you go down this chain. And then finally, the LLM. Right? And so what I can do is I can say I can configure, you know, I wanna go through the full pipeline. I only wanna use parts of it. I want to say skip the LM if the entities were already found, if I if I don't wanna pay the cost of the the LM calls every time. So the the poly schema, the way this works is these models are all trained on kind of using this same kind of schema, and that makes entity extraction fairly reliable against all of them. And that that way, I can use them in a pipeline. And so the idea is that they extract out persons, objects, location, events, organizations. And they'll extract out similar entities, and then so then I can use this pipeline to compare the entities that are extracted. And then also say if I feel like I have enough that I can stop that entity extraction. And so these entities are mapped back into Neo four j as extracted entities. These all become nodes with labels derived from the poly type and subtype. So I'm gonna add in the labels of, you know, entity, person, individual, like, object, vehicle. So it stores the names, types, subtype, descriptions, etcetera. And then for configuring extraction, I can configure this to memory settings. There's an extraction section, and I can configure which stages get run. So I can configure, like, spaCy, gliner, l m, fallback, and each can be toggled independently. And so I can decide, you know, which ones I wanna use, and then a confidence threshold. Each of these return a confidence threshold of how confidence they were in that entity extraction, and then I can limit the entity types to specific types. So I can do a, extractor type pipeline for this multistage. If I wanna just skip the multistage, I can go straight to the LLM. Extractor type pipeline. You know, like, do I wanna enable spaCy, gliner, LM, fallback, etcetera? So gliner is a it's a lightweight model that's hosted on hugging spaces. It's a for this lab, I've disabled Gliner. You need, like, a Hugging Face token to download it and run it. It's also, like, a 500 meg day 500 megabyte download, so it's a pretty large model to download. So it's been disabled for this particular lab. And then I can configure a threshold of when this entity extraction happens, how confident I wanna be to in those entity extractions to, you know, lower. I'm gonna get more entities, but I'm also getting a lot more noise. Merge strategies. So the other thing is is it goes through this pipeline. Each of these steps could extract out the same entities, and I want to figure out how I just to, you know, merge these entities together. So this the strategy of, you know, confidence, the highest confidence scores is really based on just on the confidence scoring. I could do union to create all the unique entities from all stages. So, basically, saying, you know, any entities that are found, I wanna keep them all and just put them all in a big bucket and union them together. Intersection is saying, basically, I want what that would you think of that as the the space of the glider first gonna do anti extraction, and then the LLM will validate it. And only things that all three of them, you know, have agreed on will actually end up in my as named entities in my knowledge graph. And the re so what how this happens is the same entity may may be mentioned differently. So it could be Christopher Nolan or Nolan or the director interception. So auto merge would merge those automatically. I could, you know, set it to be flagged for review, be behold threshold, just treat as distinct entities, We're gonna use a resolution strategy combining exact match matching, fuzzy string matching, semantic similarity. So the automatic extraction runs up runs in an after run after every agent response. So this is async by default. So the idea is I have short term memory that's automatically stored and and captured as part of the agent process. But then asynchronously, it's gonna kick off this pipeline to process that that short term memory and turn it into long term memory. Or I can change it from automatic and and doing, you know, async by default to doing the manual and prepopulate the graph with known entities and then use that to see domain knowledge before the agent starts. Oh, I should also mention. So you can do the entity extraction asynchronously so it doesn't block the main agent call loop, or I can do it synchronously if I if I feel like, you know, I wanna be sure that that entity extraction happens before the next, you know, agent call. And because, you know, for example, I might wanna use that long term memory, or I can do, like, offline batch mode. But I wanna say, you know what? It's so much volume of data that I'm going to have to process. I wanna, you know, offline and, you know, maybe process it somewhere else. So as an example of of a full pipeline, if I had, you know, spaCy, gliner, and LLM configured, and it sees the message, love Christopher Nolan's inception, spaCy would say, okay. Oh, I see Christopher Nolan as a person for that poly model. Gliner confirms it and says, Christopher Nolan, person is I have a confidence o nine five. But then Gliner also says, oh, I also found inception object confidence o nine two. And I configure the the pipeline to, skip entities that are already found, save time and cost. So it's just gonna go, and it's gonna look at that sentence for anything that wasn't found. And then it's going to merge using the confidence strategy to keep just the highest confidence results and then store that all back out to the knowledge graph. So Let's go through an entity extraction example. So computer extraction settings. Right? It has three stages. Right? Gliner is disabled for the workshop because it's too it's too large of a download. I can you know, so extract extraction pipeline, you know, enable spaCy gliner, LM fallback, LM temperature. I connect the memory client. I add a manual entity. So these are ones that I want to, you know, kind of add these entities in manually, you know, that are outside that automatic pipeline. Then I can test, like, deduplication, merge strategies, kind of this is everything we've talked about. So, like, here are the merge settings. I'm gonna say, merge strategy confidence. So it's just gonna keep the the the memories the entities that's extracted out of the the long term short term memory that it has the highest confidence in. Merge strategy confidence resolution settings. So this is giving me you know, for deduplication, what are my different thresholds? If it's an exact match, a one, right, identical names are always matched. Fuzzy threshold, similar names above this threshold are merged. And then semantic threshold semantics, similar entities above this threshold are merged. So I'm kind of configuring how, you know, how much confidence I want in the entity extraction, and this is going to kind of it's it's a balance between, like, noise versus, you know, accuracy and, you know, being able to get as many entities as possible. Alright. Alright. Any George, anything else, or is that No. I don't. alright. We'll go through the this last lab of combining context providers with memory tools. We'll kinda talk about this and wrap kinda end up on this one. So and then there's gonna be you know, there's two other labs that you can run on your own. Memory tools. So let's go through some of the memory tools that are available. So, right, George talked about this, difference between tools versus context providers. Context providers are called automatically on on those hooks. So, you know, before run or after run. And so you know those those context providers are always called to those exact hooks. Sometimes you want the agent to do a more deliberate action. And so what you wanna say so as an example, the user says, remember that I hate horror movies and expects an explicit storage. So agent needs to search for a specific entity before answering, and then the agent wants to look at what past reasoning traces for a similar task. So passive injection alone can't handle these, like, targeted operations. So I wanna basically give the agent a way of tools that it can use to you know, beyond just the the standard extraction to handle these kind of additional cases. So the Neo4j agent agent memory library, also provides tools. So in addition to the context provider, these tools are functions the agent can call on demand to actively manage its memory. So the context provider runs automatically on every tool sorry, on every turn. And the tools run only when the agent decides to use them. So this means that the agent has two different ways of interacting with the memory. One is this context provider automatic background memory, and the other is memory tools that are targeted operations the agent controls. So, Greg, we already talked about what tools are. Right? They're basically, a callable function exposed to the LLM. So the way they work is the agent you you define the tools, and then the agent decides when those tools get called. And there's a set of memory tools. Let me go sorry. Go back out. So these provide a way for so what you can do is you can define these tools, and you can tell the agent, hey. These are all the available tools to work with the memory. So depending on the user's query, the context, other information, call these tools as necessary. So the user might say, you know, hey. When did we talk about, you know, the movie inception from Christopher Nolan? And then the agent can have that tool to go and search its memory. You can also do things like remember preference. So I can explicitly tell the agent, hey. When the user you know, look at the user conversation. When the user talks about some of its preferences, I want you to explicitly go out and remember those user preferences and store those. And then they can use the same tool of, like before it makes, you know, a response, it can say, oh, I need to, you know, figure out what the user's preferences are and go out and retrieve those preferences before answering this question. So this allows the agent to have a lot more power and flexibility beyond just the normal context providers. You right. And the way this is done is you just call create memory tools to generate the six tool instances and then pass them into the agent's tools alongside the context provider. And then the agent instructions guide it of when to use these tools. So when a user expresses a preference, say it with a remember preference, etcetera. And these tools operate on that same memory as the the context provider, the the same knowledge graph underneath the cover. Oh, why do I keep bringing out of those? Sorry. Yeah. Yeah. So context provider versus memory tools. I mean, this is kind of the discussion we had before about context providers versus tools. Context context providers are automatic, whereas memory tools are the agent gets to decide. So I'm I'm just gonna skip the rest of the slide and go to the the lab and kinda show this really quickly. So right. The same as before I set up the memory settings. I create the memory in the tools. This is what what's different is tools create memory tools. So I create the the agent memory the same way as in, you know, write this Microsoft memory from memory client, and that same memory client can then be used to create the memory tools, and this allows it connect in the same graph database and query the same Neo four g graph database. And then I'm giving it instructions so it knows how to use these tools. So you're you're a movie recommendation assistant with persistent memory. Important. You must call the remember preferences tool every time a user states a preference, a favorite, or interest. And because I'm using a smaller model, I had to do a stronger prompt to get this to kind of really force the agent to to use these tools. So cool. I I Alex, I want to kinda just leave the last five minutes to see if you have any other feedback. Or, George, anyone has anything to say? So Yeah. Cool. Thank you very much. I don't know. George, do you do you have anything? No, Alex. No. Cool. No. Super. Thank you very much, Ryan and George, for the the intense session. I I hope everybody was was taking something out of it. Some some of the code, obviously, might have been a bit quick, but that's why you will get all the materials afterwards to to do it in your own pace, and you can check it out, the labs, and the code space by yourself. So I think that's that's should be helpful together with the the slides and the material that that Ryan and and George went through today. So I think that's that's definitely something we can we can cover again. Let me quickly launch a quick survey. So we are like you to collect, obviously, a little bit of feedback. And if, Ryan, if you could stop sharing your screen, then I can quickly put up the yeah. Thank you very much. My my follow-up deck sort of. So, you know, if you are wanting to to watch more and see more, we are doing regular livestreams on YouTube and on LinkedIn. So, for example, tomorrow, I will talk about the Neo four j definitive guide book that was published by O'Reilly and by Luan and Christoph, and they will be joining me tomorrow afternoon, Central European time, 5PM on YouTube to join and to talk about this. Obviously, NOSE is coming up. So as you all already are registered for for it, I'm I'm I'm hoping to see you all on the April 15 to to discuss and to hear more from from, you know, developers all over all over the world about knowledge graphs, graph memory, and agents with with Neo four j. We have three more workshops as part of road to node. So if you're interested in in more courses, then definitely check check it out and join us then. If you want to, you know, meet people in person and have actual face to face conversations, we have a series running that's called Graph Summits. And Munich is next next week, Paris in April, San Francisco, New York, London later in the year. So if that is something you're interested in, look for Neo four j Graph Summits. And I think Irvina will also relentlessly answer questions in the back end, so thank you very much to her. Also shared a couple of links in chat as well, and one of those is definitely Graph Academy. So if you're interested to deepen your knowledge, especially around AI with Neo four j, look at the Graph Academy courses that are available there and follow the paths. Think, Ryan, you mentioned this. If you have any questions, anything that was unclear for today, you can reach out to Ryan, but you can also reach out to our community forums where we have all the team and the wider community join us regularly and help to answer questions and help you, you know, be successful with graphs. So that's another way of of getting getting somewhere. And, I think with that, two minutes to spare. I think we are we are we are done for today. I really would like to say thank you again to Ryan and George specifically for for running the workshop, building building out the content and and and, you know, guiding us through today and Irina for help in the back end. And, obviously, to all of you, thank you for joining. Thank you for your question. Thank you for participating and being active throughout the two hours. And, yeah, I hope to see you at another livestream, another event, or the next from to notes workshop. Until then, have a good time everybody, and see you soon. Thank you.