Name: Designing Agent-Driven GraphRAG Systems with GNNs for Healthcare with Giuseppe Futia, CSI Piemonte
Uploaded: 2026-03-17T19:16:06.593Z
Duration: 1 h 51 min 16 s
Description: Designing Agent-Driven GraphRAG Systems with GNNs for Healthcare with Giuseppe Futia, CSI Piemonte

Transcript for "Designing Agent-Driven GraphRAG Systems with GNNs for Healthcare with Giuseppe Futia, CSI Piemonte": Hello, everybody, and welcome to our next road to notes workshop today. It's called designing agent driven graph for existence with GNNs for health care. I'm super excited to to have this session. So we have already had a couple of of very interesting workshops in the past with road to nodes. I'm super super happy to continue the story. And today, we you know, Giuseppe Futia from CSI Pimonte will share with us how to model data as a a as a graph and how to work with GraphRect architecture in in this in the space of healthcare and life science. We obviously can imagine everything is very regulated, everything is very sensitive, it's personal data, it's your health, your history data that can always be a little bit tricky. You wanna make sure that everything is safe and and sound and everything obviously makes sense. But then also the the the results should obviously make sense. Nobody wants to hear that they have a severe sickness because an AI tool made a made a great mistake. So it's all it's all very tricky, all very very but very fascinating at the same time. So I'm I'm I'm super happy to have Giuseppe join us today at this Road to Nodes AI workshop and lead the lead the workshop today. Before we we dive in and before I I let let Josepi take it on, I have just a couple of things. The workshop today is recorded, so you don't have to worry if you miss something or if something is unclear. You can always come back and watch the recording. I will send an email with the video to the recording afterwards, so you should get that within a couple of days after the the session is over today. In the meantime, however, if you have any questions during this workshop, we are very much encouraging you to share your questions with us, and you have the possibility to do so through the q and a panel. So in the right hand side, you should have a chat window where you see my message about the recording of the video. You should also see three buttons on the top, and one says q and a. Click on q and a and type in your question, and then we can see your questions, and we can follow along. My colleague, Jennifer, is also with us in this in this workshop today, and she will also come in and, you know, answer some questions. But we I talked to Giuseppe. We have some time during at breaks or at some some points, and Giuseppe will answer a couple of questions during during this. So it's it's interactive. Keep it interactive. Let us know what you like, and let us know what you want to know more about, and then we can we can dive in and and tailor the workshop a little bit towards towards this depending on on your needs and on your your question where you are going for. And, yeah, with that, I think that is what I have. Obviously, all the materials, slides, and there's a notebook as well that you can follow along. You can watch and and see how Giuseppe is doing it, or you can try to follow along as as you like. But no matter how you prefer to do it, there will be a link to a repo afterwards, and you can look at that and see what's what's been written there and how the code works if you wanna know more about it and wanna replicate it or maybe you know, it takes us as a baseline and build something with with graphs and in the health care system. Or maybe you own a completely different system. This is this is independent of things. And, yes, maybe a good point. GNN is graph neural networks. So, yeah, we we I I wanted to say that, but I then forgot when I read the title. Yeah. But with that, think, Giuseppe, you need to click on go on stage, which is the the green button at the bottom right of your of your screen. Here. you are. Hi, Josepe. Hi, Alexander. How are you doing? Fine. Fine. Thank you. Thank you very much for this great introduction. Yeah. Thank you. Thank you for joining us today. Thank you for spending the time with us and and for this amazing suggestion submission to the to the notes AI or to notes, you know, workshop series. So I'm I'm super happy to have you join. And, yeah, I'll I'll let you take the lead, and I'll be I'll be around, obviously, throughout the session. But the the stage is yours, Giuseppe. Cool. Thank you very much, Alessand Alexander. Thank you for asking me. This is a great opportunity to share with you and all the audience something that I'm currently explored in the CSI Pimonte, which is not the crime scene investigation, but is actually stands for consortium for information systems, which is basically a public company that serves a lot of public administrations in Italy, in terms of services, infrastructure, and so on. And, we work on different topics including the health care field. And today, we will try to see something that probably is a bit more different compared to the traditional graph rack systems that we have seen before because we will try to incorporate, as Alexander mentioned, a graph neural networks in the in the end to end in the end to end pipeline. So let's start with the presentation. I will try to switch between slides and notebooks in order to see some concrete results using code. Right? And during this presentation, here, you see the GitHub link for the code. But in this presentation, we will try to to start in we'll try to understand which are the typical problems, the typical challenges that characterize the medical domain. After that, we will see some architectural and technical strategies to, in some way, tackle and face these specific challenges. Then we will go through an exploration phase because as you probably know, graph neural networks are not so studied and deployed within these specific systems. There are some interesting research lines. But during this workshop, I will try to leverage graph neural network model within the problems that we want to solve. Right? So you will not see the traditional way of using deep learning models in which we have training data, inference data, and some evaluation. We will try to incorporate these specific components within within our pipeline. And finally, we will try to understand the impact of graph neural networks in building graph reg agents. Right? So we will try to understand how the behavior changes, how the graph or a gadget behavior changes when we incorporate the graph neural networks. So let's start with the introductory part related to the medical domain. The medical domain is particularly interesting because compared to other fields and domains, this is usually characterized by diverse and disconnected data sources. Right? Obviously, these data sources are released in different formats. Sometimes you have unstructured content, like public reports or publications and so on. Sometimes you have a a very structured and semantic information, including ontologies and so on. But there is also another difference. Right? And it is related to the policy that defines how we can use this data. Right? Because, for instance, in the context of public ontologies, probably we can, we use and incorporate in our knowledge in our knowledge base just following some, specific licenses. Right? But there are some other types of data that are more sensible, that are more critical. For instance, the clinical data. Right? So information coming from clinicians, from the patients, and so on. And in my direct experience, I've seen that, obviously, we have to this this data has to preserve. Right? And they have to be self contained in the legacy systems. But sometimes there are also situation in which you cannot move this data around around a specific system. So you have to keep the data, for instance, from the legacy in the legacy database. So in the traditional approach in which we try to materialize all the information within the within the graph database, sometimes there are context in which, the regulation prevent this opportunity. Right? So we will see also how we can leverage some of the Neo four j features that allow us to access this critical data in a virtualized manner. So this is this has been already introduced in the previous slides. But as I mentioned before, when we work with clinical information, we have to create privacy centric architectures. Right? So we have we need a technological stack that preserve the privacy of the of this data. And for this reason, for when we cannot move the data around, we cannot replicate data maybe within the same system, but we have to keep the data in the legacy database. We we have to introduce some data virtualization layer. Right? So we can access the data just on the fly when we need. And, obviously, when we use LLNs, we cannot use the traditional ones that are provided as a services. Like, I think, for instance, the OpenAI or Anthropic cloud services. Right? We have to build our server our LLM server on premise. And during this presentation, we will try to see how to do that. Right? We will just we will have just a quick overlook on it, but the key idea is that when we work with clinical data, when we work in the health care domain, we have to preserve this data. On the one hand, just keeping this information in the legacy database. And on the other hand, we have to leverage on premise LLMs. So, basically, LLMs installed on our own server. So this is the the framework, right, the the general situation in which we have to we have to work with with such data. And so we will try to identify the technological components that it can some way set that that can satisfy the specific requirements. Right? So let's imagine this very basic architecture, right, in which have information coming from different sources. We already mentioned biomedical ontologies. There are some unstructured data like publication and scientific reports, and there is clinical data. Right? And, basically, when we work in a traditional manner, we decide how we have to integrate such data and publish and materialize such data into the knowledge graph, in our case, into the Neo four j database. Obviously, this integration part can be support using LLMs. For instance, from unstructured content, we can use LLM to extract entities and relationships that are mentioned in these in these documents. But, you know, based on our requirements, we should have some on premise LLMs. Right? So the LLMs should be installed on our server. And in this demo, I will show you something done with Vu LLM, which is I don't know if some of you already know it. Maybe there are some other inference engines like Colama that are maybe more famous, but VLM is particularly effective because they adopt some specific strategies to make the inference phase very, very efficient. In particular, you can it is very powerful in processing information in batch. So, you know, for in the case in which we have to process thousands or millions of documents, this inference engine is very powerful for processing documents in parallel, for instance. So this is one way to satisfy the requirements of using a lens, right, in installed on on premise systems. And on the other end, there is this virtualization layer. So the idea is not publishing the clinical data directly in the Neo four j database, but we would like to leverage some features of Neo four j to virtualize the access to the clinical data. So the the clinical data stays where they are, stays, for instance, within our relational database, or as we will see in our example, we will just use a CSV file to to to show to to perform the showcase of this specific feature. Okay. So this is the the high level architecture. Right? And another point that, actually, it's really important. Anytime in any in any case in which we want to perform some information extraction from documents or we want to reach clinical data, obviously, the biomedical ontologies drive the process. Right? We want to detect, for instance, diseases or symptoms from the clinical data that are in some way mapped or already mentioned by the biomedical ontologies. Because if the ontology drives the extraction process, then we can integrate on the fly clinical data ontologies in order to, I mean, enable advanced reasoning based on the knowledge incorporated within within the ontology. So that's that's the idea. Right? That's the architecture that we should take into consideration. This architecture should should satisfy I mean, at least some of the requirements that are related to the medical domain. And regarding the information extraction process, we have also defined some general framework that could be useful for different tasks, right, for integrating multiple ontologies, for extracting information from clinical data, and so on. And this is the framework, right, for performing these specific tasks. Think about, for instance, the idea of ontology integration and ontology mapping. The biomedical domain actually is really rich in ontologies. Each of these ontologies actually is very specific. Right? We have ontologies that collect information about genes. You have ontology that collect information about diseases and so on. And so one of the key aspect of this work is try to connect to these multiple ontologies because one entity within an ontology, and we will see an example in a while, can in a while, can be directly connected to another entity from another ontology. And when we are able to bridge these information sources, we can leverage a more integrated knowledge that can be useful for reason. And in this pipeline, actually, the entity recognition part is not always required because as you can see in the ontologies, actually, you already have recognized entities, so you do not have recognized entities somewhere, for instance, in the text. And then there is a candidate selection process in which from the source ontology, you can have multiple, related targets, right, that are located in the in the target ontologies. And so bit, among these candidates, maybe the the correct one is just a single one. It's just a target. So the idea is that, okay. I have my recognized entities. I select a bunch of candidates, and I try to identify which is the target candidate among the among all these possibilities. Right? And in order to drive this process, we will use some information background. This information background in the ontologies is already available because maybe there is an already existing hierarchy information that can be used to support the disambiguation part. While in other situation, we do not have this specific and very well defined context. So, I mean, we have also to understand which is the background information that can we use to perform these these specific tasks. I would like to show you some demo, right, before moving to the next part. I talked about ontologies. Right? I will not provide a definition of ontology today. There are smart people that can do do this job better than me. But I would like to show you some example of a medical ontologies and how can be how how they can be connected to each other. Right? So one example is the unified medical language system, which is actually Amita Tezarus. So the key idea is that these specific knowledge base basically link to multiple and different ontologies. So it's a sort of collection of ontologies, and it basically defines the the overlap between these ontologies. Right? So in this case, the chest pain that you see here, it's already stored and defined in multiple ontologies and vocabulary. For instance, there's Nomad, which is one of the most important ontologies to define clinical data and clinical processes. And there are other interesting sources that we will see and, we will leverage for our demo, which are the ICD 10. Right? ICD 10, basically, here, you have an information here. It's basically a vocabulary that describe multiple diseases and contextualize these diseases in in a hierarchy. Right? So in the UMLS here, you have this code, ER0SetDot4, which is actually the same code that identifies this entity in the in the ICD 10 domain. And as you can see here in the light in the left side of the page, this chest pain basically is located in a specific hierarchy. Right? Because you have, in this case, other symptoms and signs involving the circulatory and respiratory systems. You have a higher definition of of this class and so on. Right? So this specific entity and disease is in some way located in a hierarchical structure represented by the ICD 10 ontology. Right? And the other thing is, another potential ontology is, that actually defines the same entity is the HPO. That stands for the human phenotype ontology here. So when we, for instance, copy this code in the human phenotype ontology website here, you have, you basically can access the chest pain entity, in this case, represented within the HP ontology. Right? And this ontology actually provides a hierarchical structure that is similar to ICD 10. But as you can see, right, from, in the in the main part of this page, there are also the disease that are connected to this specific symptom or to this specific phenotype. In this case, the information is richer than the previous ontology because we just do not have the hierarchical structure, but you have an interesting association between chest pain and the related diseases. Right? So if you have Lassa fever, probably one of the key symptom is related is the chest pain. So this ontology provides more more information, and we will try also to leverage this specific structure in order to drive the the behavior of our of our graph based agent. So let's move to the code. Right? And because I've seen that some of you mentioned, for instance, the MPNN, so the message passing neural network. Yeah. We will try to we will see how, right, we can use these specific architectures, which is the probably the most well known architecture of graph neural networks to improve some of the mechanisms that I will show you in a while. So this is the demo notebook that we will work, that we will use today to to perform some to show, right, some of the the things that we will that I have introduced in the using the slides. Basically, during this demo, we will you we will leverage different servers. Right? If we talk about on premise servers, so, basically, I have deployed the LLMs using Colab because I know that it's not on premise, but, basically, the code can be replicated also on your server. So the key idea is that we will have some services, including an embedding API, that use the Quantu model. We also, leverage, a small model developed by Google that is called MedJama. Right? That is the medical version of the well known JEM model. This is all obviously, also open source. Obviously, we will leverage Neo four j, particular a local instance in in my case. And then we will have also some graph neural network servers. Right? I deployed two different servers that have different behaviors, and I will clarify in a while how we can use this specific Genome servers. Giuseppe, can we zoom in a little bit when when we show code? I think that is helpful for everybody. And if if you then think still think it's not zoomed enough, there is a zoom icon bottom right. If you're watching this, you can zoom in some more. But if you could zoom in maybe once or twice, I think that would be helpful to set the so. we can Yeah. Thank you. K. Great. So as I mentioned before, we we we are basically deployed as a specific server that are currently running. So here, you have a collection of notebooks that, as you can see, leverage the the VLOLEM inference engine, right, to be to be delivered, to be provided. Right? And we can see. Right? We can make some some tests in order to see what happens during on this service when we run our our code. So for instance, now I'm testing the embedding server. Right? So here, you see that we have logged, right, the the request to the server. I can do it again just for testing here. And so you can see that there is another request to this server that is currently running on Colab. Right? So we can leverage these servers that are providing some specific utilities for our for our demo. And in this case, why the embedding server is useful? Because we want to have some sort of baseline, right, to translate the the textual representation of, for instance, of diseases or symptoms into a numerical representation that can be leveraged in some way, for instance, for comparison, for candidate selection that we have mentioned before for the disambiguation of entity. So these are this is the baseline. Right? So we have the QAN model that provides embeddings. So in this case, this was the the input terms. So shortness of breath, dyspnea, and tachypnea. And, basically, we compare using the similarity these these embeddings. And we can see, for instance, that probably shortness of breath and dyspnea are synonyms. Right? While shortness of breath and dyspnea are just related. Right? Because the number, this casino similarity value is lower. So probably in this case, we have some similar information. In this case, we have just some types of of relations. But we can actually directly ask to the LLMs, right, what is the difference, the clinical difference between shortness of breath and dyspnea to understand if the LLM model itself, considering this representation in the vector space, already knows this difference, and it's able to verbalize this difference. So, again, here, as I mentioned before, we have the MedJemma server running. The last run was sixteen forty six. And then if I run our MedJama server here, it provides none for a ride. So you see that, in theory, the the timing should be updated. Okay. Here, now it's logged here, and here is the c set. The the answer. Who are you? I am a large language model, created by the Gemma team at Google DeepMind because my Gemma has been provided by the same team of of of Gemma. And what is the clinical difference? And it says the clinical difference between dyspnea and tachypnea is that dyspnea is the subjective sensation of difficulty breathing while tachypnea is the objective measure measurement of an increased respiratory rate. Right? So if dyspnea is the symptom, tachypnea is sign. So, basically, it's something that is measurable. So they are not the same representation as we have seen in the in the embedding. Right? But they are in some way related. Okay. Why I use the QUAN to seven b embedding? Well, actually, I would like to use for my testing's general based model. Right? And, basically, the QAN two seven b model show actually interesting results among all the general ones on the medical domain. Right? So each it reached very good results, basically, in the question answering systems involving information related to to the medical field. So this is why I decided. But, obviously, using VuelllM, you can incorporate potential any embedding model that the VuelllM actually considers. Right? Because they basically develop specific components and the specific kernels, right, for in order to speed up the inference model. So you can basically deploy most of the models. Right? But in this case, we you should check, right, on on their website. I did not use the Gemini embedding because, you know, one of the requirements was using some on premise and on premise services. So you usually use, if I'm not wrong, Germany as a as a service, right, as a cloud service, so you cannot deploy the Germany embedding model on your server. But the idea is that, I mean, you can we can basically we need, as I mentioned before, some deployed deployed model. And here, you have, obviously, the instance of of Neo four j. In this case, as you can probably recognize some of the entities that I have already mentioned before, We have nodes to represent UMLS UMLS entities that are useful to bridge information from different ontologies. You have nodes represented representing the phenotypes. You have nodes representing the ICD 10 diseases and so on. I will just quickly introduce the graph neural network models, right, but just to build an initial intuition, and then we will go through through the details. Okay. So let me run the first GNN model. Right? And the first model applies an interesting behavior because it takes the Quen embedding, and it tries to apply a residual update, so a minimal change that could be useful to improve the quality of these embeddings. Because as I mentioned before, information related to disease or phenotypes is integrated in a structure. Right? And this structure could be hierarchical, so we would like to use the GNN to incorporate the structural information into the embeddings to see if the quality increases. Right? So the GNN rank model, I will provide you more details later. Right? But GNN ranking model basically apply these small changes to the initial credit betting whose quality is already good. And this is useful, right, for the reranking step. Right? Because, you know, when you perform the semantic search, you basically you can obtain a collection of candidates in order to if identifying the corresponding entity from the ICD 10 to HBO. Right? In this case, you can see that for I this ICD entity, it is called cervical GEA. Basically, the GNN re ranking process actually increase the importance of neck pain, right, which is actually the the true entity, the true corresponding entity, while for the quan entity, the cervical spondylosis was the corresponding one. But, actually, there is not an identity relationship between this one and this other one, because, the entity selected by Quan maybe is more specific. Right? While the GNN put this entity in the second place. Right? So here, you see the GNN ranking. And the the first entity, which is basically the most corresponding one, was put at the fifth place by Kuan, but it was reranked to the first place using this GNN reranking mechanism. We will go in the details in a while, but I would like just to describe the behavior of this specific service. And then, we have another GNN server, which is basically is used not for the re ranking but for the disambiguation part. And this is an interesting approach. Right? That is the paper of this that describes this tool. One of the is also Yan Lecun. Right? And this is interesting because we will see more in the details later, but it tries to train in a joint way some of the parameters of the LLM and some of the information coming from the graph neural network in order to improve the disambiguation. Right? And here, you see an example of a result because the periodic briefing is the source entity, and these are the target entities identifying by a pure LLM and by the Jira receiver model. And we will see that this is actually the right answer for this mapping. Okay. I've seen that there is a lot of questions. Some of them I will answer during my presentation. I tried to answer one of them. So how much RAM to run the embedding and DLLM local? Yeah. That's an interesting question. I can say that, currently, I'm using the high 100 GPU on collabs. So, basically, they are not viable for the using the free tire. You have to have a you have to pay, right, for using Collab Pro. But this is the GPU used. And the interesting thing is that the LLM inference engine is able to readapt, right, the the space in the GPU RAM in order to improve the performance. So you will notice that when you use the Vuel LM engine inference engine, you will know you will notice that the GPU RAM will be always saturated because it tries to saturate the GPU RAM in order to exploit and to speed up as much as possible the inference the inference time. Right? So if you deploy a four b model or if you deploy a 20 b model, you will always see that the the GPU will be saturated. Obviously, there are some specific parameters that you can use to decide how much RAM you want to use, but this is the default behavior. So I can show you, for instance, here. I don't know if I can see the resources here. No. Let me try to to the QAN here. Okay. The QAN model is actually smaller, and this is basically in a waiting in a waiting status in a quiz status so the it is not saturated. But just to give you an idea, the QAN two seven b model is currently consuming 14 gigabytes of the GPU RAM. So this is just to give you some some numbers. I will continue, right, on the code because, you know, we have under probably understood the behavior, right, of of different servers and why it can be useful. Right? We had the server that is able to translate the the strings, the disease of the strings into an embedding representation. We have an LLM that can reason using these these medical data. We can have the GNNs that can, in some way, support some of the specific intermediate tasks that we are currently have to have to face. Right? But let's look at the the real knowledge graph that we we already built using the the building process. And I think that looking at the knowledge graph, it's useful not just for understanding the data, but also for understanding how we can introduce the the virtualization mechanism. Right? Because at the beginning of the presentation, we mentioned this problem of working with legacy databases, but, you know, we want to leverage the medical ontology knowledge for reason on this clinical data without directly materialized the patient information within within the graph. So let's see how we can do it. First of all, let's start to explore the ontologies. As I've shown you before, the HPO ontology basically maps specific symptoms, specific anomalies to the related diseases. So if we take a very high level anomaly, like, abnormality of the endocrine system, you can see that you have this collection of diseases. And between this anomaly and this disease is disease, there is a direct connection. But, you know, here, we have just probably less than 10 candidates, less than 10 corresponding diseases, but probably the abnormality of the endocrine systems includes a huge amount of different diseases. Right? But as we already have seen in the when we when we show the ontology, there is a hierarchical representation of this data. So some of the other diseases are directly connected not to abnormality of the endocrine system, but they are directly connected to to its children. Right? So let's make a very simple and very straightforward Cypher query in which basically traverse the graph going from the high level entity to more specific entity. And our high level entity is, again, abnormality of the endocrine systems. And if we run this query, basically, we will leverage we are getting different diseases, right, that were not mentioned before. But these specific diseases are connected to some of the children of abnormality of the endocrine systems. Right? So in this case, we have type two diabetes mellitus, which is a child of this more general entity, And this specific disease is connected to this specific phenotype. Right? So these are also, there are other examples that show this specific behavior. So not all the diseases are connected to this high level disease. Some of them are connected to its children, but, basically, we can reconstruct all the the diseases, that are connected to this higher level entity by exploring its children. Right? And this is a mechanism that we will try to leverage for for our graph based our graph based agent. So let's move on. We are I'm concluding this this part related to the to the knowledge graph overview. Obviously, we did not just incorporate the the ontologies by themselves, but, we also use the the UMLS bridge, right, in order to identify the connections between these ontologies. So for instance, here, you can see a path, right, in which the first node you represented by an entity coming from the ICD 10 ontology, while the end of this path is connected to an entity that comes from the human phenotype ontology. This is an interesting exercise, right, because, this mapping exists from the UMLS, but the semantics of this mapping is not well defined, right, in the sense that there are some cases in which, for instance, we have a direct connection between a symptom, an anomaly, and the related disease. Right? In other cases, you have an identity relationship. So, basically, in this case, the source node and the target node represent the same entities described in the two different ontologies. So we basically have the UMLS bridge and call the different types of semantics that this is something that could be valuable to know, right, when we want to leverage this information in some way. And now this is the materialized graph. Right? So this is the graph that is currently stored into Neo four j because we extracted information from these ontologies, and we published such information after some transformation in the in the in the in the Neo four j graph database. K. Sorry. So let's move to the virtualized access layer. I would like to show you this, example. Right? So let's try to to type this query. Right? The match I mean, it's already written here, but let's do it again. Try to find the encounter node and return n. Right? So as you can see, the encounter node is not viable in the in the graph. Right? And, probably, you derive this information also from the collection of of classes here. But the idea is that we want to, in some way, incorporate in a in a in a virtualized manner the encounters that involve, for instance, a specific patient. This information is not materialized in the graph, but through Neo four j, using the virtualization layer, we can access this data. Right? So no encounter node information. But if we run this query here, you can see that, basically, this query provides some encounter nodes. Right? So these encounter nodes are created on the fly using the APOC library. Actually, the APOC library, basically, you you find this specific feature in the extended version of the AIBOC library. You so you have to install it as a plug in. But the key idea is that my information is currently in a CSV file. Right? This CSV file is characterized by different columns here. And with this specific procedure, we can materialize on the fly the information of the patient into the Neo four g a graph database. Right? So the encounter, as you have seen here, was not present. We we had no data directly materializing the graph, but I can basically retrieve them from the CSV file using the virtualization layer. And you know that these nodes are virtualized also because the ID, right, it's a negative ID. So, basically, Neo four j associated this negative ID to the virtualized nodes. So this is interesting for our infrastructure because we have the ontologies materialized within the graph, but then we also have we didn't move the clinical data into the graph because they stay in the legacy database, which in our case is just represented by a a CSV file. But in this way, we satisfies we satisfy the second requirements of not moving the clinical data and the critical information around, but just keep them on the original on the original source. So this is basically just, I mean, another way to see the same things. So, basically, to get the patient data, which is characterized by this Adi, directly using this specific APOC APOC procedure. Now, I would like to, yeah, finalize the graph overview. Just, try to leverage the framework that we have discussed before. So as I mentioned you before, we discussed this specific framework that can be used for the ontology mapping from the information extraction from clinical data and so on. So we will see an application of this framework using basically two of our server, the embedding server, and the LLM server for the disambiguation. And later in the presentation, we will see how we can incorporate and introduce the graph neural network servers to improve this mechanism. So let's move on, and let's continue on the code. So as you can see here, we can leverage the embedding model to identify the potential collection of candidates right here starting from this input string. And these are the collection of candidates, with their similarity score coming from the human phenotype ontology. Right? So I have this input string, and I try to identify the candidates. And for the ontology, right, this is I could put here basically the for basically a string coming from the ontology. But for the disambiguation part, so to understand which is the right one, probably the right one is chest pain because it, achieves the higher the highest score in the in the semantic search. But, let's have a look at the disambiguation. Again, this is could be, the starting entity in the ICD ontology. As I told you, you can leverage background information. So considering that these entities is stored in a hierarchical structure, you can basically incorporate in the source context some information about groups, some information about the chapter. And as you can see here, this information actually represent an entity which is higher in the in the taxonomy and in which in some way classified as chest pain on briefing. Right? Here, you have potentially a list of candidates, so a list of HP entities that could be useful to map. You have chest pain. You have chest pain on exertion, and you have a description of it. You have periodic chest pain. So when we run the LLM model to disambiguate among these candidates, basically, it says that the starting point chest pain on briefing, the most corresponding one is the pleuritic chest pain. Right? So not the general chest pain that we have here among all the candidates. And, basically, it provides some try to provide some explanation. Right? And it says, for instance, that the chest pain on breathing is a key feature of pleuritic chest pain. Right? So the chest pain could be connected to maybe multiple diseases. Right? But when you have the chest pain on breathing, maybe among all these possible candidates, the pleuritic one, so not not the general entity, is the right one according to the LLM. Right? So as you can see here, we can leverage potentially these high level structured information using as a context in order to improve the disambiguation that in this way is directly performed by by. And as I mentioned before, there is a deterministic mapping that leverage the UMLS entities as a bridge between ICD 10 and the human phenotype ontology. But then we can, for instance, construct a new relationship, right, between these entities, and this relationship comes from this candidate selection and disambiguation process. Right? So the idea is that we create this specific relationship with the results coming from the embedding model and the LLM model for the disambiguation. So, basically, in this way, we can enrich, right, our knowledge representation because, potentially, we can create new relationships that didn't exist before, didn't exist by just leveraging the UMLS mapping. But in this case, we have a mechanism to enrich our graph. And in this specific relationship, we can store the confidence and the rationale that directly come right, from the answer from the LLM. So in this way, we implemented a strategy in order to improve and extend our initial mechanism for ontology mapping by combining, right, the LLMs with the with the search embedding. So this is what we we currently have in the graph. Right? So this is what we are able to do. And now I will go back, to the presentation because I would like to introduce how the what is the role of graph neural networks in this game. Right? So how we can leverage this information that come from the structural relationship within within the graph. I've seen that most of the question are related to this topic, so I will try to answer during during the presentation. Mhmm. Okay. Let's move on. So let's move on the on the third part of this of this presentation. We will see actually, maybe graph neural networks are not well, it's not well known for all the members from from the audience, right, because maybe it's a component that is not usually not very well explored when we talk about graph and so on. So I will try to provide a bit of introduction. This introduction will involve some math, but I promise just enough math, right, to convey the idea. Right? So you will not be alone. Right? I will try to be with you in understanding how graph neural network works and how we specialize graph neural networks for our specific for our specific purpose. So someone at the beginning mentioned the MPNN architecture, someone from the audience. MPNN stands for message passing neural network, and the key component is the message passing layer. So, basically, this is architecture that we will leverage in order to transfer information from one node to the other node. As I promised, there is a bit of math. Right? But we will try to use this graphical representation in order to understand what happens in each step of the message passing layer. So the first of all, let's understand what is the goal usually of a graph neural network. The main goal of a graph neural network is to incorporate to embed the relational structure that surrounds a specific node. Right? So our main goal is that, this blue feature actually in some way incorporates the information coming from the surrounding nodes. Right? So the goal is that at the end of the training process, these nodes have some knowledge about its relational structure. So this is the goal usually of a graph neural network. So we usually start with an initial representation. So, potentially, we have some features related to the nodes. Maybe these features could be the QAN embeddings, right, that we use to translate the label of a node into an embedded representation. This is just an example. Then we have features that basically are, in some way, characterize the neighborhood nodes. Right? So the neighbor nodes. So as you can see here, the light orange nodes are the neighbors of the green one. So we have these different features. And the first step is create a message. So we would like to identify a way to transfer information from the light orange node to the green to the green node. And this is part of the equation describes how we can be flexible in creating these messages. Sometime we would like just to get information starting from the node features. So maybe the quandum bedding features of my neighborhood. Sometimes maybe the edges have some information that I would like to incorporate in the node features. So for instance, I can concatenate the node features and the node and the edge features to compose the message. So this is the part of the equation that describes the flexibility of this mechanism. Here, we have a bad words. Right? Obviously, this message should be in some way learned. So this is why these messages should be parameterized through a multilayer perception, which is basically a fully connected neural network. To understand this idea of parametrized, right, you have to and you probably know that in the case of text, for instance, the embeddings that are connected to that specific token are not fixed, right, especially in the transformer architecture. But the embedding representation depends on the surrounding words. Right? And the attention mechanism allows you to understand what is the surrounding word that counts more in order to identify the embedding of our target word. In the graph, we have a similar behavior, right, in the sense that when we learn the representation of a graph, we can actually perform different strategies in different way. So for instance, if we want to learn just the graph structure, we will try to use an unsupervised approach. So with this unsupervised approach, the neighbor the node features will be learned in order to, in some way, represent the graph structure. So the embedding will go to a specific directions during the learning process and will be parameterized in a specific way. But if we want to build, for instance, a link prediction engine, the direction in which the in which these embeddings will be learned and parameterized will will be different. Because I don't want to just understand and learn the general graph structure, but I would like to use this graph structure for the link prediction task. So in this case, the embedding that will be learned, right, will go into different directions, right, because the task that we want to solve is different. And the way in order to enable this flexibility is parameterize the message using a multilayer perception. That's the key idea. As you can see here, then we would like to identify an aggregate function. Right? So we would like to combine this information coming from the neighbors. And the smartest guys usually use this idea of permutation invariance, right, which is actually a bad word to say. If you have a sum and you have a, basically, a set of numbers, if you switch the numbers, the sum doesn't change. Right? So this is the concept of invariance. Why we call the permutation invariance? Well, think about let's imagine that we are in a not in a virtual room, but in a real room. Right? We are sit close to each other. Maybe some of us know each other. Right? So, basically, these nose connections represent the graph structure. So let's imagine to do this exercise. We are sitting, right, and then we stand up and switch our places. Right? So I go to one place to another seat. The colleague that I know go to another seat. So, basically, we switch our position. But if we consider the relational structure, so how we know each other, it doesn't change. Right? So the idea of permutation invariance is that if you have a graph that is characterized by by some nodes, and if you try to switch these nodes, the final relational structure, so the way in which the graph in the one network see the graph doesn't change. Right? So this is why graph neural networks learn the relational structure from the graph. Because I can have graph characterizing people. I can have graph characterizing what baby g's and so on, but the nature of the relationship doesn't change, for instance, if we permute the nodes or if we relabel the nodes in some way. So there is ideal permutation invariance is that I want to keep this information. I want to preserve this relational structure when I perform some transformation using a graph neural network. That's that's the idea of a permutation invariant function. And finally, I want to update. Right? I can decide how to update. I can maybe concatenate the vectors. I can just perform the mean. And, again, this is, again, a permutation invariant function. And this is not always the case. Right? Because I don't know if you have experience with some of the graph neural network architecture, like GraphSAGE, for instance, that probably is one of the most known architecture developed by Uralescovich at Stanford and so on. Basically, they are a gray gate and updating function was a language model. So something that takes a structure, right, the relational structure and put this structure into a sequence. And when we perform this transformation from a structure to a sequence, we lose the idea of permutation invariance. Because in a sequence, when we switch the words now, in this case, the output is different. Right? While in a graph, when we switch the nodes, the relational structure remains the same. So this is important to maintain this idea of permutation invariance function when we try to leverage a graph neural network. So finally, probably, we reach our goal because with the final message, we aggregate these messages, and we update the representation. And all this transformation was described by the equation below. But what happens, for instance, when we stack multiple layers in a graph neural network? Well, there is something interesting that happens because when we stack multiple layers, basically, we retrieve information from one of more distant nodes. Right? So from nodes that are far from my neighbors. So when we stack, for instance, two layers in a graph neural network, the final representation that you have here in the target node actually describes the orange nodes, but also the green ones. So, basically, I retrieve information from, basically, nodes that are not directly connected to me, but they are connected to my neighborhood. So this is what happens when we stack multiple graph neural networks. So here, this is the idea. Right? At the end of the day, after the propagation and the training, this node incorporates in some way these specific structures. And then, I mean, there is usually another layer that allow us to, I mean, incorporate, basically, the inter representation of the graph into a specific vector. It's less important, but just to convey the idea. And if you look at the code, right, that is implemented in Python geometric, Any architecture that you have seen, for instance, graph convolutional networks, graph attention networks, graph Sage, and so on, all these model are implemented using a message passing layer. What changes is the implementation within this function, but, basically, the structure is always the same. So I hope that at the end of this just enough math description, you have a better idea of how a graph neural network works. So let's try to understand how we basically, incorporate the behavior of graph neural networks into our pipeline. Right? We already described we already introduced a little bit of the behavior, but let's see more in-depth how we use the graph neural networks. That's a simplified pipeline that describe the process in which we leverage graph neural networks for adjusting the candidate selection process. As we already seen, right, the key idea is we incorporate in the nodes the embedding using the our general purpose model embedding. When introduced the graph neural networks, we basically move, actually, the embeddings using the this principle between the nodes. Right? So we basically use the message passing neural network framework to transfer the embeddings to each other. Right? So the nodes start to convey their own embeddings through the other nodes using the neighborhood structure. But we don't use actually a full and traditional defined message passing neural network. Actually, we try to reduce the impact of the graph neural network just to refine the embedding representation. So as you can see here, we pass the information from an order to the other, but, basically, we apply a weight, which is basically an an alpha value that probably we have seen previously in the code here. Let me go back at the beginning here when we talk about the graph neural network ranking. You have this alpha value that says, okay. Transfer information from the from each other, but just reweighed this information because the query embedding worked quite good. Right? So the baseline is quite good. I would like just to just to refine. And that's the idea. That's one way in which we can use the graph the graph near networks. Right? Just to provide you some numbers. Basically, when we train the model and basically evaluate our model on the validation data source, and I remind you that the task that we want to face is the ontology mapping. In the three sixty eight test ICD codes, basically, the quen model fails 34 times. So it means that the corresponding target entity is not among the first five candidates. Right? But when we introduce this GNN reranking mechanism, almost half, right, of the of these missed entities is rescued. Right? So the half of these failed cases is in some way in some way addressed and taken by the GNN ranking mechanism. And I remind you that the idea is I perform this message passing neural network to transfer the information, and I try to apply a bit refinement, right, of this of this aggregation process. And here, you have an example in which a node case, right, so in which the CRAM model fails and this is graph neural network rescues this this failing. So here, you have basically the the embedding model. Right? We get the top 20 candidates. But as you can see here, among the five candidates, basically, neck pain was not included. Because using the the rescue mechanism of the graph neural network, this neck pain basically against 18 positions. Right? So it start to become one of the top five possible selection for for the disambiguation. There are some cases in which this behavior, the behavior between QAN and GraphQL network remains the same. So in this in this case, both they both agree on the starting from this in the so there is just a single character difference. But for other cases, right, in which cervical GI, basically, there is some lexical overlap between this target entity and this one. So maybe this is why Quan selected, this specific as the most closest, entity. But in that case, you have a very different lexical. So in this case, the context, the surrounding information could be more useful, right, than the lexical than the lexical information. And, obviously, this is this has an impact. Right? Because if we use for the disambiguation, just the first five candidates, right, the results can completely change because here you have, for instance, the result of the disambiguation entity. Right? So in this case, the LLM, which actually leverage the mechanism the mechanism that I've seen we have seen before, basically, source entity and a collection of candidate's entity. So in this case, says, okay. This is the target entity. Right? Here, you see the answer from the LLM because the neck pain was not including in the first top four or top five candidates. But when we run the LLM disambiguation on a different, basically, collection of candidates in which we put, again, cervical spondylosis, which in the previous step, right, was basically the final selection of LLM, then he says, okay. Actually, neck pain is the right the right target. So when we put neck pain on top of the candidates, the LLM system is able to better disambiguate. Right? Because this entity was present also in the previous selection of candidates, but now it's not selected because this neck pain actually is the most corresponding entity. So here, you can understand probably the impact. Right? You can understand the impact of graph neural network ranking because we can put maybe better entities in the candidates. Right? And so the LLM disambiguation, we just use this flat textual information is able to select the best one. Right? Because the best one was re ranked in the initial top four or top five candidates. So this is one of the possible impact of incorporating graph neural networks in this specific pipeline. Okay. For answering some question, basically, the first question that I see here after running the GNN, the and the learn embedding store separately. For now, no. Well, in the sense that for now, I stored in the Neo four j nodes just the Quan embeddings, and this computation is performed on the fly. So the GNN server provide me the updated embeddings and provide me the re rank results. But, potentially, these embeddings can be stored, right, as a new properties of the nodes. So in this case, you have a permanent permanent layer for this information. It can be definitely be done. Right? But in this case, in these examples, they are computed on the fly. Right? And regarding the other question that I see here, GNN enhanced ontology mapping, training the GNN with link prediction as the goal. Yes. That's that's the answer in the sense that, you know, in this case, the link prediction that we want to perform is based on the UMLS bridge. Right? So what we have seen before is that, in our Neo four j database, we already have these constructed and deterministic, bridges enabled by the UMLS. So, basically, the QN, the GNN model learn how to map the entities and to adjust the embeddings using as an example the UMLS. Right? But what I would like to show you here, it's a fairness fur fairness note, right, in the sense that, basically, when I perform the inference, no UMLS nodes are present in the inference graph. So, basically, it just leverage the structural information that it learned during the training, but it's not using for the inference information coming from the neighborhood established by the UMLS relationships. So to finally answer your question, yeah, the tar the task is the link prediction, and the link that we want to predict is the UMLS, basically, bridge that comes from the the ontologies. Okay. In this target identification process in research, how much accurate info we are? I mean, I tried to answer this question, but this is just a showcase, right, to understand how the GNN can be incorporated. Obviously, for, let me say, a full research activity, we should, in some way, perform more more experiments. But this start to be interesting numbers. Right? So let me go back here. So we are able to rescue the 50% of the failed cases just using the quantum embedding. So maybe this could be an interesting direction. Right? I would like just showcase the impact of graph neural networks in this in this specific pipeline. Okay. Let's move on, right, because we have seen basically, we have seen some of the of potential u application, right, of graph neural networks to improve our our mechanisms. We have seen here again that LLM disambiguation now picks the correct candidates. Also, in the cases in which the previous candidate that was disambiguated in the previous scenario was was viable among the candidates. And this is just an example to show the structural context. Right? So, basically, you have a difference, right, in the ontology structure for these two potential candidates. Right? Every every as we described before, neck pain is detected as the first one from the GNN and the nineteenth for the QEM, while the cervical was QEM was the the rank one. And here, you can see you can have, in some way, a demonstration of what the graph neural network started. Right? Because in the case of neck pain, right, in in our implementation, we had two layers of graph neural networks. So, basically, into the neck pain, we incorporate the information from these entities and from this entity, While in the case of cervical spondylosis, we have probably got information from these two different nodes. Right? And here, you can see that, basically, the message built from from this note is is very specific. Right? It's really specific, and probably this information specialized more this specific the the the representation of this note while this part is less specific. So, basically, maybe higher structural information in this case could be leveraged by the graph neural network and could be in some ways useful for the reranking reranking part. Right? So this is the idea. Right? This is how we use the graph neural network to refine the candidate selection model for for describing for identifying the right target in the ontology mapping. And then we have another application of graph neural network that is called the gRetriever. Right? So let's move back to the presentation. Actually, here, you have the paper of JiraTriever original ones. Actually, I made some, obviously, some updates to the architecture to readapt what has been written here from from these guys. You can see that, for instance, you have Ian LeCruen among the authors, Javier Bresson, which is one of the most important research in the graph neural network community. So the idea was to readapt the mechanisms implemented in the geo retriever model for optimizing the disambiguation part of of this ontology on this ontology disambiguation task. So let's move on on the presentation, then I will show you some some code. Right? Here, you have as you remember, this was the ontology. This was our framework. Right? Right? For for ontology mapping, for information extraction. So what we have done in our framework is incorporating the candidate selection this residual GNN base re ranking. So let's move on, and let's try to understand how the gRetriever works at the higher level. The so I will let we will try to understand what is the what are the key features of this Jira receiver architecture, which is more complex than the re ranking model. Because as you can see here, the graph neural network the graph neural network component is just a specific element, right, of this of this architecture. So what is the idea? Okay. K. Here's the idea. You we have our ontology graph. Right? We perform an initial when we want to map an entity to the other entity, we perform an initial retrieval using the Quan embeddings. And the Geo retriever perform this subtract construction. Right? This PCST stands for a specific algorithms that is called price collecting standard optimization in which the the the key idea was to starting from the candidates coming from the credit retrieval, I try to develop a subgraph. Right? Basically, I developed subgraph in which I got all these candidate entities. I try to get the edges that connect these entities. I assign the inner prize to the nodes and the prize to the edges. The idea is that I assign to the nodes of this subgraph positive price, and then I assign to the edges a negative price. So the goal of these algorithms is to identify which is the potential subgraph that actually is able to obtain the highest price. Right? So in this way, this substructure is used in two different directions. First of all, this substructure is performed we perform the graph the message passing layer transformation. So we will have here, basically, our presentation of the nodes using the graph the graph neural network. And then considering that these nodes actually are basically characterized by some labels, then we use an LLM to encode these textual representations into an embedding space. So at the end of the day, we will have a new representation, some coming from the textual token that have been defined by the LLM model and a collection of soft token that coming from the GNN model. So the key idea is to try to fine tune this model using a very well known approach called LoRa in which the idea is to align the representation from the LLM the representation from the graph neural network, basically, to ex or to to leverage and to exploit this post process post train information in order to perform the disambiguation. So the key idea is to adjust the embedding coming from the graph and the embedding coming from the text that actually derive from different vector space in a way that is useful to perform the task that we have in mind that is the disambiguation part. I know that it's a really bit complex, but the key idea is to align the embedding spaces coming from the text and the embedding spaces coming from this specific subgraph in order to in order to, basically, improve the disambiguation part. So that's the idea. Right? So here, we have something that is useful for the candidate selection, and here, you have something that is useful for the disambiguation part. And that's the final representation of architecture. If you remember, we have some on premise server that in which the LLM was deployed using the Vue LLM inference engine, but we also have the graph neural networks that should be installed right on our on our service. Now let's understand the behavior in the code of this graph retriever mechanism. You have the paper. So for basically, you have the paper for getting more details. Here, you have a very high level explanations. And here, you have an aggregation of the results. Right? So, basically, using the JiraTriever, with the subgraph that we have described before using LoRa for the fine tuning allow us to improve the disambiguation compared to, basically, the similarity. Right? So we are able to gain more information for for the disambiguation part. And we can see here some some results, right, of using this Jira receiver for the embeddings for the disambiguation. So we task. So the first entities was periodic briefing, and you can see that the language model just select intermittent hyperventilation, but this is the real target. Right? And this real target does not come from from the semantic search using GRAN, but from the subgraph using that real algorithm that described before. Because as you can see here, this entity appears here, and it is able to gain good prices, right, that could be useful for the Jira receiver model for the disambiguation part. So that's the idea. Right? We can leverage this subgraph retrieved for from the ontology graph. We transform this subgraph in something that is numerical numerically useful, and we use this this solution to improve name entity disambiguation. So this is maybe most of actually, we did most of the job, right, because we created the conditions, right, to use in a flexible way all these models for the ontology mapping. Right? But in our presentation, we also discussed information extraction from clinical data. Remember, we have clinical data on our virtualized layer, and we cannot reach this clinical data because sometimes, especially in the narratives, the clinician doesn't put, I mean, for instance, ICD codes or HPO codes. Right? So we can help the clinician in annotating these narratives. And that's the idea. That's we how we can perform this task. Basically, here, you have an input information from the patient. You have a collection of, basically, categories. Right? And these categories are not written by me, but basically are derived from the high CD hierarchical structure. So we have to understand which entities the notes describe. So here is the concept. Here are the basically, the strings in the narratives. Right? And as you can see from the result of the named entity recognition, basically, for each of these strings in the narrative, you have obviously the text. You have an associated label that comes from these lists. Right? And this list could be built just, for instance, using Cypher query on top of our of our ontologies. Right? Now I'm putting them here just to make the the example clearer. So in this case, abdominal pain is, in some way, labeled using this label, and you can add more information, right, for this information extraction. So this the assertion and the temporality. This is interesting because if everything worked well in the case of chest, right, no chest pain that is labeled using disease of circularity systems, and the assertion is negated. Right? This is interesting because in the when we perform named entity recognition in the medical domain, we do not just want to see what are the entities mentioned by the clinician, but we also try to incorporate the negative ones. Right? While traditional named entity recognition, just saying, okay. This is a person. This is organization. And, obviously, if they appear in the text, basically, they are present. But in the case of the medical domain, something could not be present. Something could be negated, and an advanced named entity recognition leveraging an LLNs could perform again, could I mean, annotate also this this specific information. And then we can perform the disambiguation. Right? We have a collection of candidates. And in this case, basically, we perform the disambiguation entity. So if this is the input text, right, you have a disambiguated entities that appear quite different. Crohn's diseases of small intestines without complication, but here, you have the region. So the reason is the mentioned describe inflammation of the terminal ileum, which aligns with the code. Sorry. Which aligns with the subscription here? I just sorry. Okay. Let's try to move on. Here, you have the rationale because, basically, the Crohn disease here presents some specific lesions. Right? And so this is why the Crohn disease is selected for for the disambiguation. I would like to show you another example of how we can create on the fly a connection between the virtualized nodes and the ontology that we have in the graph. Let me run this specific query in Neo four j. Here, we have a specific mapping between the virtualized node representing the encounter and the materialized node basically represented by the ontologies. So here you have a relationship that is built on the fly, right, between the virtualized nodes and the materialized nodes. So, basically, here is the way in which we can enrich information on the fly. We can decide to store, basically, this information maybe in our legacy database. But if we cannot touch the data, we can create disconnections on the fly because here, you have a materialized node, which basically has a specific positive, extra additional ID. And here you have nodes that have negative values because they are virtualized. So you can do it on the fly. And we will see how we can leverage this information basically for improve the reasoning of a graph based agent that basically can help us in for differential recognition of a specific disease. So let's move to the graph frag agent. And In this agent, we have three tools. Let me run them. Here, you see the difference. Right? When we run our agent, we are able to identify the phenotypes connected to AVN influence. Sorry. When we just call the Medjam model, here you have the answer, the verbalized answer. Right? So here we have a mechanism to detect the exactly the exact phenotypes using the ontology. And here you have an answer that is not driven by by the ontology. I want to show you this last example before I will lose the voice completely. So here we have we try to ask the LLM based on the symptoms of the patient, what are the the most probable diseases. Right? And this is the coverage. This is the coverage computed by the agent, which leverage the ontologies and try to provide some numbers. These are the numbers. Maybe the patient presents 16, symptoms on 24 for the Lyme disease. In this case, the developmental delay appears 18 times, so more than Lyme diseases. But the coverage is quite low because here you have a sixty six percent of the coverage. Here, you have less than ten percent. So an experiment that we can do is not using the ontology, but ask DLLM which is the most probable diseases among these consider, the symptoms. So in this case, I'm not leveraging, basically, the the ontology here. I'm leveraging just the internal knowledge of the LLM. So the LLM here says that these are the best candidates, while the other one is not considered among the most probable. So this information, actually obviously, the LLM does not know the coverage computed using the ontology. It just see a collection of phenotypes and see the most probable diseases. So, basically, it says this is not probable. And in fact, the coverage from the ontologies was very low. So what is the key difference of using medical LLM for question answering and for using a graph based agent? The difference is that we can compute some numbers. For instance, in this case, the coverage that directly comes from the ontology. Right? So in this way, we are ready to build a deterministic, I mean, semi deterministic way for differential analysis. While in the case of PurellM, we cannot perform this specific task. Well, there are other examples that, provide other details, but I I think that I provide you a lot of a huge amount of information, maybe too much, hopefully not. But you can try to build all this stuff by yourself using the code that is available in the in the repository. Maybe I have to share the training models, not just the server for the inference part, but I will go to to share I will share also these these resources. And let me close my presentation just saying that if you are interested in this this topic, we recently published a book for Manning called knowledge graphs and the LLNs in action. It is available, I mean, anywhere right now. But some of the topics, right, that are touched and discussed in this specific, presentation are already mentioned, in in the book. So if you are interested in to go depth to expand your knowledge about how to use LLMs with, graphs, your knowledge about ontologies, your knowledge about graph neural networks. Basically, we try to assemble this book in order to touch the graph world from different perspectives. Right? And probably in this presentation, we have seen different perspectives about the graphs. We have seen the graph database. We have seen ontologies. We have seen representation learning models, including graph neural networks. We have seen how we can interact how we can create the interactions and connection between graphs and the LLNs, hopefully, in a way that is maybe a bit different compared to other examples that you can find somewhere else. Right? So thank you very much for for your attention, and I hope that it could be useful. I've seen then some of the question actually are related to very specific things that should be discussed more, like, can the improvement in your retrieval accuracy justify the external computational? That's a good point. Right? In your analysis, we have to consider if it was useful. For me, it was fair enough. Right? Because it helped me to improve the disambiguation part in a way that could be really useful to me. So it depends obviously on the on the use case. Yeah. That's a good point. I mean, we somebody else commented on this of what was the also, this is beyond health care. This could be also applicable to finance. It's it's yeah. Like I said, Yeah. it depends a lot if you are what your use case looks in the end and if it's if it's worth it. For some instances, I would say for finance, probably for health care, probably also, it's worth to do the extra cycle. You don't want it. You just want to avoid any any wrong, you know, outcomes at the end. For some use case, it's probably fine. You know, you don't have. to be a 100% accurate. You can you can allow yourself some errors. It's fine. But if it's if it's very sensitive, very highly regulated, or highly looked at data, then probably it makes sense to to go the extra mile this. Yep. Absolutely. I completely agree with you, Alexander, and it depends on the use case, and it depends on the structure of your of your graph information. Right? Ontologies are really useful because people, human beings, try to incorporate this knowledge in a very structured way. So, basically, it's really useful for training gen and models for disambiguating using the LLNs because they already reached, right, in their initial representations. Are some cases in which you do not have this opportunity, so this is another aspect that you have to consider before leveraging these these systems. Yeah. Yeah. And I I don't I don't want you to lose your your your voice altogether. So, well, thank you very much, Isabelle. I don't know if you if you have if it in you for for one more question or, generally, maybe we we can discuss one more thing. From Colin and a couple others just came in and talk about updating or, you know, when a new paper comes out linking neck pain to hot hot air ballooning. I don't know. How do you update your graph and graph neural networks? So I guess, I mean, it's it's this is a great foundation always, and and you can always think of this, especially when working with ontologies and and the graph. It's a great it it makes sure your graph, your entity resolution recognition is is is aligned. And then you can when when new data comes in, it can it it doesn't have to come up with with, you know, ways of of attaching itself to some random node or or something. You you know where to where you wanna connect it. You want you know what the structure looks like, and you wanna you wanna set that up in that way. Yep. How how is how is this then I guess you would run another another calculation of of the values from for the for the for the g and n part, or how how would you do it in in that in that regard? Yeah. I mean, it could be useful, right, to try to transform the representation, the numerical the initial numerical representation also for for this specific context. You know, as we discussed before, it depends. It's always a trade off. Right? Usually, this could be useful if you have the resources and the opportunities to train your your model, right, for, for these specific tasks. Obviously, this is not the this is something that could be explored more. Right? Yeah. Yeah. Cool. No. Definitely. Have a look at this and and and see. Well, I I think with that, we'll we'll we'll close it out. So thank you very much, Giseppe. If you could stop sharing the screen because I have a a little bit. a couple of slides I wanted to I wanted to put up. Perfect. Thank you very much. So if you're interested in in more, then I'd like to invite you to the next stream. Oh, this is the wrong slide. The April 1. Sorry. This is wrong. But next next session is April 1. Obviously, March 12 already has been passed. So twelfth of April 1, we'll do another session on Aura Agents to explore the Nodes AI agenda. Nodes AI is taking place on April 15, so you should have registered for it. We have two more road to nodes workshops. We do next week, one on MCP, and then the week after, we do one on agent memory. So two more very interesting, very hot topics. So if you're interested in that, join us in the next two or two notes workshops or come by nodes.ai where we talk about all these things around AI and and graphs. And and you hear from people from around world for what they're building, what they're what their experiences is are, and how they how they drive, you know, Neo four j graphs and and all the the context, the knowledge, the memory, the agents, of that together. If you'd like to meet people in person, we have a couple of graph summits coming along. One was just today in Munich. We have Paris coming up in April and then San Francisco, New York, London later in the year. So if that's a good good way of meeting people face to face from the wider craft community, but also they also have a workshop so you get your knowledge and get some hands on experience as well in the evenings or afternoons rather. And, yeah, if you are now interested, the Graph Academy is a great place to learn. There are some graph data science course, some lots of AI courses as well. So you can go to our Graph Academy and learn there. Or if you have more questions around graph neural networks and and GraphRack, I would like to invite you to our community pages where you can ask questions, where you can find like minded people, and you can discuss things like like the topics we just mentioned. And, yeah, it's a great place to exchange ideas and and then get some some thoughts out. And, yeah, the that's a great place. And, yeah, with that, I think we are at the end of session. We almost lost your voice, Giuseppe, but hopefully not not altogether. Not entirely. thank you very. Okay. that's entirely. Yeah. Good. Good. I mean, you you you deserve a break. It was really amazing. Thank you very much, Giuseppe. It was a super, super good good workflow today. It was a little bit of math, but not too much. And like Jennifer put in the comments, I think that you explained a very difficult topic very, very well. So I I I really thank you. a lot you so much. for for this today. Thank you all for watching and participating. Thank you for your questions. Thank you, Jennifer, for for answering questions in the back, and I hope you had a a good session. I hope we see you soon at another Road to Notes AI workshop or at Notes AI or somewhere else around the world. And if that's yeah. Have a good rest of your day, rest of the evening, and I'll see you soon. Take care, everybody. Thank you very much. Thank you, Gisellepe.