In the “TraumAInterfaces” project, Sven Giesselbach, Team Lead Natural Language Understanding at Fraunhofer IAIS, and his team developed the “TraumAgent” and the “FormAssistant” together with several partners. These prototypes are AI technologies based on foundation models and large language models (LLM), which have been trained on large amounts of unstructured data. Their task is to facilitate the collection and documentation of information in a clinical context. They do this, for example, by automatically recording and evaluating relevant information from conversations in the emergency room via microphones and will be adapted for on-premise use to ensure compliance with data protection regulations.

Especially in the emergency room, the medical staff faces numerous challenges. The time-critical care of seriously injured patients in the so-called trauma room, where the initial treatment of the severely injured takes place, is a complex process. A lot of relevant information is communicated quickly via spoken language. This is where the “TraumAgent” comes in: “It offers significant support during trauma room treatments, as it can clearly display relevant information, phases and guidelines,” explains Sven Giesselbach. Simply put, the “TraumAgent” is an artificial intelligence, which is connected to a live display in the trauma room and documents information. Current process steps are displayed on a screen and measures carried out are recognized. Instead of just reacting to actions, the so-called agent can independently filter relevant information and summarize them for the medical team. This relieves the burden on doctors, who have to make vital decisions in a matter of seconds. Artificial intelligence provides them with decision-making aids and at the same time automatically collects important information that is therefore not lost.

The “FormAssistant” additionally helps with automatically filling out the patient's trauma register form. This form summarizes the most important information about a trauma room treatment and enables a comparative quality analysis across hospitals. With the help of an LLM agent, the form assistant supports the medical staff with administrative tasks that normally require a lot of time. “The use of AI in the emergency room has many advantages. For example, resources can be better utilized because workflows become more efficient. This saves time and costs and improves the quality of care,” explains Sven Giesselbach.

Even dialects and accents can be easily evaluated by artificial intelligence

“Foundation models offer incredible potential to support and partially automate processes, even in critical areas such as emergency care,” explains Sven Giesselbach. It has been shown that the “TraumAgent” delivers significantly better results than previous methods. For example, the prototype does not only document information, but can also correct misinformation by harnessing its medical knowledge and external resources. This prevents hallucinations, for example, which are typical for some AI models. Hallucinations refer to convincing-sounding content that may be invented by the AI and is therefore incorrect.

The speech recognition system developed by Fraunhofer can also deal well with background noises in the trauma room as well as with dialects, accents and slips of the tongue from hospital staff. Even if the speech recognition fails, the language models can correct errors. The implementation of AI applications will be possible in the future with the help of data protection-compliant integration on cloud platforms. Alternatively, recordings could be processed directly on end devices or on the hospital server, which would meet the high data protection requirements in the German healthcare sector. For this purpose, locally operable language models will be used to ensure independence from cloud providers. Therefore, the scientists will access the LLM from the OpenGPT-X project, a European solution for companies and institutions that is currently being developed and tested in a consortium led by Fraunhofer.

Foundation models and LLM are therefore promising technologies that open numerous potential applications. For example, data acquired from the project can be used as the basis for further AI applications, or an interface to the electronic patient file can be created. This makes it possible to transfer information along the entire treatment chain. It is also possible to use applications such as “TraumAgent” and “FormAssistant” in an adapted form in many other areas such as the police, fire department or public administration.

Fraunhofer IAIS will be presenting the potential applications of foundation models in the healthcare sector at this year's DMEA exhibition in Berlin from April 9 to 11 (Hall 2.2 Booth D-108).

The white paper “Artificial intelligence in the trauma room: How LLM agents and foundation models can support the treatment of severely injured patients” was developed by Fraunhofer IAIS in collaboration with Universitätsklinikum Aachen, RWTH Aachen and Klinikum Köln-Merheim. The University Witten/Herdecke was also a project partner for “TraumAInterfaces”. The AI applications were tested in a simulated trauma room at the Klinikum Köln-Merheim. 

The white paper is available to download for free (in German language): www.iais.fraunhofer.de/traumAInterfaces