Prior-RadGraphFormer: A Prior-Knowledge-Enhanced Transformer for Generating Radiology Graphs from X-Rays

Yiheng Xiong, Jingsong Liu, Kamilia Zaripova, Sahand Sharifzadeh, Matthias Keicher, Nassir Navab

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The extraction of structured clinical information from free-text radiology reports in the form of radiology graphs has been demonstrated to be a valuable approach for evaluating the clinical correctness of report-generation methods. However, the direct generation of radiology graphs from chest X-ray (CXR) images has not been attempted. To address this gap, we propose a novel approach called Prior-RadGraphFormer that utilizes a transformer model with prior knowledge in the form of a probabilistic knowledge graph (PKG) to generate radiology graphs directly from CXR images. The PKG models the statistical relationship between radiology entities, including anatomical structures and medical observations. This additional contextual information enhances the accuracy of entity and relation extraction. The generated radiology graphs can be applied to various downstream tasks, such as free-text or structured reports generation and multi-label classification of pathologies. Our approach represents a promising method for generating radiology graphs directly from CXR images, and has significant potential for improving medical image analysis and clinical decision-making. Our code is open sourced at https://github.com/xiongyiheng/Prior-RadGraphFormer.

Original languageEnglish
Title of host publicationGraphs in Biomedical Image Analysis, and Overlapped Cell on Tissue Dataset for Histopathology - 5th MICCAI Workshop, GRAIL 2023 and 1st MICCAI Challenge, OCELOT 2023, Held in Conjunction with MICCAI 2023, Proceedings
EditorsSeyed-Ahmad Ahmadi, Sérgio Pereira
PublisherSpringer Science and Business Media Deutschland GmbH
Pages54-63
Number of pages10
ISBN (Print)9783031550874
DOIs
StatePublished - 2024
Event5th Workshop on GRaphs in biomedicAl Image anaLysis Satellite event at MICCAI, GRAIL 2023 and 1st Cell Detection from Cell-Tissue Interaction challenge in MICCAI, OCELOT 2023 Held in Conjunction with International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023 - Vancouver, Canada
Duration: 8 Oct 202312 Oct 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14373 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference5th Workshop on GRaphs in biomedicAl Image anaLysis Satellite event at MICCAI, GRAIL 2023 and 1st Cell Detection from Cell-Tissue Interaction challenge in MICCAI, OCELOT 2023 Held in Conjunction with International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023
Country/TerritoryCanada
CityVancouver
Period8/10/2312/10/23

Keywords

  • Prior Knowledge
  • Radiology Graph Generation
  • Transformer

Fingerprint

Dive into the research topics of 'Prior-RadGraphFormer: A Prior-Knowledge-Enhanced Transformer for Generating Radiology Graphs from X-Rays'. Together they form a unique fingerprint.

Cite this