Martin Goetze, a visiting student from RWTH Aachen University, presented his research at a recent SRG Pillar meeting for the Center for Intelligent Imaging (ci2) at the University of California, San Francisco (UCSF). His work explored the integration of medical images, radiology reports, and structured data into foundation models to improve AI-driven classification and segmentation in health care.
Goetze highlighted the challenge of obtaining annotated data for AI training. "Medical models are expensive just because you need these annotations," he noted. His research seeks to mitigate this cost by leveraging foundation models that generalize across tasks with fewer annotations.
His approach applies multimodal models to lung CT volumes, incorporating text embeddings from radiology reports and structured data from electronic health records (EHRs). "Foundation models work as follows: You throw lots of annotation data at them, and you have them learn essentially on their own," he explained. This strategy reduces training costs while maintaining performance.
To ensure data quality, Goetze's team curated CT scans, selecting only complete, high-quality images. "We want scans that actually go from top to bottom," he stated. The team also developed a preprocessing pipeline to structure the data for model training.
Evaluation results demonstrated that integrating structured data improved classification accuracy in certain cases. "Supervised training might help us in getting over these shortcuts," he observed, referring to AI's tendency to rely on irrelevant cues. While zero-shot classification showed promise, not all modifications yielded improvements.
Goetze concluded by emphasizing the value of UCSF's Information Commons dataset. "IC is great. There's lots of data there. Once it's de-identified, use it," he urged. His research highlights the potential of foundation models to enhance AI-driven medical applications through diverse data integration.
To learn more about the upcoming SRG Pillar meetings, visit the ci2 events page.
