Yuyin Zhou, Assistant Professor of Computer Science and Engineering at the University of California, Santa Cruz, researches the intersection of machine learning and computer vision, with a primary focus on artificial intelligence (AI) for healthcare and scientific discovery. Zhou recently presented her work at the SRG Pillar meeting for the Center for Intelligent Imaging (ci2) at the University of California, San Francisco. Her presentation, "The Thinking Eye: AI That Sees, Reads, and Reasons in Medicine," showcased her findings.
Recent research by Zhou sought to rethink the UNet architecture design for medical imaging through a 3-dimensional lens. "We were able to see that AI was able to achieve performance that is on par with human performance," Zhou explains. "More importantly, we find that sometimes [AI] can capture the particularly very, very small tumors that are even missed by annotators."
Beyond using AI for visual assistance, it can also draft text, summarize content, generate code and translate. Clinicians are often using AI tools to support the healthcare process, Zhou states. One of the fundamental challenges impeding AI research is data limitation. There is a shortage of high-quality, available data. The accessible data is often unimodal, fragmented and lacks fine-grained alignment between modalities.
Existing multimodal datasets lack detailed descriptions and local-global relations. Zhou and her team developed the Medtrinity-25M, which includes multigranular annotations, attributes and rich information. The Medtrinity-25M consists of 25 million images across 10 imaging modalities, 65 disease types and 48 anatomical structures. Her study found that pretraining on Medtrinity-25M with multigranular alignment can improve performance on VLMs and CLIP-based models.
To determine if AI can reason with medicine and think like a human, Zhou developed an evaluation pipeline that includes aspects, tasks, datasets, prompt strategies and large language models (LLMs). The next step analyzed what information the AI was enhancing: adding medical knowledge or improving complex reasoning capabilities. "Our key results are that in the medical domain, accurate knowledge is actually far more important than the reasoning capabilities," Zhou said.
On the implementation of high-quality chain-of-thought reasoning data into the medical datasets, Zhou said: "Our generated reasoning contains much fewer hallucinations and is of a much better quality."
To learn more about the upcoming SRG Pillar meetings, visit the ci2 events page.
