{"id":1101105,"date":"2024-11-18T02:14:40","date_gmt":"2024-11-18T10:14:40","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=1101105"},"modified":"2025-10-31T13:38:59","modified_gmt":"2025-10-31T20:38:59","slug":"biomedparse-a-foundation-model-for-smarter-all-in-one-biomedical-image-analysis","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/biomedparse-a-foundation-model-for-smarter-all-in-one-biomedical-image-analysis\/","title":{"rendered":"BiomedParse: A foundation model for smarter, all-in-one biomedical image analysis"},"content":{"rendered":"\n
\"A<\/figure>\n\n\n\n

In cancer diagnosis or advanced treatments like immunotherapy, every detail in a medical image counts. Radiologists and pathologists rely on these images to track tumors, understand their boundaries, and analyze how they interact with surrounding cells. This work demands pinpoint accuracy across several tasks\u2014identifying whether a tumor is present, locating it precisely, and mapping its contours on complex CT scans or pathology slides. <\/p>\n\n\n\n

Yet, these crucial steps\u2014object recognition, detection, and segmentation\u2014are often tackled separately, which can limit the depth of analysis. Current tools like MedSAM (opens in new tab)<\/span><\/a> and SAM (opens in new tab)<\/span><\/a> focus on segmentation only, thus missing out on the opportunity to blend these insights holistically and relegating object as an afterthought. <\/p>\n\n\n\n

In this blog, we introduce BiomedParse (opens in new tab)<\/span><\/a>, a new approach for holistic image analysis by treating object as the first-class citizen. By unifying object recognition, detection, and segmentation into a single framework, BiomedParse allows users to specify what they\u2019re looking for through a simple, natural-language prompt. The result is a more cohesive, intelligent way of analyzing medical images that supports faster, more integrated clinical insights.\u00a0<\/p>\n\n\n\n

While biomedical segmentation datasets abound, there are relatively few prior works on object detection and recognition in biomedicine, let alone datasets covering all three tasks. To pretrain BiomedParse, we created the first such dataset by harnessing OpenAI\u2019s GPT-4 for data synthesis from standard segmentation datasets (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

BiomedParse is a single foundation model that can accurately segment biomedical objects across nine modalities, as seen in Figure 1, outperforming prior best methods while requiring orders of magnitude fewer user operations, as it doesn\u2019t require an object-specific bounding box. By learning semantic representation for individual object types, BiomedParse\u2019s superiority is particularly pronounced in the most challenging cases with irregularly shaped objects. Through joint pretraining of object recognition, detection, and segmentation, BiomedParse opens new possibilities for holistic image analysis and image-based discovery in biomedicine.  <\/p>\n\n\n\n

\"a,
Figure 1. Overview of BiomedParse and BiomedParseData.<\/em><\/figcaption><\/figure>\n\n\n\n

Image parsing: a unifying framework for holistic image analysis <\/h2>\n\n\n\n

Back in 2005, researchers first introduced the concept of \u201cimage parsing\u201d\u2014a unified approach to image analysis that jointly conducts object recognition, detection, and segmentation. Built on Bayesian networks, this early model offered a glimpse into a future of joint learning and reasoning in image analysis, though it was limited in scope and application. Fast forward to today, cutting-edge advances in generative AI have breathed new life into this vision. With our model, BiomedParse, we have created a foundation for biomedical image parsing that leverages interdependencies across the three subtasks, thus addressing key limitations in traditional methods. BiomedParse enables users to simply input a natural-language description of an object, which the model uses to predict both the object label and its segmentation mask, thus eliminating the need for a bounding box (Figure 1c). In other words, this joint learning approach lets users segment objects based on text alone.<\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tvideo series<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"On\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

On Second Thought<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

A video series with Sinead Bovell built around the questions everyone\u2019s asking about AI. With expert voices from across Microsoft, we break down the tension and promise of this rapidly changing technology, exploring what\u2019s evolving and what\u2019s possible.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tExplore the series\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Harnessing GPT-4 for large-scale data synthesis from existing datasets <\/h2>\n\n\n\n

We created the first dataset for biomedical imaging parsing by harnessing GPT-4 for large-scale data synthesis from 45 existing biomedical segmentation datasets (Figure 1a and 1b). The key insight is to leverage readily available natural-language descriptions already in these datasets and use GPT-4 to organize this often messy, unstructured text with established biomedical object taxonomies.  <\/p>\n\n\n\n

Specifically, we use GPT-4 to help create a unifying biomedical object taxonomy for image analysis and harmonize natural language descriptions from existing datasets with this taxonomy. We further leverage GPT-4 to synthesize additional variations of object descriptions to facilitate more robust text prompting.  <\/p>\n\n\n\n

This enables us to construct BiomedParseData, a biomedical image analysis dataset comprising over 6 million sets of images, segmentation masks, and text descriptions drawn from more than 1 million images. This dataset includes 64 major biomedical object types, 82 fine-grained subtypes, and spans nine imaging modalities.<\/p>\n\n\n\n

\"a,
Figure 2: Comparison on large-scale biomedical image segmentation datasets.<\/figcaption><\/figure>\n\n\n\n

State-of-the-art performance across 64 major object types in 9 modalities<\/h2>\n\n\n\n

We evaluated BiomedParse on a large held-out test set with 102,855 image-mask-label sets across 64 major object types in nine modalities. BiomedParse outperformed prior best methods such as MedSAM and SAM, even when oracle per-object bounding boxes were provided. In the more realistic setting when MedSAM and SAM used a state-of-the-art object detector (Grounding DINO) to propose bounding boxes, BiomedParse outperformed them by a wide margin, between 75 and 85 absolute points in dice score (Figure 2a). BiomedParse also outperforms a variety of other prominent methods such as SegVol, Swin UNETR, nnU-Net, DeepLab V3+, and UniverSeg.<\/p>\n\n\n\n

\"a,
Figure 3. Evaluation on detecting irregular-shaped objects.<\/figcaption><\/figure>\n\n\n\n

Recognizing and segmenting irregular and complex objects<\/h2>\n\n\n\n

Biomedical objects often have complex and irregular shapes, which present significant challenges for segmentation, even with oracle bounding box. By joint learning with object recognition and detection, BiomedParse learns to model object-specific shapes, and its superiority is particularly pronounced for the most challenging cases (Figure 3). Encompassing a large collection of diverse object types in nine modalities, BiomedParseData also provides a much more realistic representation of object complexity in biomedicine.  <\/p>\n\n\n\n

\"a,
Figure 4. Evaluation on object recognition.<\/figcaption><\/figure>\n\n\n\n

Promising step toward scaling holistic biomedical image analysis<\/h2>\n\n\n\n

By operating through a simple text prompt, BiomedParse requires substantially less user effort than prior best methods that typically require object-specific bounding boxes, especially when an image contains a large number of objects (Figure 2c). By modeling object recognition threshold, BiomedParse can detect invalid prompt and reject segmentation requests when an object is absent from the image. BiomedParse can be used to recognize and segment all known objects in an image in one fell swoop (Figure 4). By scaling holistic image analysis, BiomedParse can potentially be applied to key precision health applications such as early detection, prognosis, treatment decision support, and progression monitoring.  <\/p>\n\n\n\n

Going forward, there are numerous growth opportunities. BiomedParse can be extended to handle more modalities and object types. It can be integrated into advanced multimodal frameworks such as LLaVA-Med (opens in new tab)<\/span><\/a> to facilitate conversational image analysis by \u201ctalking to the data.\u201d To facilitate research in biomedical image analysis, we have made BiomedParse open-source (opens in new tab)<\/span><\/a> with Apache 2.0 license. We\u2019ve also made it available on Azure AI (opens in new tab)<\/span><\/a> for direct deployment and real-time inference. For more information, check out our demo. (opens in new tab)<\/span><\/a> <\/p>\n\n\n\n

BiomedParse is a joint work with Providence and the University of Washington’s Paul G. Allen School of Computer Science & Engineering, and brings collaboration from multiple teams within Microsoft*. It reflects Microsoft\u2019s larger commitment to advancing multimodal generative AI for precision health, with other exciting progress such as GigaPath<\/a>, BiomedCLIP<\/a>,\u202f LLaVA-Rad<\/a>, BiomedJourney (opens in new tab)<\/span><\/a>, MAIRA<\/a>, Rad-DINO<\/a>, Virchow (opens in new tab)<\/span><\/a>.\u202f <\/p>\n\n\n\n

(Acknowledgment footnote) *: Within Microsoft, it is a wonderful collaboration among Health Futures, MSR Deep Learning, and Nuance. <\/em><\/p>\n\n\n\n

Paper co-authors: Theodore Zhao, Yu Gu, Jianwei Yang<\/a>, Naoto Usuyama<\/a>, Ho Hin Lee, Sid Kiblawi, Tristan Naumann<\/a>, Jianfeng Gao<\/a>, Angela Crabtree, Jacob Abel, Christine Moung-Wen, Brian Piening, Carlo Bifulco, Mu Wei, Hoifung Poon<\/a>, Sheng Wang (opens in new tab)<\/span><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

BiomedParse reimagines medical image analysis, integrating advanced AI to capture complex insights across imaging types\u2014a step forward for diagnostics and precision medicine.<\/p>\n","protected":false},"author":38004,"featured_media":1102929,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_hide_image_in_river":null,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13553],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,243984,269142],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1101105","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-medical-health-genomics","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-blog-homepage-featured","msr-post-option-include-in-river"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[849856],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144931],"related-projects":[978063],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Hoifung Poon","user_id":32016,"display_name":"Hoifung Poon","author_link":"Hoifung Poon<\/a>","is_active":false,"last_first":"Poon, Hoifung","people_section":0,"alias":"hoifung"},{"type":"guest","value":"theodore-zhao","user_id":"1104840","display_name":"Theodore Zhao","author_link":"Theodore Zhao","is_active":true,"last_first":"Zhao, Theodore","people_section":0,"alias":"theodore-zhao"},{"type":"guest","value":"mu-wei","user_id":"654207","display_name":"Mu Wei","author_link":"Mu Wei","is_active":true,"last_first":"Wei, Mu","people_section":0,"alias":"mu-wei"},{"type":"guest","value":"sheng-wang-2","user_id":"1104045","display_name":"Sheng Wang","author_link":"Sheng Wang<\/a>","is_active":true,"last_first":"Wang, Sheng","people_section":0,"alias":"sheng-wang-2"}],"msr_type":"Post","featured_image_thumbnail":"\"A","byline":"","formattedDate":"November 18, 2024","formattedExcerpt":"BiomedParse reimagines medical image analysis, integrating advanced AI to capture complex insights across imaging types\u2014a step forward for diagnostics and precision medicine.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/1101105","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/38004"}],"replies":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1101105"}],"version-history":[{"count":27,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/1101105\/revisions"}],"predecessor-version":[{"id":1154360,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/1101105\/revisions\/1154360"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/1102929"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1101105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1101105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1101105"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1101105"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1101105"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1101105"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1101105"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1101105"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1101105"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1101105"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1101105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}