Asclepius

Home Leaderboard

GitHub

中 |EN

Asclepius Leaderboard

Click on the upper right corner toJoin Asclepius Evaluation

For development, we provide the development set, which can be downloaded here.

If you wish to make your evaluation results public on the leaderboard below after the evaluation is completed, please send an email to valentinachen007@gmail.com.

Our Asclepius benchmark evaluates Med-MLLMs for the Specialty Analysis with 15 medical specialties and the Capacity Analysis with 8 clinical tasks.

⚕Specialty Analysis

	Method	Overall	Card	Derm	Endo	Gastro	GenSurg	Hem	Immun	Neurol	ObsGyn	Ophth	Orth	Oto	Path	Pulm	Urol


1	GPT-4v 🥇 OpenAI	0.522	0.609	0.455	0.526	0.434	0.370	0.633	0.485	0.550	0.504	0.405	0.214	0.363	0.489	0.399
2	Gemini 🥈 Google	0.327	0.314	0.364	0.412	0.279	0.260	0.500	0.332	0.390	0.469	0.274	0.107	0.306	0.436	0.288
3	CheXagent 🥉 Stanford AIMI	0.278	0.308	0.182	0.289	0.380	0.242	0.167	0.192	0.240	0.431	0.189	0.107	0.279	0.424	0.294
4	RadFM Shanghai AI Laborato	0.322	0.195	0.091	0.325	0.450	0.238	0.133	0.279	0.220	0.296	0.221	0.214	0.244	0.307	0.288
5	Med-Flamingo Stanford University	0.288	0.237	0.000	0.268	0.302	0.275	0.133	0.214	0.190	0.372	0.168	0.179	0.207	0.405	0.221
6	XrayGPT MBZUAI	0.210	0.178	0.045	0.191	0.163	0.117	0.167	0.135	0.110	0.205	0.142	0.036	0.086	0.151	0.129

⚕Capacity Analysis

	Method	Overall	Anato	Attr	SpaQua	DisIde	Stag	Prog	Treat	Rep


1	GPT-4v 🥇 OpenAI	0.462	0.323	0.385	0.552	0.649	0.48	0.504	0.524	N.A.
2	Gemini 🥈 Google	0.354	0.285	0.292	0.342	0.654	0.342	0.496	0.323	0.082
3	CheXagent 🥉 Stanford AIMI	0.309	0.238	0.253	0.321	0.524	0.252	0.451	0.315	0.157
4	RadFM Shanghai AI Laboratoy	0.278	0.344	0.298	0.212	0.13	0.396	0.295	0.29	0.091
5	Med-Flamingo Stanford University	0.279	0.27	0.256	0.217	0.587	0.272	0.398	0.145	0.133
6	XrayGPT MBZUAI	0.148	0.163	0.107	0.152	0.082	0.104	0.223	0.145	0.078