Evaluating Advanced AI in Oncology Education and Clinical Knowledge Assessment

Abstract

The rapid advancement of artificial intelligence (AI) has introduced powerful tools like the Multimodal Large Language Model (MLLM) with the potential to revolutionize medical practices, including oncology. This study investigates the performance of two such MLLMs, GPT-4o and Gemini Advanced, in answering oncology examination questions from the American Society of Clinical Oncology Self-Evaluation Program (ASCO-SEP) Question Bank. We extracted 832 multiple-choice questions from this bank, covering various oncological tasks such as diagnosis, treatment recommendations, and basic science knowledge. Both models were presented with these questions, and their responses were evaluated against the official answer key. Gemini Advanced outperformed GPT-4o, achieving 74.84% accuracy compared to 60% for GPT-4o. Further analysis revealed that Gemini Advanced consistently outperformed GPT-4o across all task categories, particularly in making diagnoses, ordering and interpreting test results, and recommending treatment or patient care. Both models encountered the most difficulty with questions related to pathophysiology and basic science knowledge. These findings suggest that while both MLLMs demonstrate a significant understanding of oncological knowledge, there remains room for improvement, particularly in handling complex clinical scenarios and integrating basic science knowledge. This study contributes to the growing body of evidence assessing the capabilities and limitations of AI in medical oncology, highlighting its potential role in augmenting clinical practice and medical education.