Google’s Med-PaLM 2 Successfully Answers Exam Questions

Google has unveiled new health initiatives and collaborations at its yearly “The Check Up” event. The Google Health team provided updates on new search capabilities. Additionally, it provided tools for creating health applications, and the most recent developments in AI-driven health research.

The firm revealed new collaborations to advance AI-assisted ultrasounds, cancer treatments, and TB tests during the event. Still, the newest iteration of its Med-PaLM medical language model, Med-PaLM 2, generated the most interest.

Med-PaLM

Late last year, Google initially unveiled Med-PaLM. Its goal is to provide superior solutions to medical questions. On multiple-choice questions like those in US medical licensure exams, Med-PaLM was the first AI system to achieve a score of over 60% effectively.

According to the team, Med-PaLM demonstrated a performance level similar to that of medical professionals in most tested benchmarks. The researchers discovered that the model created potentially dangerous reactions 5.9 percent of the time. This is equal to the rate of 5.7 percent seen by human experts. Google Research and DeepMind provided the initial MedPaLM model in December 2022. MultiMedQA, an open-source benchmark for medical question-answering, was used to test the model.

The MedPaLM 2

According to the MedPaLM 2 team, their model passed the USMLE MedQA medical test with an overall score of 85%. This is similar to what an expert doctor will get. This is a 20% improvement over the previous MedPaLM version, surpassing other comparable AI models like GPT-4. Also, the team scored well on benchmark exams like the MedMCQA and the MMLU clinical themes.

According to Vivek Natarajan, MCS, a research scientist at Google Health AI, this model’s success may be attributed to the researchers’ access to cutting-edge technology. Natarajan also noted that specialized medical knowledge enabled them to plan how to train the AI models precisely.

To guarantee that Med-PaLM 2 fulfills Google’s quality requirements, the team still believes there is a lot of space for development.

Med-PaLM 2 was examined by clinicians and non-clinicians from various backgrounds and nations against 14 criteria. The criteria included scientific factuality, correctness, medical consensus, reasoning, prejudice, and damage.

The team is silent on any technological modifications from Med-PaLM 1. Also, the group discovered major gaps when it came to addressing medical questions but did not go into detail about the drawbacks.

Google strives to enhance Med-PaLM in collaboration with research teams to fill in these gaps and comprehend how the language model could enhance healthcare.

The featured image is from towardsdatascience.com