As Large Language Models (LLMs) continue to transform healthcare, assessing their reliability in specialized domains like clinical nutrition is paramount. A comprehensive new study evaluates how leading LLM chatbots handle complex dietary inquiries using the Registered Dietitian (RD) exam as a benchmark.
Methodology
The research assessed three top-tier models—GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—against 1,050 multiple-choice questions from the RD exam. To determine the most effective communication strategies, the study tested four distinct prompt engineering techniques: Zero-Shot (ZS), Chain of Thought (CoT), Chain of Thought with Self Consistency (CoT-SC), and Retrieval Augmented Prompting (RAP).
Findings
All tested models demonstrated high baseline proficiency, scoring over 88% accuracy on the exam. However, performance varied significantly based on the prompting method and question complexity:
- Highest Accuracy: GPT-4o utilizing CoT-SC outperformed all other approaches, minimizing errors across multiple proficiency levels.
- Greatest Consistency: Gemini 1.5 Pro using ZS prompting delivered the most consistent responses across repeated tests.
- Role of Prompting: Advanced techniques like CoT improved accuracy for GPT-4o and Claude 3.5, while RAP proved particularly effective in helping GPT-4o tackle expert-level questions by retrieving external nutritional knowledge.
Impact
The findings illustrate that while LLMs possess strong foundational knowledge in dietetics, achieving clinical-grade reliability requires strategic prompt engineering. By tailoring models and prompts to specific task complexities, developers can mitigate risks and enhance the safety of AI-driven nutrition chatbots for both clinicians and patients.
Authors: Iman Azimi, Mohan Qi, Li Wang, Amir M. Rahmani, Youlin Li
This research was powered by the Livi Platform.
Read the full paper here: https://www.nature.com/articles/s41598-024-85003-w
Join our network for deeper insights: https://newsletter.livi.health/signup
