A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

Specialized AI models can be more accurate for specific medical diagnostic tasks than general-purpose LLMs.

A 10-year-old convolutional neural network (CNN) has demonstrated superior performance in diagnosing skin cancer compared to two leading large language models (LLMs), GPT-6 Astra and Claude Fable 5.1. In a test involving 200 biopsy-proven skin lesions, the fine-tuned Inception v3 CNN correctly identified the diagnosis 62% of the time, while the LLMs achieved only 42%. The CNN also excelled at distinguishing between potentially harmful and benign lesions, with a significantly lower rate of discharging unsafe lesions. When asked as a patient whether they needed to see a doctor, both LLMs advised seeking medical attention for nearly all lesions, including benign ones. The CNN, a 2015 architecture, was fine-tuned on publicly available dermatology images and was designed to run on a phone browser. This research highlights the continued relevance of specialized AI models for specific diagnostic tasks, even in the presence of more general-purpose frontier LLMs.

This study suggests that for highly specialized and visual diagnostic tasks like skin cancer identification, older, fine-tuned CNN architectures may still outperform cutting-edge LLMs. The LLMs, despite their advanced natural language understanding and broad knowledge bases, struggled with the nuances of dermatological images, misclassifying critical lesions like melanomas. This indicates that while LLMs are powerful, they may require specific adaptations or may not be the optimal solution for all AI-driven medical applications.

The findings have implications for the development and deployment of AI in healthcare, suggesting a continued need for domain-specific models. It also raises questions about the training data and architectural biases of large language models, which may not adequately prepare them for precise visual interpretation in critical fields. Future research might focus on hybrid approaches, combining the strengths of LLMs with specialized image analysis models, or on more effective fine-tuning strategies for LLMs in medical contexts.