A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on skin cancer diagnosis. Both misdiagnosed an invasive melanoma as harmless. A 10-year-old CNN identified it correctly. The test used 200 biopsy-proven lesions from the MILK10k dataset. The CNN achieved 62% accuracy versus 42% for each LLM. It performed best on benign moles and seborrhoeic keratoses. The LLMs favored incorrect common diagnoses like basal cell carcinoma. The CNN also discharged fewer unsafe lesions at its default threshold. The LLMs advised seeing a doctor for nearly all cases when asked as a patient. The CNN runs in a browser without uploading data. The author provided full test data and code for replication.
What changed
Frontier LLMs were assumed capable of matching or exceeding older specialized models in medical diagnosis.
What it unlocks
Using a small, fine-tuned CNN for skin cancer diagnosis instead of general-purpose LLMs.
- 200 biopsy-proven lesions tested
- 62% CNN accuracy vs 42% LLM accuracy
- 30 melanomas in test set
- 125 correct CNN diagnoses
What you need to act on it
- dermoscopic and clinical images
- fine-tuning on dermatology data
Sources