African Observatory on Responsible AI’s Post

New research: Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency New research co-authored by Samuel T. Segun, PhD, Dr. Matthew Smith, Prof Jonathan Shock, Dr. Olatunji Iyiola Emmanuel (李白) and Prof. Tegawendé F. Bissyandé. evaluated 38 AI models across more than 8,900 scholarly references spanning 24 topics. The findings show that factual recall quality follows a measurable pattern determined by model size and how well a topic is represented in training data. The development relevance is stark. Topics like climate change, with over a million associated scholarly works, returned high-quality results even from mid-sized models. Topics like school dropout prevention in rural areas, returned poor results even from the largest models tested. The authors estimate that achieving comparable reliability on such a low-frequency topic would require a model roughly 30 times larger than anything currently available. For AI deployed in health, agriculture, education, and governance contexts across lower-and middle-income countries, where locally relevant training data is frequently limited, this is a governance challenge. Read the full paper and access additional resources: https://lnkd.in/eymP6A7N

To view or add a comment, sign in

Explore content categories