Did a fact-checking chatbot teach people to check new claims on their own?
Evidence level: Level 3: Controlled comparison. Randomized trial not yet human-verified, or a comparison-group study with baseline data, with real learners. Preprints cannot be rated higher than this until they are peer reviewed. How we grade
Source: Chatbot Engagement Does Not Always Beget Metalearning: Evidence from Three Countries, arXiv preprint, 2026-09-26.
The study in plain language
Researchers ran a preregistered randomized experiment with about 2,200 people in the USA, India and Singapore, recruited through Qualtrics survey panels (full paper). People judged image-and-caption claims, some true and some false or used out of context, and got one of four things: no correction, links only, a static explanation, or a Socratic chatbot. The researchers wanted to know whether the chatbot left behind metalearning, meaning a skill that carries over to claims the chatbot did not discuss.
Right away, the chatbot group showed the largest gain in telling true claims from false ones, though the effect was small (d = 0.097). All three corrections lowered how willing people said they were to share the false claims. That outcome was a self-reported 1 to 5 rating, not observed sharing (full paper). On an unaided retest about a week later, no correction showed a statistically significant lasting advantage when results were pooled across countries. Pooled across countries, the chatbot group trended downward relative to the no-correction group (p = .090, short of conventional significance), with sharper declines in India and Singapore (full paper).
What the abstract says
Chatbots deliver real-time fact-checks, but whether a chatbot correction leaves anything behind once the chatbot is gone - metalearning, distinct from correcting misbeliefs - is untested. We report a preregistered, three-country randomized experiment (USA, India, Singapore; N ~ 2,200) on out-of-context image misinformation, manipulating a correction's channel affordances (synchronicity, bandwidth) across four conditions: Control, Links-only, Static explanation, and a Socratic Chatbot built on a validated out-of-context detector, with an unaided retest one week later. The Chatbot produced the largest immediate discernment gain (d = 0.097, p = .023). All three interventions reduced sharing of false claims (d ~ -0.12, p < .01). One week later, no advantage persisted: the Chatbot arm declined relative to Control, most sharply in India and Singapore, and in India on claims it never discussed. Decay tracked affordance level and did not vary by country. Engagement mechanisms, we argue, do not substitute for slow AI literacy.
Quoted from the abstract at arXiv. In the full paper, "sharing" in the fourth sentence is participants' stated willingness to share.
What it does and doesn't tell us
What it tells us
- Who took part: about 2,200 people recruited through survey panels in the USA, India and Singapore.
- What was measured: accuracy in judging claims, including an unaided retest about a week later, and stated willingness to share false claims.
- The plan was registered before data were collected.
What it doesn't tell us
- It is a preprint, so it has not been through journal peer review yet.
- Participants were retested once, about a week later, so it does not show what happens with repeated instruction over a term or a year.
- The participants were survey-panel members, not students in a class, so it does not show how chatbot feedback affects school learning.
- It measured what people said they would share, not what they actually shared.
- We checked the design against the full paper and kept Level 3: people were randomized one by one, and preprints cannot be rated above Level 3. Level 4 would also require peer review and a person's full verification.
Read it yourself
Read the full study at arXiv. Evidence levels describe study design and who took part. They are not endorsements or official ratings.