Method

How items are collected, how research is graded, and what the evidence levels mean.

What we grade

We grade a specific finding about learners, not a journal, product or article as a whole. We grade how much the source lets you trust the claim, not whether the result was positive: a well-run study with a null result can earn a high level.

Standards we borrow from

Our rules borrow concepts from the What Works Clearinghouse standards (study design, attrition, baseline equivalence), the ESSA tiers of evidence, Evidence for ESSA (caution about researcher-made measures and very short studies) and GRADE (downgrading for risk of bias, indirectness and imprecision). We do not issue ratings under any of them, and our levels are not official tiers.

Evidence levels

Evidence levels describe how much a source lets you trust a claim about learners, based on study design, who took part and what was measured. They are not endorsements, and they are not official WWC, ESSA or GRADE ratings. See all labels and badges.

Level 3: Controlled comparison

Randomized trial not yet human-verified, or a comparison-group study with baseline data, with real learners. Preprints stop here.

Level 2: Correlational

Associations in real-learner data with statistical controls, or a comparison without baseline equivalence. Perception- or engagement-only outcomes stop here.

Level 1: Descriptive or exploratory

Qualitative, case study, survey, pilot or deployment with real learners or educators, and no comparison group.

Technical test

Only AI models, simulated students, benchmarks or audits of AI outputs. Tells us about the technology, not about student learning.

Commentary

Conceptual, opinion, policy or framework work, and news coverage of others' work until the study it describes is itself graded.

No study cited

A launch, announcement or claim with no traceable study.

Not yet classified

No abstract or methods available yet, for example a publisher listing.

Key rules

  • Real learners first. Studies run only on AI models, simulated students or benchmarks are always a Technical test, however rigorous.
  • Caps. Preprints can reach Level 3 at most until peer reviewed. Perception- or engagement-only outcomes stop at Level 2. Vendor-linked studies and single-session studies are held at Level 3 until checked.
  • Downgrades. A level drops for a mismatched population, fewer than 100 learners (or fewer than 2 classes or schools per condition) with no confidence interval or power information (our own threshold, not a WWC number), clustered designs analyzed as individuals, or researcher-made measures with no reliability evidence.
  • Level 4 is never automatic. A strong design with real learners and learning outcomes can only be confirmed by a person reading the full paper (attrition, baseline equivalence, measures, preregistration). No item currently holds Level 4.
  • Unknown means unknown. Missing details are shown as not reported, never guessed.

How research is graded

  1. Scripts collect items from public feeds: education news outlets, peer-reviewed journals and preprint servers such as arXiv. Items that do not mention AI and education are filtered out.
  2. Research items get a provisional level from fixed, published rules applied to the abstract.
  3. Two independent AI graders and blindly grade each study against this method. When they disagree, the lower level is used and the study is flagged.
  4. Every study at Levels 1 to 3, and every flagged study, then gets a second review against this method. For the October 2026 back catalog that review was done by an AI reviewer on the editor's behalf; no label has yet been verified by a human, and each card shows its review status. Off-topic items are dropped.
  5. News, commentary and launches are labeled by rule: news coverage is labeled Commentary until the study it describes is located and graded, and launches with no traceable study are No study cited. Research items not yet through the grader review show a Provisional badge.

The back catalog started as 283 arXiv studies collected over the past year. 110 are in the tracker; 173 were dropped as off-topic (not about education or learning): 50 off-topic (queue decision), 123 off-topic (keyword screen, not in queue).

Learning principle tags

Retrieval practice, Spacing and interleaving, Feedback, Cognitive load, Worked examples and explicit instruction, Metacognition and self-regulation, Motivation and engagement, Reading and writing, Inclusion and accessibility, Tutoring and personalization. Tags are applied by keyword rules when an item mentions a principle; a tag means the item discusses it, not that it is good evidence about it. Separate topic tags cover assessment and grading, academic integrity, and policy and governance.

Disclaimer

Evidence Signal is an automated tracker. Items are collected from public sources and labeled by fixed, published rules, not reviewed one by one by a person. Labels describe what kind of evidence a source cites. They are not ratings, endorsements, or advice. Summaries are excerpts from the original source, which is always linked. Read it before you act on anything here. Spot an error? Use the corrections link and we'll fix it and log the change.

Sponsor policy

Evidence Signal accepts sponsors whose products or services serve educators, but sponsorship never buys coverage, placement in the tracker, or a better evidence level. Every sponsored item is labeled "Sponsored" and kept separate from the automated tracker. Sponsors cannot review or edit content before it is published. We do not accept sponsors that sell student data, make claims of learning results they won't back with a source, or market directly to children. Labels on a sponsor's own products are set by the same fixed rules as everyone else's, and we may decline or end any sponsorship that conflicts with these terms.

Corrections

Spot a wrong label, broken link or error? Email a correction. Fixes are listed in the corrections log.