Level 2: CorrelationalLearners: K-12 studentsOutcome: Learning or achievementPeer-reviewed (journal reference listed)Researcher-made measureClass-assigned, no clustering adjustment
2026-07-26 · arXiv (back catalog) · Study
Climate change is a socio-scientific issue: it rests on science but cannot be settled by science, because any serious response forces people to weigh costs, values, and competing interests under uncertainty. Helping students make such decisions well is a central aim of science education, and the arrival of generative...
Why this label for Can AI-Assisted Inquiry Enhance Students' Decision-Making Skills in Socio-Scient
- Level 2: Correlational (human override): Full paper (arXiv 2608.28617, Methods and Limitations) says intact classes were assigned to conditions and outcomes were tested at the student level; clustering is discussed only as a limitation, with no clustering-adjusted analysis. Our method lowers a level for class assignment without adjustment.
- Changed from: Level 3: Controlled comparison
- Grader A: R5 design ceiling: three student groups had pretest data and comparable starting levels; random assignment is not reported.
- Grader B: R5 design ceiling: three-group pretest-posttest quasi-experiment (270 secondary students, 90 per group, comparable baseline) gives Level 3; inter-coder agreement reported, so no R6 measurement downgrade; n>=100 so no imprecision downgrade; not an RCT, so not a Level 4 candidate
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Human grade change 2026-10-21, approved by Matthew Rhoads, Ed.D. (approved in chat, 2026-10-21) Method
Level 1: Descriptive or exploratoryStudy type: Case study / case series
2026-10-09 · arXiv search: AI + education · Study (preprint)
Generative AI is increasingly studied as a tool for supporting lesson authoring; the creation and adaptation of classroom materials, yet little is known about how it should fit into secondary computing teachers' subject-specific practices. We conducted two complementary qualitative studies with eight secondary...
Why this label for From Instructional Intent to Classroom-Ready Lessons: Investigating Teacher-AI C
- Level 1: Qualitative focus groups and think-aloud with eight teachers.
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Level 1: Descriptive or exploratoryLearners: TeachersOutcome: Teacher practicePreprintSmall or impreciseStudy type: Case study / case series
2026-10-09 · arXiv (back catalog) · Study (preprint)
Generative AI is increasingly studied as a tool for supporting lesson authoring; the creation and adaptation of classroom materials, yet little is known about how it should fit into secondary computing teachers' subject-specific practices. We conducted two complementary qualitative studies with eight secondary...
Why this label for From Instructional Intent to Classroom-Ready Lessons: Investigating Teacher-AI C
- Level 1: Descriptive or exploratory: Consensus level confirmed against the methodology.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: focus groups and think-aloud sessions with eight teachers are qualitative exploratory research.
- Grader B: R5 design ceiling: qualitative focus groups and think-aloud with 8 teachers, so descriptive/exploratory Level 1
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryProvisional (rules only)
2026-10-07 · arXiv physics.ed-ph · Study (preprint)
Interactive quantum circuit simulators are increasingly used to support introductory quantum computing education for learners from varied disciplinary backgrounds. These environments can also generate detailed response logs of submitted circuits, but such data are rarely transformed into evidence that instructors can...
Why this label for Using Learning Analytics to Study Novice Learners' Work on Quantum Circuit Simul
- Level 1: Descriptive or exploratory: provisional rules-only grade, pending grader consensus and review: R5 design: descriptive (F3): Descriptive, qualitative, pilot or deployment study with real participants.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryProvisional (rules only)
2026-10-02 · arXiv physics.ed-ph · Study (preprint)
Public Exams are an assessment system designed in the context of Biology Education Research to increase student engagement in the exam design process. Students are shown incomplete versions of test questions ahead of time, with the format of the question revealed but some details withheld.
Why this label for Adapting Public Exams for Introductory Physics: A Mixed-Methods Investigation of
- Level 1: Descriptive or exploratory: provisional rules-only grade, pending grader consensus and review: R5 design: descriptive (F3): Descriptive, qualitative, pilot or deployment study with real participants.
- Preprint: not peer reviewed; capped at Level 3
- Motivation and engagement: matched 'student engagement'
- Assessment and grading: matched 'Exams'
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Behavior or engagementPreprintResearcher-made measureSimulated learnersStudy type: Cross-sectional
2026-09-29 · arXiv (back catalog) · Study (preprint)
When a student is stuck, a tutor faces the assistance dilemma: help given too early can hinder productive struggle, while help withheld too long leaves the student in a frustrating, persistent impasse (i.e., wheel spinning). Generative AI tutors increasingly use guardrails restricting answer-giving, yet little is...
Why this label for Examining Variation in How Guided AI Tutors Resolve Student Impasses
- Level 1: Descriptive or exploratory: Real tutoring logs analyzed descriptively, then simulated conditions; graded on the real arm as 1.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: associations in authentic student sessions support a correlational level; simulated tutor conditions do not raise it.
- Grader B: R4 mixed study graded on the real-learner arm only: correlational analysis of authentic sessions (Level 2), downgraded under R6 measurement because the impasse/recovery coding has no reported reliability
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Tutoring and personalization: matched 'Tutors'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Learning or achievementPreprintResearcher-made measureStudy type: Case study / case series
2026-09-28 · arXiv (back catalog) · Study (preprint)
The rapid proliferation of large language models (LLMs) in the context of education has introduced significant challenges in enforcement of academic integrity, especially in programming courses. We present Argus, an automated detection system for LLM-assisted student work in undergraduate C programming assignments.
Why this label for Argus: Academic Integrity in the Era of Generative AI
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: retrospective flagged-use and exam-performance findings report correlation without statistical controls.
- Grader B: R5 design ceiling: system description plus a simple correlation between detector flags and exam scores, with no statistical controls reported, so Level 1; the detector measure is unvalidated
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryProvisional (rules only)
2026-09-27 · arXiv physics.ed-ph · Study (preprint)
The African School of Fundamental Physics and Applications, also known as the African School of Physics (ASP), was initiated in 2010, as a three-week biennial event, to offer additional training in fundamental and applied physics to African students with a minimum of three-year university education. Since its...
Why this label for Activity Report on the Ninth African School of Fundamental Physics and Applicati
- Level 1: Descriptive or exploratory: provisional rules-only grade, pending grader consensus and review: R5 design: descriptive (F3): Descriptive, qualitative, pilot or deployment study with real participants.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Perception or attitudePeer-reviewed (journal reference listed)Perception onlyStudy type: Cross-sectional
2026-07-22 · arXiv (back catalog) · Study
The rise of generative AI (GenAI) in higher education has prompted urgent debates surrounding academic integrity and ethical use. This study examines cross-cultural differences in student perceptions of GenAI use, comparing responses from students at Canadian and South Korean universities.
Why this label for Did Alice Do Wrong? Cross-Cultural Differences in Student Perceptions of Generat
- Level 1: Descriptive or exploratory: Consensus level confirmed against the methodology.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: the scenario survey compares student judgments but reports no controlled intervention or statistical controls.
- Grader B: R5 design ceiling: cross-sectional scenario-based survey comparing Canadian and Korean students' perceptions, so descriptive Level 1 with a perception-only outcome
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Not reportedOutcome: Teacher practicePreprintStudy type: Cross-sectional
2026-07-14 · arXiv (back catalog) · Study (preprint)
With the increased use of generative AI (GenAI) applications such as ChatGPT, higher education institutions (HEIs) have released a range of guidelines and policies to direct adoption within their institutions. In computer science (CS) courses GenAI adoption is especially high and the implications for student learning...
Why this label for A Comparative Analysis of Institutional and Course Generative AI Policies within
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: secondary analysis of policy and syllabus documents is descriptive.
- Grader B: R5 design ceiling: descriptive secondary document analysis of institutional and course AI policies, with no learner outcomes.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: K-12 studentsOutcome: Learning or achievementPreprintSmall or impreciseResearcher-made measureStudy type: Case study / case series
2026-07-07 · arXiv (back catalog) · Study (preprint)
This paper presents the design and outcomes of a seven-weekend AI storytelling program developed for Black girls aged 10-12. Grounded in Afrofuturism and Black feminist thought, the program adopted AI-enabled counter-storytelling, supported the development of foundational AI literacies, and fostered future-oriented...
Why this label for What If AI Carried Her Imagination? Black Girls as Creators in an AI Storytellin
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: learner artifacts come from a case study without a comparison group.
- Grader B: R5 design ceiling: qualitative case study of a weekend program for girls aged 10-12, based on analysis of learner artifacts, with no comparison group
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Technical testProvisional (rules only)Study type: Technical test (no real learners)
2026-10-01 · arXiv physics.ed-ph · Study (preprint)
Earth's ionosphere is primarily generated by the Sun's ionization of atoms in the upper atmosphere. This plasma environment is highly dynamic, exhibits strong spatial and temporal, variations and significantly impacts power systems, navigation, and satellite and ground communications.
Why this label for A Student Sounding Rocket Experiment to Probe Electron Density in the Lower Iono
- Technical test: provisional rules-only grade, pending grader consensus and review: R4 real-learner gate (F1/F2): Outputs of models, simulations or audits; any humans act as raters, references or data sources, not learners.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Technical testProvisional (rules only)Study type: Technical test (no real learners)
2026-09-28 · arXiv physics.ed-ph · Study (preprint)
The idea of engaging critically and creatively with physics theories is a stretch for many physicists and physics educators. Most physicists become enculturated into the belief that physics definitions, laws, and equations are read from the book of nature.
Why this label for Community agreements pedagogy in physics
- Technical test: provisional rules-only grade, pending grader consensus and review: R4 real-learner gate (F1/F2): Outputs of models, simulations or audits; any humans act as raters, references or data sources, not learners.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Technical testLearners: Higher-ed studentsOutcome: Model performancePreprintOpen data or materialsStudy type: Technical test (no real learners)
2026-07-21 · arXiv (back catalog) · Study (preprint)
This full paper in the research-to-practice track presents methods for assessing student teams in tabletop exercises (TTXs). TTXs enable learner teams to prepare for workplace tasks and practice crisis responses, such as resolving cybersecurity incidents.
Why this label for Assessment in Team Problem-Solving Exercises in Computing Education
- Technical test: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R4 real-learner gate: student team records are used to test assessment methods; the reported outcome is method performance.
- Grader B: R4 real-learner gate: data from 81 participants is used only to check clustering and LLM scoring against instructor scores; the only outcome is how well the methods perform, with no learner outcome (accepted at FIE 2026, peer-reviewed version not matched)
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Technical testLearners: Simulated or AI-onlyOutcome: Model performancePreprintStudy type: Technical test (no real learners)
2026-06-12 · arXiv (back catalog) · Study (preprint)
Large Language Model (LLM) based AI educational content generation systems are increasingly being developed, yet no standardised benchmark exists to systematically evaluate them. This study introduces LessonBench-V1, a benchmark dataset comprising 647 human-written lessons paired with LLM-based reverse-engineered...
Why this label for LessonBench-V1: A Benchmark Dataset for Evaluating AI Lesson Generation Agents
- Technical test: Two independent AI graders gave the same level.
- Review status: Confirmed against the published method (AI-assisted review): both graders agreed
- Grader A: R4 real-learner gate: lesson-generation benchmark has no participating learners or learning outcome.
- Grader B: R4 real-learner gate: a benchmark dataset for lesson-generation agents with no learners
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review): both graders agreed Method
Technical testLearners: Simulated or AI-onlyOutcome: Model performancePreprintStudy type: Technical test (no real learners)
2026-05-08 · arXiv (back catalog) · Study (preprint)
Automated short answer scoring (ASAS) is shifting from discriminative, fine-tuned models to large language models (LLMs) used in few-shot settings. This paradigm leverages LLMs broad world knowledge and ease of deployment, but limited task-specific data may reduce alignment on complex scoring tasks.
Why this label for Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degra
- Technical test: Two independent AI graders gave the same level.
- Review status: Confirmed against the published method (AI-assisted review): both graders agreed
- Grader A: R4 real-learner gate: student answers are existing test data for automated scoring, not a real-learner arm.
- Grader B: R4 real-learner gate: automated scoring agreement on existing student responses; the outcome is model-human agreement, not learner outcomes
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review): both graders agreed Method
CommentaryProvisional (rules only)
2026-10-08 · arXiv physics.ed-ph · Study (preprint)
Students often perceive scientific experiments as clean, straightforward procedures with predictable outcomes. Students' beliefs about the nature of experimental physics typically remain inconsistent through the initial stages of undergraduate education.
Why this label for Measuring the Speed of Light with a Time-of-Flight Setup: An Authentic Experimen
- Commentary: provisional rules-only grade, pending grader consensus and review: R3 genre gate: No empirical learner data described; conceptual, framework, policy or argument paper.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryProvisional (rules only)
2026-09-30 · arXiv physics.ed-ph · Study (preprint)
Undergraduate research experiences provide students with an opportunity to apply knowledge learned in the classroom to authentic scientific problems. In this work, we describe the development of a special-topics course in Plasma Physics and Engineering (PPE) at a small liberal arts college using a Course-Based...
Why this label for Teaching a plasma physics and engineering class at a liberal arts college using
- Commentary: provisional rules-only grade, pending grader consensus and review: R3 genre gate: No empirical learner data described; conceptual, framework, policy or argument paper.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryLearners: Not reportedOutcome: Not reportedPreprintStudy type: Systematic review / meta-analysis
2026-09-24 · arXiv (back catalog) · Study (preprint)
Generative artificial intelligence has moved from a disruptive novelty to a recurring part of software-development and computing-education workflows, while software agents are beginning to act across repositories, command lines, browsers, tests, and other tools. The educational problem is no longer whether students...
Why this label for Judgment-Centred Software Engineering Education: A Post-Hype Review and Framewor
- Commentary: Two independent AI graders gave the same level.
- Review status: Confirmed against the published method (AI-assisted review): both graders agreed
- Grader A: R3 integrative narrative review and proposed education framework, with no systematic study synthesis described.
- Grader B: R3 genre gate: a structured integrative (narrative) review plus a conceptual framework, with no stated systematic search criteria or controlled-study synthesis.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review): both graders agreed Method
CommentaryLearners: Not reportedOutcome: Not reportedPreprintStudy type: Expert opinion / commentary
2026-04-26 · arXiv (back catalog) · Study (preprint)
Generative Artificial Intelligence (GenAI) presents a governance challenge for STEM assessment. Unrestricted access can enable task outsourcing that undermines the validity of traditional assessments, while blanket prohibitions are difficult to enforce, may drive use underground, and do little to prepare students for...
Why this label for Positioning Generative Artificial Intelligence in STEM Assessment: When to Requi
- Commentary: Two independent AI graders gave the same level.
- Review status: Confirmed against the published method (AI-assisted review): both graders agreed
- Grader A: R3 genre gate: the proposed STEM assessment framework is illustrated with examples, not tested with learners.
- Grader B: R3 genre gate: conceptual ECD-based governance framework for GenAI in STEM assessment, illustrated with physics examples, with no learner data.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review): both graders agreed Method
