AI Research Assistant: Use Cases (5/5)

I found the source. Here is my verification:

Source identification

The cited source matches the paper in your Zotero library:

• Title: Task-unrelated thought during educational activities: A meta-analysis of its occurrence and relationship with learning
• Authors: Wong, A. Y., Smith, S. L., McGrath, C. A., Flynn, L. E., & Mills, C.
• Year: 2022
• Journal: Contemporary Educational Psychology, 71, 102098
• Citekey: @wongTaskunrelatedThoughtEducational2022

Citation format check

“Wong et al., 2022” is correct. With five authors, APA 7 uses “Wong et al.” from the first citation onward. The year is correct.

Content / claim verification

Claim 1: “students engage in mind wandering approximately 30% of the time during learning activities”

This is accurate. The abstract states: “students are off-task about 30% of the time during educationally relevant activities.”

One small terminological note: the paper uses “task-unrelated thought” as its primary term (with mind wandering as a parenthetical gloss). If precision matters for your framing, you could note that the 30% figure specifically pertains to task-unrelated thought in educational contexts. But “mind wandering” is defensible given the paper explicitly equates the two in its title framing.

Claim 2: “These frequent episodes of mind wandering have been shown to negatively impact on learning outcomes”

This requires a correction. The meta-analysis found a negative correlation (r = −0.27, a small-to-medium effect), not evidence of causal impact. “Negatively impact on” implies causation that a meta-analysis of predominantly correlational studies cannot establish. Suggested rewording:

“These frequent episodes of mind wandering are associated with poorer learning outcomes (Wong et al., 2022).”

or, more precisely:

“These frequent episodes of mind wandering share a small-to-medium negative relationship with learning outcomes (Wong et al., 2022).”

This preserves the finding accurately without overclaiming causality.

I’ve now reviewed the draft conclusion against your published and under-review work. Here is my assessment.

Overall Verdict

The conclusion is broadly consistent with the trajectory of your previous work, but it makes several extensions and shifts that need to be flagged. The most significant tension concerns your stance on objective detection measures (eye-tracking), which your thesis and conceptual paper treat as a complementary research tool, while this conclusion frames their deployment in adaptive educational technologies as potentially harmful to learner agency. The shift is defensible, but it is a new argument, not a restatement of prior positions.

1. Points of Strong Alignment

Learner agency and goal-dependent appraisal

The claim that “the appraisal of whether a thought is aligned with their goals remains their own” is well-grounded in your prior work:

• Thesis conclusion (Ch. 8): “it is not the off-task thoughts that affect how well a learner is learning; it is the learner’s judgment concerning these off-task thoughts that impacts learning.”
• Thesis (Ch. 8, future research): “the influence of off-task thoughts on learning should not only be studied in relation to goals set by the researcher, but also in relation to goals set by learners.”
• Conceptual paper (Cognitive Science): The COPES framework positions the Evaluation of thoughts against learner-set Standards as the learner’s own metacognitive act.

The draft’s first paragraph faithfully extends this line of reasoning.

Normalising off-task thoughts

The call to “normalise their experience of off-task thoughts” aligns directly with:

• Thesis (Final Remarks): “Instead of seeing off-task thought as the bane of successful learning… it is viewed… as something that is outside of the learners control and thus needs to be accepted.”
• Conceptual paper: “It is normal for learners to experience TUTs, regardless of their self-regulation capabilities.”

Naturalistic setting as a research imperative

The draft’s call for studying mind wandering “in a naturalistic setting” is consistent with a core recommendation from the thesis: “research aiming to determine the relationship between SRL and off-task thoughts should be conducted in a naturalistic setting, as the transferability from experimental to naturalistic settings is not given.” The case study paper (Metacognition and Learning) was itself an argument for this.

Internalised refocusing

The concern about technologies that “absolve the learner of the need to develop their own… internalised refocusing skills” maps onto the thesis discussion of how self-regulated learners may “inherently refocus their thoughts back to the task through an internalised process whereby they are not explicitly aware of their off-task thoughts at all.” You are applying this concept to the technology question, which is a natural extension.

2. Points of Tension or Potential Misalignment

The stance on objective detection measures (eye-tracking)

This is the most significant area of tension. In your thesis and conceptual paper, eye-tracking and objective detection measures are framed positively as a research complement to self-caught thought reports:

• Thesis (Ch. 8, implications for theory): “Such an approach to measuring off-task thoughts could be achieved by using objective modalities for off-task thought detection (for an overview see @kuvarDetectingWhenMind2023), in combination with self-caught thought reports.”
• Thesis (Ch. 8, future research): Recommends “self-caught thought reports and objective measures of off-task thoughts [@kuvarDetectingWhenMind2023].”
• Case study paper (Limitations): “This limitation could be overcome by including an objective measure of off-task thoughts that does not rely on the participants’ self-reports, such as eye-tracking (Kuvar et al., 2023).”

The draft conclusion, by contrast, frames the transfer of detection technologies to adaptive educational technologies as a risk to autonomous self-regulation. This is not a contradiction per se: your prior work discusses eye-tracking as a measurement tool for research, while the draft discusses its deployment in real-time intervention systems. However, a reader familiar with your thesis would notice that you previously advocated for objective measures and are now raising concerns about their application. The distinction between measurement for research and real-time intervention in adaptive systems should be made explicit in the manuscript to avoid the appearance of inconsistency.

The eye-mind reader (Mills et al., 2021) as a cautionary example

The draft cites the eye-mind reader as an example of a technology that risks undermining autonomous self-regulation. However, the literature note for this paper (@millsEyeMindReaderIntelligent2021) describes it positively: the system “promotes long-term comprehension by detecting and responding to mind wandering” and “led to an improved delayed learning outcome.” The system prompts self-explanation upon detection, which is itself a generative learning activity.

There is a tension here: the eye-mind reader does not simply “make the decision for the learner.” It detects mind wandering and then prompts the learner to engage in a self-explanation, which is a metacognitive activity. The system does not pre-emptively decide whether the thought was congruent with the learner’s goals; it intervenes after detection. Your critique would be more precisely targeted at systems that classify thoughts as detrimental or beneficial on the learner’s behalf without involving the learner in that appraisal. The eye-mind reader, as described, actually engages the learner in a response. You may want to either sharpen the critique or choose a different example.

“Neutral appraisal” language

The draft calls for enabling “a neutral appraisal of whether a thought is helpful to their specific learning trajectory.” The word “neutral” is slightly at odds with your COPES-based framing, in which the appraisal is inherently subjective: the learner evaluates thoughts against their own Standards. “Neutral” might be read as implying an objective stance, which is the very thing you are arguing against. Consider whether “learner-centred appraisal” or “self-directed appraisal” would be more consistent with the theoretical framework you have been building.

3. New Claims That Extend but Do Not Contradict Prior Work

These arguments are new in your oeuvre. They are logical extensions but have not been substantiated in your prior publications:

1. “This transforms our role from external judges to facilitators” — The thesis discusses implications for practice in terms of scaffolding SRL skills and teaching reaction tactics, but does not frame the researcher’s or educator’s role as “facilitator” versus “external judge.” This is a new rhetorical positioning.

2. “These technologies may absolve the learner of the need to develop their own meta-awareness” — This is a novel claim. Your prior work discusses meta-awareness as a skill that learners develop through metacognitive monitoring cycles, but you have not previously argued that external detection systems could displace the development of this skill. The Molenaar (2022) HHAIR concept supports this argument (ALTs “take over (offload) regulation from learners”), but you are applying it to mind wandering detection specifically, which Molenaar does not address.

3. “This could lead the learner to become reliant on the system rather than their own regulatory capacity” — This is a dependency argument that is not made in your prior work. It is plausible and consistent with the HHAIR framework, but it is an empirical claim that would benefit from being framed as a hypothesis rather than a stated consequence.

4. Citation and Reference Concerns

• Molenaar (2022): This refers to “The concept of hybrid human-AI regulation” (Computers and Education: Artificial Intelligence, Vol. 3, 100070). It is in your Zotero library but has no literature note in the vault. The citation is appropriate for the argument about autonomous self-regulation, as Molenaar explicitly argues that ALTs offload regulation from learners. However, note that Molenaar’s concept is about gradual transfer of regulation from AI to human, not about abandoning AI support entirely. Your draft’s argument is stronger than Molenaar’s position: where she advocates for hybrid regulation with gradual transfer, you seem to caution against the system making the decision at all. Make sure you are not overextending Molenaar’s claim.

• Mills et al. (2021): As noted above, the eye-mind reader prompts self-explanation rather than making a pre-emptive decision for the learner. The characterisation in your draft may misrepresent the system’s actual function.

• Kuvar et al. (2023): This is correctly cited as a systematic review of mind wandering detection methods, consistent with how it is cited in your thesis and conceptual paper.

Summary of Recommendations

1. Clarify the measurement vs. intervention distinction. Your prior work endorses eye-tracking as a research measurement tool. The draft critiques its deployment in adaptive intervention systems. Make this distinction explicit so readers do not perceive inconsistency with your thesis.

2. Sharpen or replace the eye-mind reader example. The system prompts self-explanation, which engages the learner. If your critique targets systems that pre-emptively classify thoughts without learner involvement, the eye-mind reader is not the strongest example. Alternatively, reframe the critique to acknowledge that the eye-mind reader engages the learner but still bypasses the learner’s own detection of the thought.

3. Reconsider “neutral appraisal.” Your COPES framework positions appraisal as the learner’s subjective evaluation against their own Standards. “Neutral” may undercut this.

4. Flag the new claims as hypotheses, not established consequences. The arguments about absolving learners of meta-awareness development and creating system dependency are extensions of your prior work, not substantiated by it. Framing them as risks or hypotheses would be more consistent with the calibrated hedging in your other publications.

5. Consider creating a literature note for Molenaar (2022) in the vault, given that you are now citing it in manuscript drafts.

I now have a comprehensive picture of all peer reviews Daniel has received across his publications and the current JCAL v2 draft. Let me compile the analysis.

Cross-Review Analysis: Recurring Critiques Applied to the JCAL v2 Draft

I have reviewed the v2 draft against all peer reviews you have received across the following submissions:

• Mind Wandering Reactions Case Study (LAK25, then Metacognition and Learning, rounds 1 and 2)
• SRL and Mind Wandering Conceptual Paper (Metacognition and Learning, then Cognitive Science)
• TRI Meta-Analysis (Review of Educational Research rounds 1 and 2, Psychonomic Bulletin & Review, Contemporary Educational Psychology)
• SmartQuiz Experiment (AIED 2026)
• ECTEL 2024 Doctoral Consortium
• LAK 2024 Doctoral Consortium
• HERGA 2023 Poster

Below are the critiques from these previous reviews that also apply to the current JCAL v2 draft, ordered by severity (likelihood of recurrence and impact). Each entry identifies the source review(s), the specific issue in the v2 draft, and a concrete suggestion.

1. Theoretical model-measurement misalignment (COPES vs. self-report SRL-O)

Source reviews: LAK25 meta-review (Reviewer 1 + meta-reviewer); Metacognition and Learning Reviewer 2 (Comment 10).

Previous critique: The COPES model (Winne) conceptualises SRL as a series of dynamic, task-level events. Winne has advocated for decades for trace-based measures. Using a self-report aptitude questionnaire (SRL-O) to operationalise COPES constructs is a fundamental theory-measurement mismatch. The LAK25 meta-reviewer stated explicitly that this cannot be resolved by arguing self-reports are acceptable for meta-awareness; the entire theoretical model would need to be changed (e.g., to Zimmerman’s) or the measurement approach changed to traces.

How it applies to the JCAL v2: Section 2.6 draws on the COPES model as the theoretical framework, and Section 4.4 uses the SRL-O self-report questionnaire to measure SRL aptitude. The same mismatch exists. The JCAL v2 does not address or even acknowledge this tension. A reviewer familiar with Winne’s work (or the LAK25 meta-reviewer, if they review for JCAL) would flag this immediately.

Suggestion: Add an explicit justification for why a self-report aptitude measure is acceptable within the COPES framework for this specific research question. The argument should be that RQ2 examines the relationship between SRL aptitude (a trait-like characteristic) and meta-awareness frequency (also trait-adjacent), not SRL events. Alternatively, reframe the theoretical framework to use a model that is more compatible with aptitude measures (e.g., Zimmerman’s three-phase model or Pintrich’s framework), or acknowledge the mismatch in the limitations section. Do not leave it unaddressed; the LAK25 meta-reviewer considered this a fatal flaw.

2. No reliability estimates reported for SRL-O subscales in the current sample

Source review: Metacognition and Learning Reviewer 3 (Comment 23).

Previous critique: The reviewer noted that the manuscript reported external Cronbach’s alpha benchmarks (0.76-0.88 from prior studies) but placed the current sample’s reliability estimates in a supplementary table. The reviewer called this “methodologically problematic” because reliability is sample-dependent and external benchmarks should not substitute for transparent reporting in the present dataset.

How it applies to the JCAL v2: The v2 draft (Section 4.4) describes the SRL-O subscales used and cites Broadbent et al. (2022) but does not report Cronbach’s alpha (or equivalent) for the Metacognition, Online Effort Regulation, or Online Task Strategies subscales in the current sample of 138 participants. This is exactly the same omission that was previously flagged.

Suggestion: Report internal consistency estimates (Cronbach’s alpha or McDonald’s omega) for each SRL-O subscale in the current sample, either in the Procedure section or in a table. This is a straightforward addition that pre-empts the critique entirely.

3. Causal language for correlational findings

Source reviews: Metacognition and Learning Reviewer 3 (Comment 25: “conflates correlation with causation and overgeneralizes from the findings”); RER Reviewer 1 (Comment 15: “causal language is not warranted given the non-experimental nature of the data”); Psychonomic Bulletin & Review Reviewer 3 (Point 4: “educational implications overgeneralize from lab studies”).

How it applies to the JCAL v2: Several passages use directional or causal language for correlational findings:

• Abstract: “indicating that meta-aware learners perform better on subsequent knowledge tests” implies that meta-awareness causes better performance.
• Lay summary: “Students who reported being better at self-regulating their learning reported fewer unrelated thoughts” (this one is correctly hedged).
• Conclusion: “only if learners are unaware of their off-task thoughts does it negatively affect learning” implies causation from a correlational result.
• Discussion (Section 6): “an increase on the scale corresponded to a decrease in the number of thought reports” is acceptable, but “meta-aware learners perform better” is not.

RQ2 and RQ3 are correlational/exploratory, not experimental. RQ1 is experimental but produced null results. The positive correlation between thought reports and knowledge gain (tau = .15) is a weak effect and should not be interpreted as evidence that meta-awareness improves learning.

Suggestion: Audit the abstract, discussion, and conclusion for causal language. Replace “meta-aware learners perform better” with “meta-awareness was positively associated with knowledge test performance.” Replace “only if learners are unaware… does it negatively affect learning” with “the negative association between off-task thoughts and learning may be limited to instances where learners lack meta-awareness.” Add a sentence in the discussion explicitly noting the correlational nature of RQ2 and RQ3 findings and that no causal claims can be made.

4. “So what” / practical implications underdeveloped

Source reviews: JCAL Reviewer 2 (Comment 16: “biggest weakness is the lack of clear answers to the ‘so what’ question”); Cognitive Science Reviewer 1 (Comment 4: “insufficient discussion of how educators might leverage this understanding”); RER Reviewer 1 (Comment 16: “I’d like to see more recommendations for practice, and not just future research”).

How it applies to the JCAL v2: The practical implications paragraph in Section 7 is brief and essentially says: these approaches do not work for short videos, so consider video length, and encourage note-taking. The conclusion focuses on theoretical implications but the practical takeaway is thin. The main novel finding (positive correlation between thought reports and learning) is not translated into a concrete actionable recommendation. The lay summary says “Teaching students awareness of unrelated thoughts could result in more efficient learning sessions” but this is not elaborated in the discussion.

Suggestion: Expand the practical implications to address: (a) what should educators do given that neither interpolated testing nor self-explanation improved outcomes for short videos? (b) what does the positive correlation between meta-awareness and learning suggest for instructional design? (c) what specific strategies could be used to foster meta-awareness (e.g., brief reflection prompts, awareness training)? Connect these to the Randall et al. (2022) reference already cited, which studied self-regulatory strategies to reduce mind wandering.

5. Inter-rater reliability (kappa = 0.47) is moderate and could be questioned

Source reviews: RER Reviewer 1 (Comment 10: “Cohen’s Kappa values were not particularly high”); RER Reviewer 2 (Comment 3: “low inter-rater-reliability has implications for the overall reliability of the screening process”).

How it applies to the JCAL v2: Section 4.5 reports an initial Cohen’s kappa of 0.47, which is moderate agreement. The draft states that all disagreements were discussed until 100% agreement was reached. However, a reviewer could question whether the initial moderate agreement indicates that the coding scheme itself is ambiguous, and whether the resolution process (discussion to consensus) introduced coder drift or bias. This is especially relevant because the thought type categories (TUT, TRI, ED, on-task) involve subjective judgment.

Suggestion: Add a brief justification for the moderate kappa, noting that thought report coding involves inherent subjectivity in interpreting free-text responses. Report the number of thought reports coded and the number of disagreements. Consider reporting kappa after the first round of discussion (not just the initial value) to show improvement. If possible, report a final kappa on a held-out subset to demonstrate post-consensus reliability.

6. TRI category conflates metacognitive monitoring with interference

Source reviews: RER Reviewer 2 (Comment 1: “TRI definition has a large overlap with metacognition”); Psychonomic Bulletin & Review Reviewer 3 (Point 1: “the distinction between TRI vs. metacognition seems blurred”); Contemporary Educational Psychology Reviewer 1 (Major Concern 1: “Labeling such processes as ‘interference’ is problematic. It conflates potentially beneficial cognitive activities with genuinely distracting thoughts”).

How it applies to the JCAL v2: Section 4.5 defines TRI as “a thought about the task that represented an interruption in the flow of the task (e.g., thinking about the task’s difficulty, questioning understanding).” “Questioning understanding” is a metacognitive monitoring judgement, not interference. This is the exact conflation that was flagged across three separate submissions of the TRI meta-analysis. A reviewer who has encountered this critique before (or who is familiar with the metacognition literature) would flag it.

Suggestion: Refine the TRI coding definition to distinguish between metacognitive monitoring (which is on-task and beneficial) and genuine task-related interference (thoughts that interrupt the flow without contributing to learning). Consider whether “questioning understanding” should be coded as on-task rather than TRI. Add a sentence acknowledging the conceptual overlap and explaining the coding decision.

7. Data exclusion and outlier handling

Source reviews: AIED 2026 (Review 2 and 4: “excluding participants based on outcome variables risks introducing bias”; “exclusions appear uneven across conditions”); RER Reviewer 1 (Comment 12: “more information on outlier criteria”); Contemporary Educational Psychology Reviewer 2 (Point 5: “outlier detection should be based on the model, not descriptive evaluation”).

How it applies to the JCAL v2: Section 4.7 excludes 42 participants for incomplete data, then applies outlier detection using “the standard deviation method, a cutoff of two standard deviations, and the interquartile range method” simultaneously, excluding all identified outliers. The resulting group sizes are uneven (47, 51, 40). The use of multiple simultaneous outlier criteria is unusually conservative and the rationale is not provided. The uneven group distribution after exclusion could introduce bias.

Suggestion: Justify the use of multiple outlier detection methods. Report how many participants were identified as outliers by each method and how many were excluded in total due to outliers (as distinct from the 42 who did not complete the study). Consider reporting sensitivity analyses with and without outliers to show robustness. Address the uneven group sizes in the limitations.

8. No figures for results presentation

Source reviews: Metacognition and Learning Reviewer 1 (Comment 8: “provide at least 1 or 2 figures for your primary results rather than 10+ large tables of stats”); Psychonomic Bulletin & Review Reviewer 1 (Point 4: “forest plots should be in main text, not supplementary”).

How it applies to the JCAL v2: The results section (Section 5) presents all findings in tables (Tables 1-5). Figure 1 is a procedure diagram. There are no figures visualising the primary results (e.g., group comparisons, SRL-thought report relationships, correlation scatter plots). This was previously flagged as a readability issue.

Suggestion: Add at least one figure visualising the primary result of interest. For RQ1, a bar plot of thought reports by group with error bars. For RQ2, a coefficient plot showing the GLM estimates for each SRL-O subscale. For RQ3, a scatter plot of thought reports vs. knowledge gain. This improves readability for a journal audience and pre-empts the critique.

9. Terminology density / jargon in abstract and introduction

Source reviews: Metacognition and Learning Reviewer 1 (Comment 1: “results discussed in the abstract were quite jargony”); Metacognition and Learning Reviewer 3 (Comment 20: “presupposes reader familiarity with domain-specific terminology”); ECTEL 2024 DC Reviewer 2: “significant overlap of complex SRL terms, but they are not all well-addressed.”

How it applies to the JCAL v2: The abstract uses terms like “interpolated testing,” “self-caught thought reports,” “meta-awareness,” “metacognitive monitoring,” and “online task strategies subscales” without definition. The lay summary is more accessible, but the structured abstract itself remains jargon-heavy. JCAL has an interdisciplinary audience (education + technology), not all of whom will be familiar with mind wandering terminology.

Suggestion: Add brief parenthetical definitions for key terms in the abstract (e.g., “self-caught thought reports (participants self-report off-task thoughts as they notice them)”). Ensure that the first use of each acronym in the introduction is accompanied by a definition. The lay summary already helps here, but the structured abstract should be independently accessible.

10. Individual differences in meta-awareness ability not addressed

Source review: Cognitive Science Reviewer 1 (Comment 2: “the model assumes relatively uniform ability to detect TUTs; needs to address how working memory capacity, executive function, and attention control might moderate the SRL-TUT relationship”).

How it applies to the JCAL v2: The JCAL draft assumes that all participants have equivalent capacity to become meta-aware of their off-task thoughts. The finding that Metacognition subscale scores negatively predict thought reports (the opposite direction of what the scale ostensibly measures) could partly be explained by individual differences in meta-awareness ability. The draft discusses this in terms of “metacognition without awareness” (Reder, 1996) but does not address stable individual differences in the capacity for meta-awareness.

Suggestion: Add a brief discussion in the limitations or discussion section noting that individuals vary in their capacity for meta-awareness (citing Grabman & Dodson, 2024, which is already in the Cognitive Science version). Acknowledge that this variability could partly explain the unexpected direction of the Metacognition subscale results. This is a low-effort addition that pre-empts a critique you have already received.

11. Video duration and ecological validity

Source reviews: JCAL Reviewer 2 (Comment 11: “intervention was very brief… inappropriate to expect these skills to change”); Psychonomic Bulletin & Review Reviewer 3 (Point 4: “educational implications overgeneralize from lab studies”); JCAL Reviewer 1: “more control of variables such as video content.”

How it applies to the JCAL v2: This was already raised by JCAL’s own Reviewer 2 in round 1 and is partially addressed in the limitations (Section 7). However, the practical implications paragraph still generalises from a single 8-minute video to recommendations for educators. The null findings for RQ1 could be entirely attributable to the video being too short for mind wandering to accumulate, which undermines the practical recommendation that interpolated testing and self-explanation are ineffective.

Suggestion: More carefully qualify the practical recommendations to specify that they apply to short videos only. State explicitly that the 8-minute video limits the generalisability of the null RQ1 findings and that the effectiveness of these interventions may differ for longer videos. This is already partially present but should be tightened.

Summary Table

The most critical item is #1 (COPES-measurement misalignment). This was considered a fatal flaw by the LAK25 meta-reviewer and it remains unaddressed in the JCAL v2. I would prioritise addressing this before the next review round, as a reviewer familiar with Winne’s work will almost certainly raise it again.