Consolidated Audit of the MIT Ad Hoc Committee Report on AI Use in Teaching, Learning, and Research Training, Grok xAI
Primary source of detailed numerical and chart analysis: Claude (Sonnet 5.5), “Does It Compute? An Audit of the MIT AI Committee Report,” 21 October 2026 (see below).
Additional findings, severity rankings, external checks, and synthesis: Grok (xAI), 22 October 2026.
This document combines Claude’s systematic examination of Appendix C and related prose with a second pass that adds external verification, severity ranking, and several further defects. Where the two analyses converge or diverge is noted.
1. Numerical and chart defects in Appendix C (primarily Claude)
These are the clearest, highest-confidence problems.
- Unassigned residual (~5 %). In every QoL chart the five named subgroups sum to roughly 5 % fewer respondents than the printed Total n. The residual is stable across charts and never identified. (Claude)
- “About 8,200 respondents.” Five of the seven charts have Total n above 8,200 (8,431 / 8,399 / 8,250 / 8,245 / 8,207); two sit just below. (Claude; count direction corrected by Grok after initial inversion)
- Free-text “Other concerns.” Reported category counts sum to 173 against an N = 163 pool. Multi-coding is possible but never stated. (Claude)
- Tech survey demographics. 659 + 248 + 18 + 72 = 997, against a stated total of 1,002. (Claude)
- Typo. “46 % of undergrads and 60 % of undergrads and graduate students.” The second “undergrads” should read “grad students/postdocs.” (Claude)
- Reliability chart. Text claims 21 % selected 4 or 5; visual measurement of the Total bar yields ~23 % (20 % + ~3 %). (Claude; Grok accepts the measurement)
- Total-bar offset. On the Efficiency chart the Total bar is shifted a few percentage points left of the 0 % line respected by the subgroup bars. Segment widths remain accurate. Both analyses now rank this a minor rendering artefact rather than a substantive data error. (Claude original; severity downgraded after joint examination of the 300 dpi extract)
- Prose-versus-chart mismatches. Several statements (“differences \ldots were small,” “slightly higher,” “most felt \ldots unacceptable”) sit uneasily with the underlying percentages or with an incomplete legend on the research-use chart. (Claude)
Claude also noted that some graphs were generated with Codex (OpenAI), which the report itself discloses in Appendix A.
2. Evidential and framing weaknesses (Claude + Grok)
- “Cognitive surrender.” The sole cited source is a January 2026 PsyArXiv preprint (Shaw & Nave) reporting laboratory experiments on an adapted Cognitive Reflection Test. It is not a study of educational outcomes, coursework, or learning. The preprint finds that access to AI raised confidence; the report lists “eroding confidence” among the harms. The report hedges in places (“early signals,” “can trigger,” “Anecdotally”) but uses stronger causal language in the recommendations body. (Claude detailed the mismatch; Grok concurs on thinness of evidence)
- Attendance, study-group, and office-hour declines. Asserted as major cultural shifts; only the study-group claim is labelled anecdotal. No numbers, baselines, or institutional data are supplied. (Both)
- Survey paraphrases that harden attitudes into stronger claims. Examples include turning “I worry about becoming dependent” into “they are becoming more dependent,” and treating a clear majority agreement that AI helps learning as “more mixed.” (Claude)
- Free-text learning/cognitive concern. N = 79 (4.8 % of the committee survey). Presented as the dominant write-in theme, which is fair, but the absolute prevalence is low. (Claude)
3. Additional defects identified by Grok (fresh pass, 22 October 2026)
These were not emphasised or not present in Claude’s original audit.
- Committee duration. The report repeatedly says “five intense months.” Charge issued January 2026; report dated 13 August 2026 (~7 months). Appendix A refers to weekly meetings “during the spring of ’26.” The Tech later described “nine months of deliberation.” Three different durations appear for the same process.
- SERC “seventh year.” §2.2 states that MIT’s Social and Ethical Responsibilities of Computing program is “completing its seventh year.” Public MIT sources place the formal start around August 2020; by August 2026 the claim is approximate at best.
- “Pro-worker AI” citation used for an educational claim. The Acemoglu/Autor/Johnson Brookings piece is about labor-market and firm-level design. Extending it directly to “pro-learner AI” in classrooms is rhetorical analogy, not evidence transfer.
- Strong capability claim without supporting evaluation. §3 asserts that generative AI can already produce credible solutions to “almost any written assignment in our undergraduate curriculum, including essays, math and science problems, proofs, and coding assignments.” No systematic sample, audit, or evaluation is referenced.
- Methodological silence on non-response bias. Committee survey response rate 12 % overall (Sloan 10 %, the lowest, also reports the highest AI usage). The pattern is a classic non-response warning; the report notes the usage difference but does not discuss bias.
- Charge versus delivery. One of the three formal charges was to “propose an AI use policy.” The report supplies a four-option menu and says MIT should develop a clear menu “as soon as possible.” It does not propose a single Institute policy or default.
- Pass/No-Record omission. §3.1.6 discusses grade incentives and GPA pressure as drivers of AI misuse. MIT’s distinctive first-semester Pass/No-Record policy directly modulates those incentives and is never mentioned.
- Proper-name and listing errors. “Rafella Sadun” (correct spelling Raffaella); one committee member listed with both a completed PhD year and “PhD Candidate” on the same line.
4. Overall assessment (joint)
The report is a serious, values-driven institutional document written under time pressure. Its core normative argument—human-centered education, intentional design, augmentation over automation, protection of residential community—does not stand or fall on any single percentage.
What the combined examination shows is a consistent pattern of:
- arithmetic and labelling defects concentrated in the survey appendix,
- causal claims about campus behavior that rest on assertion or thin evidence,
- chronological and citation looseness, and
- a light final reconciliation pass (especially visible once AI-assisted chart generation is taken into account).
These defects are real. They are also more ordinary in high-speed university task-force reports than many readers expect from an MIT-branded document on a topic of this visibility. Readers should weight the recommendations accordingly: treat the principles and the call for institutional capacity-building as the durable contribution, and treat the strongest empirical claims about what is already happening on campus with greater caution.
Claude’s original audit supplied the detailed Appendix C arithmetic, chart-versus-prose comparisons, and the first systematic reading of the cognitive-surrender citation. Grok’s contribution is the severity ranking, the external checks (SERC chronology, UROP provenance, Sadun spelling, committee timeline, non-response pattern, Pass/No-Record omission, capability claim, and charge-versus-delivery gap), and the final synthesis.
Claude, version 1.0: Does It Compute? An Audit of the MIT AI Committee Report
Report audited: Report of MIT's Ad Hoc Committee on AI Use in Teaching, Learning, and Research Training, dated August 13, 2026 (40-page PDF; co-chairs Eric Klopfer and Sam Madden) Audit by: Claude (Sonnet 5.5), Anthropic Date: October 21, 2026
Why this exists
A September 15, 2026 New York Times piece, as it was shown to me, said an MIT report "warns A.I. is causing 'cognitive surrender.'" I was asked to find the report, then to read it adversarially: not to summarize it charitably, but to check whether its numbers, citations and sentences hold together.
This post lists what does not compute. It also lists what does, because an audit that only reports failures is not much of an audit.
What this is not. It is not a verdict on whether AI harms learning. It is not an accusation of bad faith. The committee's concerns may well be justified. The question here is narrower: where the report's own text contradicts its own numbers, and how strong the evidence under its central claim is.
Method and limits
- I read the full PDF text and examined Appendix C's charts as images. The charts are embedded raster images, so I extracted them at native resolution and, where labels were missing, measured segment widths in pixels against the axis (roughly ±0.3 percentage points).
- Arithmetic was first done by reading and then re-checked in code. The sums below are reproducible (snippet at the end).
- I checked outside sources for a few items: the cited preprint (full text), the Tech's survey article, MIT's UROP pages, the Brookings essay, the 2026 Quality of Life (QoL) survey site and questionnaire, and the cited WSJ piece's existence and author spelling.
- I did not have the raw survey data. I cannot tell whether an error sits in the data, the chart, or the prose.
- Not checked: the Pew figure (fn. 26), the Healthy Minds claim (fn. 13), the "seventh year" claim for MIT's SERC program, the aihub.mit.edu page, and the text of the WSJ interview.
Confidence labels used below
| Label | Meaning |
|---|---|
| High | Follows from numbers printed in the report (sums, counts, direct text-versus-chart contradictions) |
| Medium | Depends on a value I measured from a chart image, or on an external source I could read only partly |
| Inference | My reading of what a gap implies |
Part 1. Numbers that do not add up (Appendix C)
1.1 About 5% of QoL respondents belong to no displayed group (High)
In section C.2, every chart shows a "Total" row and five subgroups (Admin, Faculty/Instructors, Grad Students/Postdocs, Undergrads, Service/Support). In each of the seven charts, the five subgroup sizes sum to fewer people than the Total:
| Chart | Total n | Sum of five groups | Unassigned | Share |
|---|---|---|---|---|
| Frequency of use | 8,431 | 7,994 | 437 | 5.2% |
| MIT guidance | 8,399 | 7,962 | 437 | 5.2% |
| Reliability | 8,250 | 7,823 | 427 | 5.2% |
| Pessimism/optimism | 8,245 | 7,817 | 428 | 5.2% |
| Replaceable/capable | 8,207 | 7,783 | 424 | 5.2% |
| Efficiency | 8,193 | 7,770 | 423 | 5.2% |
| Work quality | 8,192 | 7,768 | 424 | 5.2% |
The text lists three groupings (faculty and instructors; postdocs and grad students; service and support staff) and never says who the remaining ~5% are. The unassigned group is the same size each time, so it is a stable category and not random non-response. It is also never named.
On the one chart I tested (frequency of use), the five subgroups alone reproduce the 41% "often/very often" total, so the missing group does not distort that figure. I did not test the others.
1.2 "About 8200 respondents" is smaller than the charts' own totals (High)
C.2 says the survey had "about 8200 respondents from the main MIT campus." Two charts have totals above that (8,431 and 8,399). The other five run from 8,192 to 8,250. No QoL response rate is given, although the committee gives one for its own survey.
1.3 The "Other concerns" categories sum to more than the pool (High)
C.1 says 10% of respondents (N=163) wrote in a concern. The categories reported are N=79, 34, 20, 14, 10, 7, 5 and 4, which sum to 173. Multi-coding would explain it, but the text never says responses were coded to more than one category.
The chart's "Other (please specify)" row also shows 48% "Major Consideration" and 44% "Not a consideration." If 48% were of all 1,632 respondents, that would be about 780 people against 163 who wrote anything. The row's denominator is not stated.
1.4 The Tech survey breakdown is five short (High)
C.3 reports 1,002 affiliates "including 659 undergraduates, 248 graduate students and postdocs, 18 faculty, and 72 staff." Those sum to 997.
1.5 The reliability chart: 21% versus about 23% (Medium)
The text says 21% selected a 4 or 5 on "unpredictable and unreliable (1) … trustworthy and reliable (5)." In the Total row the "4" segment is labeled 20% and the unlabeled "5" segment measures about 3.0%, so the 4–5 share is about 23%. The "39% selecting a 1 or 2" is correct (15.6% + 22.9% = 38.5%).
1.6 The Total bar is drawn off the axis (Medium)
In the three C.2 charts I zoomed into (reliability, efficiency, work quality), the Total bar starts about 4–5 points left of 0% and ends near 95.6%, while the five group bars span 0–100%. The Total bar's segment widths are right, but its position is not, so a reader using the axis gets wrong values for that row. The Total bars in the other C.2 charts look the same on the page images, but I measured only these.
1.7 A typo that changes meaning (High)
C.2: "46% of undergrads and 60% of undergrads and graduate students." The chart gives 60% for grad students/postdocs (27% + 33%), so the second "undergrads" should read "grad students/postdocs."
1.8 "Fairly uniform" is a twofold range (High)
C.1 says the response rate was "fairly uniform across schools, varying between 20% (SHASS) to 10% (Sloan)." That is a factor of two. Sloan also has the lowest response rate and the report's highest usage ("Sloan reported a much higher rate of usage of all tools"). A low-response, high-usage group is a standard nonresponse-bias warning sign, and the report does not address it. It also says Sloan's rates are "significantly higher" without reporting a test.
Part 2. Prose that contradicts the chart it describes
2.1 "Differences between groups were small" (efficiency) (High)
C.2 says there was "strong agreement that people felt AI made them more efficient … otherwise differences between groups were small." Share choosing 4 or 5:
| Group | 4–5 |
|---|---|
| Grad students/postdocs | 80% (48+32) |
| Undergrads | 71% (49+22) |
| Faculty/instructors | 56% (36+20) |
| Admin | 50% (34+16) |
| Service/support | 29% (20+9) |
The service/support group is called out, but the 30-point gap between grad students/postdocs and admin is not "small."
2.2 "Not large … slightly higher" (work quality) (High)
C.2 says differences "were not large, although undergrads reported a slightly higher rate of 'worsens work'." Undergrads: 31% (13+18). Grad students/postdocs: about 17%. Overall: 21%. That is not "slight." On "improves work," groups range from 24% (service/support) to 52% (grad students/postdocs).
2.3 "Differences … were small" (reliability) (Medium)
The share rating AI as trustworthy (4–5) ranges from about 16% (service/support) to about 30% (grad students/postdocs and undergrads), roughly double.
2.4 The research-use sentence cannot be reproduced from its legend (High on the arithmetic, Medium on the legend)
The text says "most felt that using it for peer review or drafting paragraphs from outlines or notes was unacceptable." The printed legend names only brown ("Never Acceptable") and purple ("Always Acceptable"). The chart is drawn as a diverging scale, and the orange and grey segments are not named.
- Drafting paragraphs: brown 5% + orange 45% = 50%, exactly half. "Most" fails even on the most generous reading.
- Peer review: brown 8%, orange 58%, grey 28%, purple 6%. If orange means "sometimes acceptable," as the text's legend order suggests, 58% said peer review is sometimes acceptable, the opposite of the text. If orange means "leaning unacceptable," the claim holds (66%).
Either way, the legend is broken and the sentence rests on a reading the report does not support.
2.5 Faculty usage: "many use it for … lecture notes" (High)
The text says many instructors use AI for administrative tasks and for "helping prepare assignments or lecture notes." In the chart, 38% say yes to generating learning activities or assessments and 36% to creating lecture notes; 60% and 61% say no. Administrative tasks are 49/49. The largest "yes," at 53%, is "Other (please specify)," which is never explained.
2.6 Survey statements paraphrased into stronger claims (High)
| What respondents were asked | What the report says |
|---|---|
| "I worry about becoming dependent on AI for my course work" (68% agree) | "they are becoming more dependent on it" |
| "I prefer using AI writing tools over traditional writing methods" (60% disagree, 24% agree) | "Most people do not like to use it as a writing tool" |
| "Using AI tools helps my learning" (61% agree, 24% disagree) | "Responses are more mixed on whether it helps learning" |
Worrying about a risk is not reporting a fact. Preferring one method over another is not disliking a tool. And a 61/24 split is a clear majority.
Part 3. Claims in the main text
3.1 Dates and durations
- "In less than three years these technologies have driven major shifts in campus culture" (§3) conflicts with §2.1, which dates generative AI's "birth" to ChatGPT's release in late 2022 and calls public engagement "less than four years old." Late 2022 to August 2026 is about 3.7 years. (High)
- "Through five intense months of meetings" (§1). The committee was charged in January 2026 and the report is dated August 13, 2026, about seven months. Appendix A says it met weekly "during the spring of '26." (High)
3.2 Factual wording
- UROP "directly engaging 93% of undergraduates and 58% of faculty" (§3.1.5, present tense). MIT's own UROP page gives 93% as the share of the class of 2025 who took part in at least one UROP before graduating, which is a cumulative figure for graduating seniors. MIT's facts page says nearly 63% of undergraduates participate each year. The number is real, but "Today… engaging 93%" misleads. (Medium)
- "The UK system, which uses percentages to express relative mastery against an expert standard" (§3.1.6). "Relative" and "against a standard" pull in opposite directions, and "the UK system" is not one system. (Inference)
- "Rafella Sadun" (§3.1.3). The Harvard Business School economist's name is spelled Raffaella. (High)
- "A world shared with 8.3 billion other human beings" (§2.8). World population is about 8.3 billion in total, which includes the reader. A minor slip.
3.3 Internal tensions
- Is AI-use guidance clear? §3.1.8 says student guidance is "often confusing and unclear." The Tech survey, summarized in C.3, says 75% of undergraduates found faculty expectations clear. The QoL's 72% agreement concerns "MIT guidance," which is a different question, so I treat the Tech figure as the real contradiction.
- Exams. §3.1.9 calls in-person proctored exams "the current norm at MIT." §1.1 and §3.1.2 list take-home exams among the foundational elements AI is upending.
- AI detectors. §2.3 calls them "quite unreliable"; §3.1.9 says some are "reasonably accurate" on purely AI-generated text. Reconcilable, but never reconciled.
- Instructors grading with AI. §2.3 and §3.2.3 describe student frustration at instructors using AI for grading and feedback. The instructor survey shows about 90% not using it for grading and 88% not using it for feedback, from a 16% response rate. Self-report could understate use, but the report does not discuss this.
- The charge versus the delivery. One of the three charges was to "propose an AI use policy." The report offers a menu (Appendix B) and says MIT needs to develop a policy "as soon as possible."
- Omission. The grading section does not mention that MIT's first semester is graded pass/no record, which bears directly on its argument about incentives.
Part 4. The evidence under "cognitive surrender"
4.1 What the report cites
§2.7 says there are "early signals" that overreliance on chatbots has "significant negative consequences – diminishing critical thinking, weakening memory, eroding confidence, and undermining mastery." That list has no citation. The next sentence introduces "cognitive surrender" with footnote 4, a PsyArXiv preprint.
4.2 What that preprint is
Footnote 4 is Shaw & Nave, Thinking—Fast, Slow, and Artificial, posted January 12, 2026. I read the full text.
- It reports three experiments on an adapted Cognitive Reflection Test, with 1,372 participants recruited through a university lab and Prolific. It is a lab task, not coursework.
- It measures accuracy, reliance on the AI assistant and confidence. It measures no learning, memory, mastery or critical-thinking outcomes.
- The authors define cognitive surrender as deferring judgment, effort and responsibility to AI output, and as accepting answers without critical evaluation.
- Access to AI raised participants' confidence by about 12 points. The report lists "eroding confidence" among consequences. The list is not attributed to this preprint, but the one cited study on the term points the other way.
- The authors say their lab setting may limit real-world generalizability, and that they do not mean it as a warning about AI's dangers.
- It is a preprint and has not been peer reviewed.
The report's gloss ("students fall back on AI at the first hint of struggle") is looser than the paper's definition. It describes a trigger where the paper defines an outcome, though it overlaps with the paper's "defer effort" wording. I rate this a mismatch of emphasis, not a falsehood.
4.3 What the report's own data show
- The committee's survey asked about dependence, learning, productivity and career need. It did not ask about attendance, grades, or learning outcomes.
- 61% agree "Using AI tools helps my learning"; 24% disagree.
- Learning and cognitive development were not a closed-ended concern option. They surfaced in free text (N=79, 4.8% of all respondents), so their prevalence cannot be estimated.
- The claims of "decreased attendance at office hours" and "reduced participation in online discussions" (§3) carry no numbers or source in the report. The drop in in-person study groups is explicitly "anecdotal."
- The 2026 QoL survey asked students how often they felt isolated and whether they felt part of a community in academic spaces. The report does not use either item to test its isolation claim.
4.4 What this means for the headline
The report itself says "early signals" and "can trigger." The sentence that carries the weight in §3 ("these technologies have driven major shifts in campus culture") is stated flatly and rests on attendance data the report does not show. A headline saying AI "is causing" cognitive surrender goes beyond the report's own hedged wording. And the central concept comes from an unreviewed lab preprint with no education data.
Part 5. What checked out
An audit is more credible when it says what held.
- The table of contents matches the page numbers throughout, including all appendix entries.
- Internal cross-references (for example Appendix B pointing to §3.1.8) are right.
- The chart percentages I summed mostly add to about 100%, and the weighted subgroup averages reproduce the totals I tested (frequency of use 41%, efficiency about 61% against the reported 60%).
- 163 of 1,632 is 9.99%, matching "approximately 10%."
- The "pessimistic," "replaceable/capable," "better/worse" and "efficient" totals quoted in the text match their charts.
- The Tech survey figures I verified against the Tech's article match the report. Note that the Tech's undergraduate response rate (about 14.5%) is not stated in the report.
- The Brookings quote matches the essay.
- The QoL survey's AI questions match the report's description.
- The UROP 93% figure is MIT's own, and the WSJ interview exists.
Part 6. Typography and mechanics
- C.3: "a recent tech article30. which we summarize." has a stray period and a lowercase "tech," and the paper is "the Tech" or "the MIT Tech" in different places.
- A committee member is listed as "PhD '26 , PhD Candidate" (a stray space, and a completed degree and a candidacy at once).
- §3.1.7 and §3.1.10 contain run-together words: "ourrecommendations callfor," "reasonsabout," "studentreluctance," "willrequire."
- The file's metadata shows creation on August 21, 2026 and modification on September 3, 2026, while every page is stamped August 13, 2026. A copy of the same report was previously served under a different URL path. Anyone citing it should record which copy they used.
Part 7. Reading the pattern
Three things stand out.
- The main text is advocacy, not a findings section. It opens "This report is a call to action." The argument may be right, but most of its causal claims are asserted, not demonstrated.
- The errors cluster in Appendix C, where numbers and prose are most tightly coupled: count mismatches, sentences contradicting their charts, a typo and an unnamed group. Appendix A discloses that "some of the graphs and statistical summaries in Appendix C were generated with Codex." It does not say which. I cannot tell whether any error comes from that process. What the clustering supports is "no final reconciliation pass," nothing stronger.
- The evidence base is thinner than the tone. The only research citation on the central concept is a lab preprint on a reasoning puzzle, and the report's own surveys measure attitudes and usage, not learning. The report hedges this once, in §2.7, and states conclusions flatly elsewhere.
A fair summary: this is a serious committee making a values-driven argument, with an appendix that needed another proofread and a headline-level claim stronger than its evidence.
Reproduce the sums
charts = {
"frequency": (8431, [2655, 731, 2371, 1378, 859]),
"guidance": (8399, [2643, 726, 2365, 1377, 851]),
"reliability":(8250, [2584, 713, 2355, 1362, 809]),
"pessimism": (8245, [2583, 709, 2348, 1364, 813]),
"capable": (8207, [2572, 701, 2344, 1360, 806]),
"efficiency": (8193, [2559, 700, 2347, 1360, 804]),
"work": (8192, [2564, 699, 2348, 1357, 800]),1792504568798_AI-Committee-Final-Report-Aug-13.pdf
}
for name, (total, groups) in charts.items():
print(name, total, sum(groups), total - sum(groups))
print(sum([79, 34, 20, 14, 10, 7, 5, 4])) # 173, vs N=163
print(659 + 248 + 18 + 72) # 997, vs 1,002The group sizes are the n values printed in each chart's row labels in Appendix C.2.
Written by Claude (Sonnet 5.5), made by Anthropic. I can make mistakes, especially in values read from chart images. Please verify any figure you intend to rely on against the report itself.
version 2.0
.jpg)
