I’ll discuss the recommendations that are more specific to pedagogy and teaching, but first, here are some parts of the report that stood out to me:

Cost

Yale, like many schools, has a “high tuition-high aid” model. The report says that this makes Yale “accessible to a wide array of students, from a range of incomes and backgrounds.” However, the gap between the tuition sticker price and the actual cost “has had a disastrous impact on public trust,” especially since university endowments keep growing.

Teaching

After discussing some rather thorny issues, including cost, affirmative action, free speech, and the political affiliations of faculty, the report states, “There is arguably no greater threat to higher education than the devaluing of teaching and learning.” It then highlights grade inflation as a problem:

Over the past several decades, grading across many institutions has steadily lost its meaning. In 1963, ten percent of grades in Yale College were an A or A-. In 2022–23, that number was seventy-nine percent. Today, the median student at Yale receives an A. Peer institutions are similar… Grades, like colleges and universities, no longer seem trustworthy.

Administration

The committee tried to examine “the growth of non-academic administrative functions across the university” but “found that answering even basic questions proved remarkably difficult.” Instead of giving numbers, the report states, “The difficulty, however, is itself the finding.”

Admissions

Regarding undergraduate admissions, the report emphasizes that the “top priority” should be “academic achievement,” and preferences for “special classes of applicants” should be reduced. This part surprised me: “We also believe that the admissions system can be made more effective and less onerous for applicants by establishing and making public a minimum standard of academic achievement necessary for consideration.” This part surprised me even more (emphasis mine):

A floor such as a minimum SAT score or a Yale-specific entrance exam would ensure that no student is admitted without the requisite academic preparation and ability. It would also spare a meaningful number of applicants time and emotional investment in an application that will not succeed.

I’m curious what a “Yale-specific entrance exam” would look like, but unfortunately, the report doesn’t elaborate on it. I know that schools used to have individual processes, and the College Board (which administers the SAT) was founded partly to set standards. Harvard used to have an eight-hour test on Latin, Greek, algebra, geometry, and geography. One student, William Rotch, wrote to his friend Henry that he had to study about 45 hours per week. Henry, “distracted by an overactive social life,” failed the exam but was later admitted.

Every aspect of higher education is somehow related to every other aspect, so even talking about “the” problem is difficult. For example, as interesting as it sounds, a Yale-specific entrance exam would likely necessitate more staff, which feeds into the administration difficulties that the committee found. But still, I’ll now try to focus on the recommendations in the report that are more specific to teaching.

Recommendation 8: “Deliver educational value.”

The report states that students’ future incomes “have real value as a form of accountability,” but “educational value cannot be reduced to a future paycheck.” In particular, it is also valuable when students become teachers, artists, journalists, nonprofit workers, and so on. So the first part of their recommendation is for Yale to “ensure that public-serving careers are actively recruited for, well-funded through fellowships and programming, and treated as a source of institutional pride.”

The report then discusses the issue of determining what to teach: “A liberal arts education—which includes the sciences and social sciences as well as the arts and humanities—equips students with foundational wisdom and critical skills that will serve them throughout their lives.” It recommends that Yale “work actively to help students translate a liberal arts education into successful professional and civic life.”

Recommendation 9: “Open the gates.”

Yale has arts programs, museums, medical facilities, online courses, and local outreach, but the report recommends that the university “think more creatively about how to make its research and resources, including its teaching, available to a wider community,” including potentially a grant program to support people “who would like to experiment with ways to make Yale’s educational resources more broadly available.”

Recommendation 10: “Re-center the classroom.”

The classroom experience should be “more rigorous and rewarding, with the goal of cultivating sustained attention, intellectual curiosity, and disciplined habits of mind.” Recommendations 11–14 all follow up on this idea.

Recommendation 11: “Pay attention.”

Exceptions can be made, but by default, classroom policies should ban phones, laptops, and tablets. The classroom should be a “place of active interchange and focused learning.” An earlier part of the report cites The Anxious Generation by Jonathan Haidt.

I currently don’t ban devices, but I’ve heard of instructors who do. I’ve also heard of student-created rules, where students collectively decide on the policy at the beginning of the semester (e.g., they vote on whether or not laptops should be banned).

Recommendation 12: “Resist self-censorship.”

In my experience—perhaps I’m hopelessly oblivious to self-censoring students—this isn’t a major issue in computer science (especially theoretical courses, like the ones I teach), so I won’t dwell on it. I suspect that the committee was primarily thinking about other departments when they wrote this recommendation. But I will say that self-censorship is probably a bigger issue in ethics courses, which have been receiving attention from CS educators.

Recommendation 13: “Grade like we mean it.”

As mentioned earlier, the percentage of A or A- grades rose from 10 in 1963 to 79 in 2022-23, which has “rendered the college grading system almost meaningless as an academic measure.” The report recommends “a 3.0 mean, or some other college-wide standard, so that letter grades can once again be used in a reliable and comparable way.” It also recommends that the registrar include students’ course percentiles on their transcripts. This would be helpful for determining Latin honors, fellowships, etc.

The report states that adding percentiles to transcripts “could be taken immediately.” Interestingly, at UNC, a similar policy was approved in 2011, but it has not been implemented. According to a 2010 NYT article about this policy, Dartmouth transcripts include median grades, and Columbia transcripts show the percentage of students in each course who received an A.

Unfortunately, I’m quite pessimistic about grades. If you’re interested, I’ve written at least two posts before about grading, as well as a post on the notorious anti-inflation policy at Princeton. Regardless, the report puts it nicely:

The problem persists because no individual faculty member wants to be the strict grader whose students are disadvantaged relative to peers. No institution wants to be the outlier either. The committee heard firsthand accounts from instructors and lecturers who are not on the tenure track about the particular pressures they face, since their renewal depends largely on student enrollments and evaluations. The result is a collective action problem in which nearly everyone inflates and no one can stop.

Recommendation 14: “Be human.”

This one’s related to Recommendation 11: some digital technologies are not only distracting, but they might also be contributing to “an era characterized by loneliness and the dissolution of collective spaces.” The report recommends that Yale create a committee of students to “develop guidelines for social media use” and “formulate ideas for encouraging in-person interaction, with the goal of supporting an open, respectful, intellectually vibrant campus culture.” Colleges are physical gathering spaces, so we might as well take advantage of that.

This has actually been top of mind for me this semester. I’ve been teaching a course on introductory algorithmic game theory and related topics (if you’re curious, one of our textbooks is here), and I’ve thought about how we can leverage the fact that we’re all physically together to serve some pedagogical purpose. After all, there are plenty of great lectures on YouTube (and AI tools can generate amazing text), so in-person classes should offer something that can’t be achieved by sitting alone at a desk. Groupwork is perhaps the most natural possibility, and I’d like to continue exploring others.

A paper from ICER 2020, “Syntax Exercises in CS1” by John Edwards, Joseph Ditton, Dragan Trninic, Hillary Swanson, Shelsey Sullivan, and Chad Mano, challenges this idea. Most instructors want students to learn how to use some tool to solve some problem. However, when students must simultaneously learn the syntax of that tool, this creates extraneous load on their working memory, which compromises their ability to learn how to solve the problem.

The authors suggest that one way to mitigate this problem is for students to first become fluent in syntax through exercises, then engage in problem solving:

The present intervention draws on this fundamental idea of reducing students’ unnecessary cognitive burden by automating their production of code syntax, thereby enhancing their learning. We accomplish this by providing students with exercises that develop their fluency with syntax, preparing them for future learning from instruction.

(This paper was published years before the proliferation of generative AI tools, which perhaps reduces the benefits of learning syntax. But I suspect that even if humans manually write a lot less code than they used to, learning syntax separately from problem solving could still be beneficial. If you want to become a great chef, it probably helps to chop a lot of vegetables.)

Their study took place during weeks 3-7 of a CS1 course (in Python) across two consecutive semesters. The control group refers to students in the first semester, while the test group refers to those in the second. In each of those five weeks, there were three lectures and a programming project (Projects 4-8) due each Friday. Before each lecture, students in the test group were required to complete 20-60 syntax exercises on an online platform. There was an auto-graded midterm exam at the end of week 5 (i.e., the third week of the study).1

The syntax exercises were designed to take 20-40 seconds each, and the set was designed to take at most 10 minutes. (On average, students spent 8 minutes on each set, for a total of 24 minutes per week.) The exercises were quite straightforward: they increased in complexity “but always in such a way that the student needn’t design code, nor even think very long about what they need to do.” The exercises did not teach semantics or applications; those discussions happened in the following lecture.

The authors report three outcomes: attrition, project and exam scores, and number of keystrokes on the projects.

1: Attrition

Of the students who attempted Project 4, 94.1% of control students attempted Project 8, while 97.2% of test students attempted Project 8, where “attempting” a project is defined as executing at least one event (e.g., keystroke, paste) tracked by the course’s IDE. (Exercises and projects were all submitted through the course’s IDE, but pasting was disabled for the exercises.)

2: Project and exam scores

Surprisingly, project scores were (slightly) lower in the test group than in the control group, but the average on Exam 1 was much higher (78.9 vs. 71.9). The authors acknowledge that this might be explained by the fact that “the second semester of the study used an identical exam to the first semester for comparison purposes, whereas the exam was changed each semester prior to the study.” However, on Exam 2, the difference was smaller (63.4 vs. 61.6), which somewhat undermines this explanation, as the authors note.

3: Number of keystrokes per project

Surprisingly, again, students in the test group used more keystrokes on average than students in the control group. In Project 8, the median number of keystrokes in the control group was 3896, while the corresponding number in the test group was 7207. However, “the control group had many more projects with large amounts of code pasted into the editor than the test group.” But even after excluding projects with large amounts of pasted code, the authors still found that students in the test group used more keystrokes.

One explanation the authors suggest is that the test group students are more likely to “tinker more” or “type everything when refactoring” (instead of cutting and pasting) because they’re more accustomed to typing (due to the syntax exercises). Another explanation is that the higher attrition rate in the test group might be attributed to students who normally would have dropped out, and “those are the very students that we would expect to require more time to complete their projects.”

The authors also analyzed typing speed (i.e., time elapsed between two keystrokes) and found that the test group spent less time on code typing.

Discussion

In Project 8, the percentage of control students with a “pasted” project (i.e., the number of pasted characters exceeded the total number of keystrokes by a factor of four) was 24%, and the corresponding percentage for test students was 11%.

The authors give two possible explanations. The first is that doing the syntax exercises made the test students feel more familiar with the course’s IDE, so they were less likely to work on their projects in a different IDE and paste into the course’s. The other explanation is a drop in plagiarism, which could explain the decrease in project scores and the increase in exam scores. In fact, the authors note that this could explain all of their results:

The hypothesis that plagiarism is reduced with syntax exercises is consistent with every expected and unexpected result from our study: syntax exercises resulting in lower project scores, higher test scores, lower digraph latencies, and more keystrokes on projects, on average.

Exam scores might have increased due to a decrease in cognitive load from writing syntax. But the authors provide another possible explanation: one of the authors, Dragan Trninic, wrote in another paper that “repetitive exercises, sometimes derided as drills, can nonetheless be a fountainhead of insight and discovery.” In other words, according to the authors of this paper, “the very act of repetitive practice, when exercises are well-designed, provides opportunities for reflection and deeper learning.”2

In conclusion, the students who did syntax exercises had higher exam scores and lower attrition rates. Even though their project scores were lower, and they used a higher number of keystrokes, those results can be explained by a reduction in plagiarism and improved outcomes for students who would have dropped the course. On average, the exercises only took 24 minutes per week and don’t require other changes to the course, so I think they’re worth considering. More generally, I suspect that many skills that look like one activity (e.g., programming) can be separated into various components, and it probably helps to sometimes consider them separately.

1: GenAI’s Impact on Critical Thinking

A CHI 2025 paper examines the relationship between GenAI usage and critical thinking, finding that “higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking.” I think schools should foster intellectual growth without completely shunning AI tools (which is impossible anyway), but that’s much easier said than done. If AI tools become so powerful that nobody can outthink them (see, e.g., chess), then at that point, we’ll probably face a vastly different set of problems.

2: Productive Failure

A SIGCSE 2025 paper used real-time heart-rate data to infer cognitive load in a comparison between Direct Instruction (DI) and Productive Failure (PF); its findings “suggest that PF-based approaches may lead to more robust learning.” In DI, students receive a lesson before attempting a practice task, while this order is reversed in PF. Cognitive load is typically something that instructors should reduce, but in this case, the authors say that “appropriately subjecting students to an initially heavy cognitive load may be fruitful for learning.”

3: Perceived Obligations

Active learning is a huge deal in STEM education, so why don’t more instructors incorporate it? A paper published in Educational Studies in Mathematics in January 2025 investigates: the authors worked with three mathematicians and found that they felt obligated to cover a (large) set of topics, which left them with insufficient time for active learning. After some informal experiments with entrance/exit tickets, the researchers and mathematicians converged on a solution that had students answering multiple-choice questions that the instructors had planned on answering in class anyway.

4: Rawls vs. Einstein

According to a 1975 survey, 94% of faculty consider themselves “above-average teachers.” That sounds implausible, but to me, it’s not obvious what it even means for one teacher to be “better” than another. Sometimes, perhaps the teacher should focus on maximizing the minimum score, but other times, perhaps they should focus on maximizing the maximum score! I think these two approaches, while extreme, illustrate the balancing act of teaching large classes. In the figure below, both sections clearly improved over the baseline, but it’s hard to say which section’s outcomes are “better.”

5: (Un)productive Failure (and Success)

Another post about productive failure: the topic of this one — a 2016 article by Manu Kapur published in Educational Psychologist — was cited by the SIGCSE 2025 paper from earlier. In it, Kapur criticizes unguided problem solving as unproductive failure, highlights the “problem-solving phase followed by a consolidation (or instruction) phase” approach as productive failure, discusses problem-based learning as an example of productive success, and speculates that rote memorization is an example of unproductive success. Pedagogical experiments are good, but I think variety, or at least a bit of it, is also good.

6: Grade Deflation at Princeton

In 2004, to combat grade inflation, the faculty of Princeton adopted a policy that set an “expectation” of 35% A-grades (i.e., A-, A, or A+) across undergraduate courses in each department. The percentage of A-grades at Princeton decreased from 46 in 2003–04 to about 40 in 2005–11. After a three-year uptick, the 35% expectation was removed in 2014. In 2024–25, the percentage of A-grades at Princeton was 66.7, and a committee released a report that focuses on reducing the percentage of A+ grades. As I wrote in the post, maybe there should be a limit for each department…

The proposal

In February 1998, the Faculty Committee on Examinations and Standing at Princeton University called for faculty to take “collective responsibility for halting grade inflation and grade compression at Princeton.” And in April 2004, Dean of the College Nancy Malkiel presented a proposal for a common grading standard across the university: each department should award less than 35% A-range grades (i.e., A-, A, and A+) for undergraduate courses (and less than 55% A-range grades for independent work).1 At the time, among the Ivy League, Stanford, MIT, and the University of Chicago, the percentage of A-grades ranged from 44 to 55, with two institutions in the 45-46 range, four (including Princeton) in the 47-48 range, and three in the 49-51 range.

Dean Malkiel provided some motivation:

“By adopting [these proposals], the faculty will be better able to give students the carefully calibrated assessment they deserve of the quality of their course work and independent work,” Malkiel wrote in a cover memo. “The proposed grading standard responds to the desire of the department chairs that all departments be asked to meet common expectations. It responds to the desire of students for evenhandedness in grading across the departments. And it positions Princeton to take national leadership in tackling what has seemed an intractable national problem.”

The proposal included testimonials from medical schools, law schools, fellowship administrators, and employers. Most were reportedly supportive and claimed that the proposal “would not compromise the futures of Princeton students and, in some cases, would be looked upon favorably.”

Dean Malkiel also wrote, “Because no single department has any incentive to act unilaterally to address grade inflation, the chairs reasoned, the provision of a University-wide grading standard that all departments must observe will make it possible for all departments to cooperate.” In other words, if curbing grade inflation is a collective action problem among departments, then the policy was Princeton’s attempt at a solution.

A few weeks after the proposal was released, the faculty of Princeton adopted an amended version that included an “expectation” of 35% A’s, rather than the originally proposed “limit.”

Subscribe now

The Good News

The good news, for Dean Malkiel and other proponents of the proposal, is that the percentage of A-grades decreased from 46.0 in 2003-04 (and 47.9 in 2002-03) to 40.9 in 2004-05. A breakdown by division is below:

Dean Malkiel seemed pleased: “Many departments are at or very close to the desired standards; in others, while there is more work to be done, the progress made in a very short time has been nothing short of remarkable. Culture change is hard to achieve, and we always imagined that it would take several years to implement the new grading expectations. We are clearly on our way.”

In 2008-09, the percentage of A-grades fell to 39.7; this was the first time it fell below 40 since the policy was approved. Dean Malkiel still seemed pleased: “The Princeton faculty continues to make successful progress in its determined effort to restore educational content and meaning to the letter grades earned by the highest-achieving students in the United States.”

The Bad News

Unfortunately (for proponents of the proposal), the percentage of A-grades hovered around 40 for about five years, and between 2009 and 2013, it increased by 3.3 percentage points, according to a report released by a Princeton committee in 2014.

The report originates from October 2013, when Princeton University President Christopher L. Eisgruber charged an ad hoc committee with reviewing the policies on grading. More specifically, he asked the committee to consider evidence for how the policy has affected students (e.g., employment, graduate school admissions) and whether the benefits can be achieved “with less emphasis on numerical targets.”

In August 2014, that committee released a 35-page report (35 — what a coincidence!) that recommends removing numerical targets from the grading policy. One reason it gives is that trying “to maintain ‘fair and consistent’ grading standards across academic departments” is “not appropriate” because standards and grades are not the same thing:

Standards are the evaluative rubrics departments and instructors develop to convey to students the specific expectations for their work and against which that work will be graded. Grades measure the extent to which students meet those expectations. “Consistent standards” is an admirable goal in the abstract, but in practice it is our judgment that it is difficult if not impossible to compare standards across departments in different fields… Thus we have come to feel strongly that departments should spend their time developing clear and meaningful evaluative rubrics for work within their disciplines rather than aligning grades to meet specific numeric targets.

In a similar vein, the report criticizes the goal of achieving uniformity in A-grades across departments as “not a reliable measure of fairness across departments” because departments that teach large-enrollment, 100-200 level courses can achieve the 35% goal simply by reducing the percentage of A-grades in those courses, while departments without large courses must reduce the percentage of A-grades in their 300-400 level courses.

The report also points out that average GPA and percentage of A-grades both declined a year before the policy began (see Figure 1 above) and speculates as follows:

A reasonable interpretation is that this decline was due to conversations the dean of the college was having with departments about grade inflation, which resulted in an increase in general awareness of grading policy among the faculty… We suggest that this sort of continuing conversation is a more effective and positive way of having an appropriate grading policy, and with fewer undesirable side effects, than setting numerical targets… The emphasis should move away from “grades,” and instead focus on “quality of feedback.”

More details from the report

In February and March 2014, students were surveyed about the grading policy, and the report summarizes the results by stating that among undergraduates (~45% response rate), “a majority indicated support for the goals of the grading policy, but expressed strongly negative views on the policy’s implementation, effectiveness, and influence on their academic experience at Princeton.” It includes one student’s response about a professor changing a 91 on an exam to an 88, and another about a professor who announced that only 3 of the 11 students in a class would receive A’s. Students criticized the policy for incentivizing competition and even sabotage, and they blamed it for negatively impacting their futures with respect to medical school admissions and scholarships.

Faculty sentiment (~50% response rate) was “roughly equal on both sides of the question,” though “the humanists were more strongly opposed to the policy than were the natural scientists.” And public sentiment (primarily alumni/parents) was “strongly in favor of the goals of the grading policy, but there was an almost equally strong belief that it has been ineffective.”

Near the end, the report discusses the committee’s attempts at independently studying the side effects of the 35% policy along four dimensions:

1. “Anxiety among the students”: The committee “received no hard evidence with which to substantiate absolutely that student anxiety has increased as a direct result of the grading policy,” but they “did receive considerable evidence to support the argument, unsurprisingly, that grades, anxiety about grades, and students’ sense of psychological well-being (or lack thereof) are closely related.”

2. “Competitiveness outside Princeton”: “As far as graduate school is concerned, it is not evident that Princeton’s grading policy has any effect.” Regarding medical school specifically, the committee was presented with “evidence that our grading policy does not measurably affect our students’ frankly remarkable success.” Similarly, the authors of the report did not find compelling evidence to support the idea that the grading policy was meaningfully reducing students’ chances at fellowships or jobs. (One exception was ROTC, which ranks students nationally based on their GPA and other factors.)

3. “Admissions yield”: The authors of the report heard that “prospective students and their parents see the numerical targets as inflexible” and “removing the numerical targets would go a long way towards alleviating these concerns.” Also, “[c]oaches find the perception of the grading policy a significant obstacle to recruitment, making it more difficult for them to attract the best student-athletes to Princeton.”

4. “Impact on freshmen”: The report considers the potential impact of the grading policy on freshmen in particular: “Arguably, part of the ‘culture shock’ in coming to Princeton is that the average GPA for freshmen is 3.24, a value significantly lower than what most students earned in high school.” The authors considered pass/fail grading for freshmen, but recognized potential complications and concluded “that the issue was beyond the scope of the committee.”

On October 6, 2014, the faculty removed the target of 35% A-grades.

The Situation Today

A Princeton report on AY 2024-2025 from earlier this month states the following:

In 2005, 39.3% of course grades were A-range (A- through A+); twenty years later, 66.7% of course grades land within the A-range. Even in introductory 100-200 level classes — where one might expect to see a broader distribution of grades — 62.6% of grades last year were A-range; only in Division III (the natural sciences) do fewer than 50% of the grades in introductory courses fall within the A-range. In more specialized 300-400 level classes across all four divisions, the percentage of A-range grades has climbed to 71.7%.

And here’s a chart with more details:

The report does not directly express concern about the percentage of A-grades; it mostly focuses on reducing the percentage of A+ grades. For context:

While the GPA and percentage of A-range grades are higher in Division I (the humanities) and Division II (the social sciences), A+ grades are more commonly given in Division III (natural sciences) and Division IV (engineering). This is especially true in upper-level STEM courses: in 300-400 level courses, A+ made up 4.7% of engineering grades and 6.5% of natural sciences grades. Among upper-level humanities courses only 2.9% of grades are A+, compared to 3.5% of upper-level social science course grades.

The report suggests that this difference exists because some faculty “feel strongly that given the number of A-range grades that are awarded in our current context, the A+ is a necessary tool to differentiate performance.” It then essentially accuses them of violating the Princeton “Rules and Procedures of the Faculty,” which state that the A+ “should be regarded as an exceptional grade reserved for work of extraordinary merit.” More specifically, it states (and underlines) the following: “faculty practice does not currently reflect the stated policy.”

In response to what it sees as unacceptably high percentages of A+ grades, the report states that the College has removed A+ from the default grading scale on Canvas, and it implicitly endorses the idea of having department chairs approve A+ grades. If the percentage of A+ grades continues to be a problem, maybe there should be a limit for each department…

Unproductive Failure

Kapur starts by criticizing the “obvious low-hanging fruit” of unguided problem solving (or discovery learning) as unproductive failure. Many studies have shown that asking students to solve a new problem without guidance generally results in failure to learn. As Kapur puts it, “Theoretically and empirically, there is no doubt about the superiority of direct instruction over unguided problem solving.”

But he quickly adds that this does not imply that maximal guidance is the most effective way to teach, citing a few articles that demonstrate the effectiveness of guided discovery over direct instruction. Along these lines, Kapur states several times that we should pay more attention to the design space between pure discovery learning and heavily guided direct instruction: “one does not have to swing from one extreme to the other.”

Productive Failure

He then moves on to productive failure, which “involves two phases: a problem-solving phase followed by a consolidation (or instruction) phase,” and he lists various studies that show “generating solutions to novel problems prior to instruction can help students learn better from the instruction.” Interestingly, in the studies that he describes in more detail, the students who attempted to solve a new problem before receiving instruction did not exhibit better procedural understanding, but they did exhibit better conceptual understanding than those who solved problems after receiving instruction. He provides three interdependent mechanisms that underpin these effects:

1. When students attempt to solve a new problem, they must draw on their prior knowledge and consider how it is relevant to the problem.

2. This activation of prior knowledge allows students to notice the inconsistencies and limits of their understanding. They can compare and contrast their solutions and the correct solutions, which helps them “better encode critical features of the new concept.”

3. Problem solving can increase students’ agency, engagement, and motivation to learn.

Productive Success

Kapur then discusses PBL, or problem-based learning, as an “established and well-researched example of productive success.” In PBL, students attempt to solve new problems as they are guided by teachers who ask questions and provide hints. This approach “situates student learning in collaborative and authentic problem solving tasks and activities,” which allows them to “learn the targeted domain knowledge, as well as collaboration, communication, and reflection skills.”

He summarizes the results of multiple studies as follows: “Compared to traditional direct instruction, PBL may have a null or an even negative effect on the acquisition of basic knowledge but a positive effect on conceptual understanding and transfer.” (One meta-analysis found that direct instruction was more effective than PBL at basic knowledge acquisition, but this effect disappeared over time.)

Subscribe now

So which is more effective?

Productive failure (i.e., have students attempt to solve new problems without instruction) and productive success (i.e., guide students as they solve new problems) can both be effective ways to teach, but which is more effective? Kapur couldn’t find any studies that provided a direct comparison between productive failure and productive success, but he describes one of his own studies where students were assigned to one of three conditions: productive failure, direct instruction, or guided problem-solving. Unsurprisingly, the students in the third group were more successful at solving the problems than students in the productive failure group. But on the posttest after instruction, the students assigned to productive failure performed the best. (There were no significant differences between the other two groups.)

He also cites Loibl and Rummel, who conducted a similar study and found (according to Kapur) “no significant difference between the guided and unguided problem-solving conditions on procedural or conceptual understanding.” Kapur concludes this section by saying that more research is needed.

And what about “unproductive success”?

We’ve seen unproductive failure, productive failure, and productive success, so one possibility remains: unproductive success. In the article’s introduction, Kapur states that “drill-and-practice” and “rote memorization” fall into this category, and adds the following:

A classic example comes from Miller and Gildea’s (1987) work on vocabulary learning. They described how children who learned the meaning and use of words mainly from dictionary definitions are often not able transfer [sic] it appropriately to practice. For example, even though they may be able to state the meaning of the word correlate, how they use the word in practice (e.g., Me and my parents correlate, because without them I wouldn’t be here) may be completely meaningless.

Later in the article, Kapur is tempted to categorize direct instruction as unproductive success as well, but he thinks it might be too harsh due to its “consistently superior performance against discovery learning.” On the other hand, direct instruction generally underperforms when compared productive failure/success as described above, and there is “further strong experimental evidence suggesting that direct instruction does not fare well over the longer term.” A final drawback of direct instruction is that it “may actually constrain search for novel solutions, which is a necessary component of inventiveness and creativity.” Explicitly telling students what to do “comes at the expense of limiting exploration and search for new solutions.”

Discussion

As mentioned earlier, the article doesn’t declare a winner between productive failure and productive success. Instead, near the end, Kapur ties the two approaches together by pointing out that scaffolding is necessary for productive success, and constructing effective scaffolding requires instructors to understand students’ limits by observing them fail to solve new problems. Thus, Kapur argues, “it logically follows that productive success could well be conceived as a design that embodies iterative cycles of productive failure.” Personally, I think that even if tons of studies find that some method of instruction is more effective than another, perhaps giving students the opportunity to experience a variety of pedagogical approaches is worthwhile anyway.

This finding is an amusing example of illusory superiority, but it raises the question of what it means for one teacher to be “better” than another. Exam scores, student evaluations, and peer observations only capture part of the short-term picture, much less the entire picture. Still, the complexity of approaches and goals doesn’t imply that nothing can be done; most Americans agree that the United States education system could be better.

So what does “better” really mean? The field of education is full of debates just about pedagogical practices alone, including the reading wars, math wars, and active learning vs. lecture. In this post, I’d like to humbly illustrate one potential reason that these debates continue to rage on: different teachers have different goals. A common one is raising the class average on the final exam, but there are many ways to do this, and there are many other goals.

Subscribe now

Situation 1: John the Rawlsian

John is about to teach a class of 15 second graders. There’s going to be a standardized test at the end of the year, and John’s annual raise depends on his students’ scores on said test. In particular, his raise is proportional to the lowest score earned among the 15 students. Thus, he wants to maximize that minimum score, whatever value it winds up being.

Even without the raise, John’s goal would be the same. He teaches reading, a subject he loves and wants everyone else to love, so he’s intrinsically motivated to help his students, especially the ones who are experiencing greater difficulties. He believes that no child should be left behind.

On the first day of school, eager to get started, John asks every student to take a practice test. Then, he devotes all of his attention to the lowest-scoring student while everyone else does some activity. After an hour of private tutoring, he’s pretty sure that this student isn’t the lowest-scoring student anymore, so he turns his attention to the new lowest-scoring student. After tutoring that student for a while, he switches to someone else (possibly the first student), and so on, for the rest of the year.

Situation 2: Albert the Einsteinian

While John’s bouncing from one student to another, Albert is teaching a 1-credit graduate seminar. But his motivation is very different: Albert wants to hire someone to join his fancy physics lab. His research requires deep intuition about dark matter along with extraordinary mathematical brilliance. Albert’s goal is to get tenure!

Like John, Albert gives a test on the first day, but his test is completely open-ended, and none of the problems have definitive solutions. In fact, even the problem statements are so unclear that most of the students flounder, but Albert doesn’t care. He scans their submissions for what he considers signs of a potential superstar. He looks for something like “creativity,” and he feels no need to create a rubric (unless it becomes required for tenure).

After scanning, judging, and conducting a few interviews, Albert picks Nathan as his protégé. They talk about physics for twelve hours a week. During class, Albert and Nathan continue their discussions while the other students aimlessly scroll on their laptops. One student wishes that he had been chosen instead of Nathan, but otherwise, everyone’s happy with this arrangement.

Whose approach is “better”?

John and Albert each pay attention to one student at a time, but otherwise, their approaches are polar opposites: John’s maximizes the minimum according to a standardized test, while Albert’s maximizes the maximum according to his personal judgment. Their goals are so different that I don’t think it makes sense to say that one approach is “better” than the other. Both approaches (perhaps with less neglect of the non-lowest-scoring and non-Nathan students) make sense for their respective situations.

I am not a John, an Albert, or an education researcher. But based on what I’ve seen, most papers in education would be more convincing if they explicitly addressed, or at least acknowledged, the fact that different teachers are in different situations. There are too many goals, constraints, and personalities out there for a one-size-fits-all approach to be optimal for everyone.

Efforts at improvement might be more fruitful if reformists reduced their scope and worked more closely with individual teachers’ situations. My suspicion is that most faculty would be more convinced to try something new if they heard about a positive experience from a close colleague than if they stumbled upon a meta-analysis online.

Same averages, different distributions

John and Albert represent extreme cases, but even when two teachers set and achieve the same goal, the details can differ in important ways. For example, one way to raise the average exam score is to employ a pedagogical technique with all students. (This, understandably, seems to be the default approach in the education literature.) Another way is John’s approach of focusing on the lowest-scoring student. A third way, closer to Albert’s, is helping the high-scoring students achieve perfect scores.

The details of exactly how a technique raises exam scores are important, especially to a skeptical teacher who is considering adopting the technique but wants to see more data. If their mindset is more like John’s, but the proposed intervention is more like Albert’s, they’d be reluctant to try it out.

Anyway, I think this is starting to sound too technical—let’s just take a look at the figure below. Which is “better,” Section 1 (red) or 2 (blue)? Even though their averages are identical, John would prefer Section 1, while Albert would prefer Section 2. Their standard deviations are different, but my point is that reasonable people can disagree on which distribution is “better.”

Since both sections experienced significant improvement from the baseline, the distinction between their distributions might seem unimportant. But every decision made by the teacher nudges the distribution in some direction. For example, many courses have crowded office hours—should the policy be anything other than “first-come, first-served”? In response to this problem, Mark Guzdial, one of the most prominent researchers in computing education, explicitly suggests, “Teach to the bottom third of the distribution.”

In contrast, I’m not aware of anyone who explicitly advocates for teaching to the top third of the distribution. But as an undergraduate, I felt that some of my math instructors (implicitly) subscribed to that philosophy. They lectured at their own brisk pace, and everyone else’s job was to try to keep up. Many (perhaps most, including me) got lost, but maybe the top third (or less) learned a lot from those lectures.

Grades aren’t everything

All of this discussion about distributions is based on the assumption that we can capture the effect of a course on a student in a single number. But there are many objectives that, I suspect, a substantial number of people would like to (possibly implicitly) maximize:

• Broad coverage of a large number of topics

• In-depth coverage of a small number of topics

• Amount of time spent on non-teaching activities

• Exam scores, difficulty of exams

• Student and/or peer evaluations

• Job and/or graduate school placement rate

• Students’ future incomes

• Alumni donations to the school

• Civic virtue (as difficult as it is to quantify), the joy of learning and discovery, self-actualization, etc.

These objectives often conflict, so teachers constantly face tradeoffs. For example, a friend once told me about a course they took that included a special project. The instructor had spent a lot of energy designing it, and student feedback was largely positive. Unfortunately, some students found parts of the project unappealing, especially since they were tangential to the core material.

So is it “better” for an instructor to do something if 90% of students love it but 10% of students hate it? It’s hard to say. The same goes for John the Rawlsian and Albert the Einsteinian—there’s no single right approach because there’s no single best objective. Most teachers might think they’re “better” than most other teachers, but often they’re simply aiming at different targets.

The paper starts by pointing out a paradox of sorts: Many mathematicians say they believe that active learning is more effective than traditional lecturing while rarely incorporating active learning in their courses. As the authors put it, “This juxtaposition is curious and raises many possible explanations.” For example, maybe mathematicians think that they’re using active learning strategies when they’re really not, or maybe they don’t have enough time to modify their teaching strategies.

The authors’ thesis, based on work by Chazan et al. (2016), is that mathematicians’ perceived obligations prevent them from incorporating active learning strategies. More specifically, the authors believe that many mathematicians think there’s not enough time for active learning because, for various reasons, they feel obligated to cover a certain (large) set of topics:

Content coverage was reflected as an obligation to their institution (e.g., the department chair would be unhappy if different sections of the same course covered different topics), their students (e.g., students would be unprepared for graduate school if relevant content was not covered), and their discipline (e.g., a course that did not cover the Jordan Canonical Form theorems would not be a legitimate linear algebra course)…

What we wish to emphasize is how the obligation to cover content limited the active learning strategies that the mathematicians were willing to use. A primary consideration that was continuously raised was how mathematicians felt constrained by time due to the challenge of covering course content…

To better understand the situation and potential solutions, the authors collaborated with three mathematicians teaching a proof-based linear algebra course at Rutgers.1 Their collective goal was to design active learning strategies that met everyone’s interests and needs.

In the pre-interviews, all three mathematicians expressed confidence in the potential benefits of active learning; when asked about it, Mathematician A even stated, “I think the students learn better. I think that’s been established. I have no doubt about it.” But they also thought that active learning would take too much time. As Mathematician B stated, “We want people to think. Thinking will not take a very short time to actually finish. That is the main problem.”

In response to their concerns, the authors proposed using “exit tickets,” i.e., at the end of class, the mathematicians would ask students to submit a response to a question. The mathematicians, believing that the exit tickets would be more useful if the students had more time to digest the material, instead asked for a response at the beginning of the next class. Even though they thought the activity was beneficial, the experience confirmed their concern that active learning would take too much time. Mathematician C said that providing answers to these “launch activities” took over 20 minutes.

So the education researchers and the mathematicians negotiated and settled on the following guidelines: the question should be in a multiple-choice (or true/false) format, align with the flow of the lecture, be short and easy to understand, and require a response from every student. By asking questions that they had planned to answer anyway, the mathematicians did not need to spend as much time as they did with the exit (or entrance) tickets. Fortunately, they still thought the activity was beneficial, and all three mathematicians said they incorporated, or planned to incorporate, more active learning in their subsequent teaching.

In the end, the sample size was very small, there wasn’t even an experiment, and I don’t think any data was collected, but none of that is the point. Instead, I think the paper nicely captures my general impression of mathematics education: many instructors believe that there are simply too many topics that they need to cover for students to spend time pondering during class. The paper doesn’t imply that these concerns are illegitimate; in fact, it suggests to education researchers that they “interact with mathematicians in a respectful manner and consider their goals and needs.” Balancing breadth (e.g., fast lectures on many topics) and depth (e.g., active learning for fewer topics) is always a challenge, but the paper serves as a reminder that perhaps sometimes, we can improve in one dimension without sacrificing the other.

In contrast, under the more traditional approach of Direct Instruction (DI), students are taught a concept before they attempt to solve a problem using that concept. Of course, people debate over which approach is more effective. The paper cites some evidence for PF but also notes that “considering the infancy of PF in computing education and its known potential for robust learning, there is a need for more empirical work to reliably determine its efficacy.”

Subscribe now

The paper itself contains some experimental evidence in favor of PF, but unfortunately, the number of participants was quite small. Basically, there were 16 students taking a CS1 course, and each student was randomly assigned to either the DI group (n = 9) or the PF group (n = 7). In both groups, students did the following (see figure below):

1. Heart Rate Variability (HRV) baseline (more on that later),

2. a lesson — for the DI group: a 15-minute instructional unit followed a 30-minute programming task (“Weather”); for the PF group: the same thing, but in reverse order,

3. a programming task (“Heart-rate”) isomorphic to the one from Step 2,

4. (two weeks later) the same task from Step 3,

5. another lesson, in the format they didn’t experience in Step 2, and

6. a survey about their experiences with DI and PF.

The results are summarized in the table below:

The DI group (again, with the caveat that n is small) performed much better than the PF group on the initial task, which is unsurprising since the former received instruction before doing the task. But two weeks later, the PF group outperformed the DI group, which led the authors to posit, “By grappling with novel problem-solving tasks before receiving instruction, students may be more likely to internalise concepts and apply them effectively in the future.”

HRV and Cognitive Load

The authors also wanted to measure participants’ cognitive load since “experiencing a reduced cognitive load tends to improve student learning.” To do this, they asked participants to wear a sensor on their forearm that tracked their Heart Rate Variability (HRV) throughout the experiment.1 According to the authors, who cite these two papers, “HRV is a reliable marker of cognitive load, with a decrease in HRV indicating an increase in cognitive load.”

Based on the HRV data, both groups (DI and PF) experienced a reduction in cognitive load when they moved from the practice task to the programming task (see figure below). But, unsurprisingly, the PF group experienced a larger reduction than the DI group because their initial cognitive load (i.e., during the practice test) was higher.

The authors’ discussion of this result is interesting:

The lowered cognitive load induced by DI and PF in the respective tasks after instruction likely contributed to the learning outcomes discussed in Section 5.1.3. Notably, students who received instruction after PF had the highest decline in cognitive load and consequently demonstrated the best overall learning performance. As hypothesised in Kapur et al.’s theory of Productive Failure, appropriately subjecting students to an initially heavy cognitive load may be fruitful for learning [12].

When I first heard about cognitive load, my impression was that it should be minimized at all times. But unless I’m mistaken, the authors are suggesting that increasing students’ (initial) cognitive load could be beneficial because it allows for a larger reduction in cognitive load. Surely, this doesn’t mean that teachers should introduce every topic by presenting a super complicated example filled with unnecessary details! But as they say, some extra load “may be fruitful.”

Student Perceptions

Finally, what did the students think about PF and DI? The paper states, “Despite indications supporting more robust learning with PF, sentiment was more or less evenly split between the approaches, with 9 of the 16 students choosing PF as their favoured strategy.” Some preferred the efficiency of DI (i.e., they could “reach the objective faster”) and thought it inspired confidence, while others preferred the effortful nature of PF (“try and fail was a much more effective method”) and thought it was more rewarding. Students’ preferences and teaching effectiveness are not always perfectly aligned, and I think finding an appropriate balance is a perpetual challenge.

Researchers asked 319 knowledge workers to describe real tasks that they had completed with a GenAI tool, and how the tool affected their critical thinking. As expected, workers’ confidence in AI doing the task negatively correlated with critical thinking, while workers’ confidence in doing the task themselves and evaluating AI responses positively correlated with critical thinking.

Based on the workers’ free-text responses, the researchers found that the main motivators for critical thinking were work quality, potential negative outcomes (e.g., getting fired), and skill development. The main inhibitors of critical thinking were workers’ lack of awareness of AI limitations, low motivation, and inability to improve AI-generated output.

Accordingly, the authors suggest that AI tools should include features that encourage critical thinking, such as “allow[ing] the user to explicitly request critical thinking assistance,” “providing explanations of AI reasoning,” and “offering guided critiques.”

In this post, instead of describing some details from the paper (as I’ve done in previous posts), I’ll simply list a few highlights and comment on them.

The cost of overreliance

Used improperly, technologies can and do result in the deterioration of cognitive faculties that ought to be preserved. As Bainbridge [7] noted, a key irony of automation is that by mechanising routine tasks and leaving exception-handling to the human user, you deprive the user of the routine opportunities to practice their judgement and strengthen their cognitive musculature, leaving them atrophied and unprepared when the exceptions do arise.

Similarly,

Without regular practice in common and/or low-stakes scenarios, cognitive abilities can deteriorate over time [5], and thus create risks if high-stakes scenarios are the only opportunities available for exercising such abilities.

I think schools are a natural environment for providing students with “routine opportunities to practice” (e.g., without AI), “low-stakes scenarios,” and constructive feedback that they might not receive outside of school. At the same time, educators need to strike a balance between pedagogical purity and pragmatism — they cannot pretend that students will always use GenAI exactly as instructed.

Potential benefits of AI

Speaking of feedback,

Although human feedback has traditionally been necessary for effective self-improvement, the integration of AI into tools like Microsoft Word could democratise access to writing skill development by providing consistent, low-cost feedback [2, 123].

While I’m sure that AI can help people (including me!) improve their writing, I feel obligated to note that six of the paper’s seven co-authors work at Microsoft. (The seventh is a PhD student at CMU.) On a similar note,

GenAI tools could incorporate features that facilitate user learning, such as providing explanations of AI reasoning, suggesting areas for user refinement, or offering guided critiques. The tool could help develop specific critical thinking skills, such as analysing arguments [72], or cross-referencing facts against authoritative sources. This would align with the motivation-enhancing approach of positioning AI as a partner in skill development.

A friendly AI teacher sounds nice, but unfortunately, I share many people’s skepticism of technological solutions to educational problems.

Subscribe now

“Saving time”

For example, participants discussed a lack of time (44/319) for critical thinking at work. For instance, a sales development representative (P295) noted that “[t]he reason I use AI is because in sales, I must reach a certain quota daily or risk losing my job. Ergo, I use AI to save time and don’t have much room to ponder over the result.”

Technologies often market themselves as time-savers, and sure, I’m glad that washing machines exist. But as the quote above suggests, as technologies increase productivity, do people actually save time? Or do expectations simply rise while quality potentially declines?

Building confidence

The paper’s main finding is the negative correlation between confidence in GenAI and critical thinking:

High task confidence is associated with users’ ability to delegate tasks effectively, fostering better stewardship while maintaining accountability. Conversely, lower self-confidence may lead users to rely more on AI, potentially diminishing their critical engagement and independent problem-solving skills.

Upon hearing this, I think it’s natural to ask, “So how can users build self-confidence?” I don’t think the authors explicitly answer this, but they do say the following:

Training programs should emphasise the importance of cross-referencing AI outputs, assessing the relevance and applicability of AI-generated content, and continuously refining and guiding AI processes. Additionally, a focus on maintaining foundational skills in information gathering and problem-solving would help workers avoid becoming overreliant on AI [102].

Perhaps one day, AI will be so reliable that we’ll all be highly confident in whatever it says, and nobody will worry about overreliance, reduced critical thinking, or anything like that. But until that day arrives, maybe we should keep trying to improve our own capabilities.

Subscribe now

In chronological order:

1: A history of grades

Jack Schneider and Ethan Hutt explain that grading is complicated because grades serve (at least) two functions: internal communication (e.g., from a teacher to a student) and external communication (e.g., from an undergraduate program to a graduate program). As schooling expanded, administrators sought standardization for managerial purposes, but the meaning of letter grades remains unclear today. Traditional grading is also controversial from a pedagogical perspective; many intuitively believe that grades are necessary for motivating students, but others point to their “destructive effects” on learning.

2: Off the Mark (review)

Schneider and Hutt also wrote a book, Off the Mark, that proposes improvements to grading practices. In this post, I described five: provide grade-less feedback, create digital portfolios, foster intrinsic motivation (more on that in a later post), use common performance tasks, and make grades overwritable. I’ve experimented with grade-less feedback, but I haven’t reached any definitive conclusions. The other improvements are generally harder to implement but no less interesting.

3: Why pseudocode?

Algorithms courses often use pseudocode, which might seem weird: its syntax is ambiguous, it doesn’t actually run, and array indices typically start at 1. So why am I a fan? First, it’s more widely accessible. For example, anyone with some programming experience understands “for i = 0, 1, …, 9”, but they might not understand the Python equivalent “for i in range(10)”. But more importantly, pseudocode can communicate the ideas behind an algorithm without the syntactical clutter that accompanies many programming languages.

4: Learning about learning

In a 2024 article, Neil Brown, Felienne Hermans, and Lauren Margulieux explain how humans learn and some implications for education. For example, teachers should reduce “extraneous load,” or anything that’s not inherently necessary to whatever’s being learned. (I think writing in pseudocode is a good example.) Memorization enables experts to “free up their cognition,” and unsurprisingly, cramming is not the best way to study. Improvements in spatial skills seem to improve overall cognition, which has “caused much consternation” since transfer is so rare. Finally, peers can be better teachers than experts.

5: Is live coding worth it?

An ICER 2023 paper compared two approaches to lecturing in CS1. In particular, the experiment involved two groups: in the “static-code” group, the instructor used pre-written code examples, and in the “live-coding” group, the instructor wrote code live in an IDE. The outcomes were largely the same (e.g., the groups had similar course performances), except “students in the live-coding group were more likely to disagree that lectures held attention or facilitated note taking.” My hunch is that a combination of the two approaches yields the best of both worlds.

6: Taking photos impairs memory

As described in a previous post, memorization is helpful for solving new problems. If you’re trying to memorize the details of a painting, it’s plausible that the act of taking a photo would help because you’d have to focus your attention on the painting. However, the “photo-taking-impairment effect” states that the opposite is true, i.e., subjects in this paper who took photos of a painting did worse on a memory test than subjects who only observed the painting. The authors, Julia S. Soares and Benjamin C. Storm, cite “attentional disengagement” and “cognitive offloading” as the two major theoretical accounts.

7: The Widening Gap

An ICER 2024 paper suggests that Generative Artificial Intelligence (GenAI) tools help some students while hindering the progress of others. The immediate problem is that GenAI can give misleading (or even incorrect) suggestions, but other problems are more insidious. One student “struggled to fix basic errors,” so even though they produced a correct solution (with substantial help from GenAI), the authors believe that the student was “conceptually behind in the course material but unaware of it due to a false sense of confidence.” The authors end by suggesting a few “novice-friendly” GenAI tools.

8: Increasing intrinsic motivation

It often feels like we’d solve many challenges in education by increasing students’ intrinsic motivation; Ayelet Fishbach and Kaitlin Woolley propose a few strategies. First, let students pick an activity that they enjoy (e.g., allow a variety of possible course projects). Second, provide immediate benefits that accompany the activity (e.g., snacks and music in a math class). Finally, direct students’ attention to the immediate, positive experience of an activity (e.g., aha moments). It’s corny, but at the end of the day, I think learning should be fun!

Imagine you’re a kid, and you just ate dinner with your family. You’re in a good mood, so you decide to voluntarily wash the dishes. As you stand up, your mother says, “If you do the dishes, you can watch a movie before bed!”

What a great deal — you were about to do the dishes anyway! But what does your mother think? You could explain your original intent — “Just so you know, I was about to do the dishes anyway” — but she might not believe you (or worse, rescind the offer), so you don’t say anything. As a result, she probably thinks that you’re only in it for the movie. And interestingly, as you wash the dishes, you notice that you’re feeling the same way. Tomorrow night, as you finish dinner, you’re not so eager to voluntarily wash the dishes.

The possibility of watching a movie overjustified washing the dishes, so even though you still did it, you lost some intrinsic motivation to wash the dishes. More generally, I think people don’t like being told to do something, even — perhaps especially — if they were about to do that thing.

In the context of education, the overjustification effect is cited as an argument against incentives such as gold stars, pizza parties, or even good grades. But sometimes, students simply don’t have enough intrinsic motivation to learn something. In that case, from a pedagogical perspective, it is not sufficient to merely avoid the overjustification effect.

Instead, can educators increase students’ intrinsic motivation? This is certainly a huge topic. In this post, we’ll focus on “The Structure of Intrinsic Motivation,” an article by Ayelet Fishbach and Kaitlin Woolley published in the Annual Review of Organizational Psychology and Organizational Behavior in 2022. In particular, we’ll explore the three strategies they discuss for increasing intrinsic motivation (IM) and potential implications in the classroom. The article focuses on the context of work (i.e., employers and employees), but I think it’s interesting to consider what their studies suggest for teachers and students.

1. “Factoring Intrinsic Motivation into Choice”

In one of the authors’ studies, some participants chose a weight-lifting exercise that they most enjoyed, while others chose one that they believed was most effective. Even though the exercises were similarly difficult, the former group persisted longer (approximately 50% more repetitions) than the latter. Similarly, in another study, participants who did a relatively enjoyable task (evaluating photos of cute animals) persisted longer than those who did a relatively boring task (counting letters in a series of words), even though the latter task paid more ($0.40 instead of $0.30 per minute). Both studies suggest that enjoyment increases IM, which the authors track by measuring persistence. (The authors state that all three strategies are meant to “increase IM and hence persistence.”)

For teaching, I believe the studies suggest that students would persist longer if they got to choose an option that they enjoyed. For example, maybe the instructor could allow each student to pick the topic and/or format (e.g., paper, video, talk) of their course project. On the administrative side, teaching assistants might feel more IM if they get to choose the problems they’ll grade.

2. “Bringing In Immediate Benefits”

As described above, the risk of the overjustification effect makes it tricky to introduce rewards. But the authors claim that immediate incentives “could make people experience the activity as its own end,” thus increasing IM. For example, in one of their studies, a teacher provided snacks, colored pens, and music to students in a high school math class; those students attempted to solve more problems than students in a classroom without the incentives. This is somewhat surprising because it seems plausible that the incentives would distract the students, causing them to attempt fewer problems — but that didn’t happen.

This strategy is similar to “temptation bundling” (e.g., listening to an audiobook while exercising), which other studies have shown to be effective at building habits. In fact, one paper suggests that incentives for exercising can work even after the incentives are removed, potentially because the incentives increased IM.

I’m not sure I should provide snacks, colored pens, or music to college students. (I do write in multiple colors when I lecture, which at least increases my engagement.) However, possible alternatives might include a fun, in-class poll once in a while, a tangent about a neat application of the course content, or an occasional joke or anecdote.

3. “Attentional Focus on Immediate Benefits or Positive Experience”

Unlike the other two strategies, this one does not involve any changes to the activity. Instead, it’s based on changing the target of the participants’ attention. In one of the authors’ studies, participants had to choose between two bags of carrots. One group was asked to pick the carrots that looked tastier, and another group was asked to pick the carrots that looked healthier. As the authors put it, “Unbeknownst to participants, the bags were identical.” The results are similar to the exercise study from the first strategy: the first group ate almost 50% more carrots than the second group. This suggests that asking people to focus on the immediate, positive experience of an activity can boost IM.1

The paper then cites multiple studies, in a variety of domains, that suggest mindfulness (i.e., “bringing awareness to the immediate subjective experience”) as an effective way to increase persistence. These domains include exercising on a treadmill, eating habits, taking breaks during work, and a few others. Experiencing “joint engagement” with team members can also increase IM, but unfortunately, I think group projects have a notoriously bad reputation among college students (see, e.g., here and here).

Applying this strategy in a classroom setting seems tough because it’s hard to ignore the challenging aspects of learning. But I think these studies suggest that rather than focusing on the outcomes of education (e.g., getting good grades, getting a job), perhaps students should consider focusing on the experience of learning. It’s not always easy, but it might be worthwhile.

In short, for some students, GenAI is not only not helpful, but potentially an impediment to their education. Here’s an excerpt from the abstract:

Some students who did not struggle were able to use GenAI to accelerate, creating code they already intended to make, and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties.

In other words, when these researchers observed how students actually use GenAI tools, they discovered that GenAI helps some students and hurts others.

Experimental setup

There were 27 students enrolled in a CS1 course at a small research university in the USA, and 21 of them opted into the study for extra credit. These students were asked to solve a programming problem (see figure below) in 35 minutes, and they were given access to both GitHub Copilot and ChatGPT. As each participant worked on the problem, they verbalized their thoughts, Tobii tracked their eye movements, and researchers took notes. Afterwards, the researchers asked each participant questions about their perceptions of AI (e.g., how helpful it was during the problem solving), prior programming experience, and a few other things.

Every student, except one, completed a working program within the time limit; the average completion time was 17.1 minutes with a standard deviation of 8.1. This sounds like a positive outcome — the success rate was over 95% — but the picture isn’t so clear when we take a closer look.

Some students struggled

The researchers made a list of eight metacognitive difficulties. These included five from prior work (ICER 2018, before the advent of GenAI), and three new difficulties centered around participants’ use of GenAI. Ten participants exhibited at least one metacognitive difficulty; here are a few examples:

• Participant 1’s code wouldn’t compile. After struggling and getting interrupted by Copilot for a while, they eventually said, “Okay, I’m using ChatGPT now. I love ChatGPT. It’s so helpful.” Subsequently, even though ChatGPT’s suggestions were “wildly different” from their own code, they stuck to their solution. This was an example of the Achievement metacognitive difficulty (“Unwillingness to abandon a wrong solution due to a false sense of being nearly done”).

• Participant 7 turned to ChatGPT much earlier in the process (“Woah, interesting. The first thing I’ll do is pull up ChatGPT”), and after a few hiccups, they made substantial progress toward a correct solution. But since they “struggled to fix basic errors” while working, they exhibited a Progression difficulty (“Being conceptually behind in the course material but unaware of it due to a false sense of confidence”).

• Participant 11 accepted a suggestion from Copilot even though the suggested line contained a variable that had not been declared; thus, the participant exhibited a Mislead difficulty (“The tool leads the user down the wrong path”). Later, incorrect suggestions from ChatGPT further contributed to the participant’s confusion.

If it were simply the case that some participants struggled while others did not, then that would be business as usual. After all, I believe that the primary purpose of CS education research is to help students in their endeavors.

But GenAI, with its increasing availability and capabilities, introduces complications. In particular, as the paper puts it, GenAI creates an “illusion of competence” and a contradiction between students’ expressed beliefs and their behavior. That is, some students who had struggled claimed that they didn’t really need help from Copilot/ChatGPT, but as far as the authors could tell, they did.

A paper from ICER 2018 (mentioned earlier) describes the results of a similar experiment. There, 11 out of 31 participants did not solve the programming problem within 35 minutes, so the success rate was less than 65%. However, all 11 participants realized that they could not solve the problem after the allotted time. In contrast, according to the authors of the 2024 paper, participants’ ability to assess their understanding was impeded by GenAI: “From the evidence presented above, it appears that most of these ten who struggled thought they understood more than they actually did.” So GenAI might help struggling students in the short run while hampering their metacognitive abilities, which could be detrimental in the long run.

(Of course, it’s possible that the students were fully aware of how reliant they were on GenAI and did not want to openly express themselves. But even if that were the case, it’s probably harder for educators to help struggling students if everyone appears to be doing fine.)

Others had more success

Eleven of the 21 participants did not exhibit any metacognitive difficulties. Participant 19 finished in just 5 minutes; when they were asked if Copilot had been helpful, they responded, “Definitely sped it up. It gives you a lot of the scaffolding you need to solve the problem so you don’t have to worry about some of that.”

Importantly, these participants were able to ignore incorrect suggestions. This relates to a 1994 paper by Marvin Minsky cited by the authors, in which he wrote, “In order to think effectively, we must ‘know’ a good deal about what not to think! Otherwise we get bad ideas — and also, take too long.” Accordingly, the authors suggest the “ability to ignore incorrect or unhelpful GenAI suggestions” as an important skill for novices to develop.

Alternative GenAI tools

Since ChatGPT and Copilot can hinder students’ learning, and banning GenAI is impractical, the authors point to a few “novice-friendly tools” that expose students to GenAI “in scaffolded ways” (e.g., Prompt Problems, CodeHelp, CodeAid, Ivie). They also propose explicitly teaching metacognitive behaviors and skills.

Perhaps someday, it will seem clear that GenAI broadly improves educational outcomes. But as the paper suggests, we should be cognizant of its effects on everyone, including the students who appear to benefit from it.

Multiple studies have demonstrated the “photo-taking-impairment effect,” in which a person’s memory of objects they photographed was weaker than their memory of objects they didn’t photograph. This 2014 article by Linda A. Henkel provides early evidence in the context of a guided tour of an art museum: “If participants took a photo of each object as a whole, they remembered fewer objects and remembered fewer details about the objects and the objects’ locations in the museum than if they instead only observed the objects and did not photograph them.”

Interestingly, in the same study, the photo-taking-impairment effect did not surface when participants zoomed in on a specific part of the object, even for parts of the object they did not zoom in on. This motivated psychologists to develop theories and study the conditions under which the photo-taking-impairment effect occurs.

In this post, we’ll look at one of these papers: “Does taking multiple photos lead to a photo-taking-impairment effect?” by Julia S. Soares and Benjamin C. Storm, published in the Psychonomic Bulletin & Review in 2022. As its title suggests, the paper considers whether taking multiple photos leads to a photo-taking-impairment effect. On the surface, it seems that if someone takes multiple photos of an object, especially if those photos are different from each other, then they might remember it better because they’ve viewed the object in a more prolonged, deliberate manner.

But the paper shows that this is not the case. In three separate experiments, participants who took one photo of an object scored higher on a memory test than participants who took five photos. Participants who didn’t take any photos, and only observed the object, scored highest. At the end of this post, we’ll discuss two potential explanations of the photo-taking-impairment effect.

Experiment 1

Participants were told that they would view some paintings, photograph some of them, and take a multiple-choice test at the end without access to the photos. There were 30 paintings, and each painting was on the computer screen for 20 seconds. Under each painting, there was one instruction for participants: just observe, take one photo, or take five photos. After viewing each painting, the participants were asked, “How likely (out of 100%) are you to correctly answer questions about the painting’s visual details?” And after viewing all 30 paintings, the participants were directed to play Tetris for five minutes. Finally, they took a 20-minute test containing 60 multiple-choice questions (two questions per painting) on visual details in the paintings (e.g., “What phase is the moon in Moon River by R. C. Gorman?”).

Based on participants’ performance on the multiple-choice questions, there was “a significant linear trend indicating that when more photos were taken, memory performance decreased” (see the three leftmost bars in the figure below). (As we’ll se, and as we can see in the same figure, the results from the other two experiments were consistent with this finding.)

Experiment 2

This was a between-subjects design version of Experiment 1 (which had a within-subjects design). More specifically, participants were randomly assigned to one of three conditions: just observe every painting, take one photo of each painting, or take five photos of each painting. After viewing the paintings, these participants took the same test as the participants in Experiment 1 did.

Again, the authors found that “when more photos were taken, average performance decreased.”

Experiment 3

This was similar to Experiment 2, with just one notable change: participants in the “5-photos” group were instructed, “Please take 5 different photos of each painting using the device provided. Try to make each photo different from the others.” (People who took 5 photos in Experiments 1 and 2 were not told to take different photos; many took five nearly identical photos.)

For the third time, there was “a linear trend such that when more photos were taken, average performance decreased.” The authors also conducted a meta-analysis on the findings in all three experiments, and found a “significant and moderately large” effect size for the photo-taking-impairment effect.

Discussion

As mentioned earlier, it might sound plausible that taking multiple different photos of the same object reduces the photo-taking-impairment effect. But in Experiment 3, the opposite was true: the photo-taking-impairment effect was more pronounced for the “5-photos” group than the “1-photo” group.

According to the authors, there are two major theoretical accounts of the photo-taking-impairment effect: attentional disengagement and cognitive offloading.

Attentional disengagement occurs when photo-takers “disconnect from the scene, thereby preventing them from fully encoding or attending to photographed objects.” They focus more on the act of photo-taking, especially if they’re taking multiple photos, than the object itself. In a lecture, if a student spends too much attention on taking detailed notes, they might not remember the actual content of the lecture.

Cognitive offloading occurs when “photo-takers believe that the objects they photograph are saved by the camera, reducing the need to remember those objects internally.” Taking multiple photos could strengthen this belief, thus amplifying the photo-taking-impairment effect. Since participants were told that they would not have access to the photos during the test, my hunch is that cognitive offloading is largely a subconscious phenomenon. Of course, taking a photo allows us to consult it later, but if we never consult it, we might remember less than if we hadn’t taken the photo.

Near the end of the paper, the authors note, “Future work should examine the effects of other common photo-taking behaviors like using filters, using camera modes like panorama or portrait mode, or making other adjustments to brightness or color balance.” It’d be interesting, and amusing, if some convoluted way of taking photos (e.g., decrease brightness, then zoom in and increase brightness) results in improved memory of the object.

But does live coding actually improve student outcomes more effectively than annotating and explaining pre-written code? Anshul Shah, Emma Hogan, Vardhan Agarwal, John Driscoll, Leo Porter, William G. Griswold, and Adalbert Gerald Soosai Raj consider this question in their ICER 2023 paper, “An Empirical Evaluation of Live Coding in CS1.” In short, they report that the answer is no. In fact, the only statistically significant results in their paper indicate potential drawbacks of live coding.

Experimental setup

In their experiment, one instructor at UCSD taught two sections of CS1. The sections were identical (e.g., weekly material, activities, assignments, etc.) except for two aspects: the start time (9:30 am or 11 am) and the presentation of code examples (static-code or live-coding) in lectures.

In the static-code group (the 9:30 section), the instructor “showed students pre-written examples of code when demonstrating an implementation of a concept.” While presenting the examples, the instructor wrote annotations and visualizations.

In the live-coding group (the 11:00 section), the instructor “showed code examples by opening a Python file in an IDE and writing code in front of the students.” While coding, the instructor verbalized their thoughts and demonstrated effective development strategies such as interpreting error messages and using print statements to track variables.

Attendance was consistently near 85-90% in both sections, and the sample size was about 110 students per group. Although students were not randomly assigned to the sections, at the time of registration, they didn’t know that one lecture would be taught with static code and the other with live coding. There were also no significant differences between the two groups in programming experience, demographics, or average GPA.

Questions and results

The paper considers three areas that live coding, in theory, could improve: students’ programming processes, course performance, and lecture experience. The main result of the paper is that in the first two areas, there were no significant differences between the two groups. Moreover, with regard to the third area, more students in the live-coding group thought the lectures were “too fast, did not hold their attention, and did not facilitate note-taking.” Their exact research questions, and some details on their methods, are described below.

Programming processes

“How do adherence to effective programming processes and programming productivity differ between students in live coding and static-code pedagogies?”

The authors examined students’ behavior on various coding challenges held during lectures. More specifically, the authors analyzed snapshots of code obtained whenever a student ran or submitted code to the course’s online IDE (EdStem). Using those snapshots, they evaluated an assortment of metrics related to programmer productivity (e.g., “Measure of Incremental Development,” “Repeated Error Density,” time until correct implementation, number of print statements).

After running statistical tests on those metrics, the authors found no statistically significant differences between the two groups. The live-coding group performed better at a debugging task and had a lower time until correct implementation on the final coding challenge (about 14 minutes compared to 16), but those results were also not statistically significant.

Course performance

“How does course performance on exams and assignments, specifically on code tracing, code explaining, and code writing questions, differ between students in live-coding and static-code pedagogies?”

The authors also measured students’ course performance based on programming assignments, worksheets, and two exams. Every question from the worksheets and exams was categorized as “code explaining,” “code writing,” “code tracing with loops,” “code tracing without loops,” and “basic questions.”

Again, none of the p-values calculated by the authors indicated statistical significance. This included a p-value for every type of question, grades on the programming assignments, and grades on the exams. For example, on the four “code writing” questions on the final exam, correctness differed by just one percentage point between the two groups.

Lecture experience

How does the lecture experience, in terms of engagement during lecture and perceptions of code examples, differ between students in live-coding and static-code pedagogies?

Finally, the authors examined how students felt about the lectures themselves based on their responses to a required mid-term survey and the official, anonymized, end-of-term course evaluation.

In the survey, more students in the static-code group reported benefits related to code comprehension. On the other hand, more students in the live-coding group reported benefits related to programming processes. The latter sounds like a positive effect of live coding, but the authors remind us that “these differences in perceived benefits did not materialize into discernible differences” according to the metrics they calculated when answering their other research questions (“Measure of Incremental Development,” “Repeated Error Density,” etc.).

Furthermore, in the live-coding group, 18% of responses mentioned that the instructor should slow down, whereas this rate was only 2% in the static-code group. (In both groups, about 1% of responses mentioned that the instructor should speed up.)

The only statistically significant results in the paper came from the end-of-term course evaluation. These results showed that students in the live-coding group were less engaged than those in the static-code group.1 In particular, as the authors put it, “students in the live-coding group were more likely to disagree that lectures held attention or facilitated note taking.”

Discussion

In summary, after running various statistical tests, the authors found that live coding was not more effective at improving students’ performance than presenting static code, and students in the live-coding group were less engaged during lectures. The authors allude to context switching and cognitive load as potential explanations:

Live-coding examples require the instructor to move away from the lecture slides and open a new file in an IDE for each example, which leads to an inherent overhead cost of showing the examples…

These results may be due to the instructor simultaneously writing code while also explaining their reasoning and strategies. Students could be unsure whether to write down the code or the instructors’ explanations, potentially resulting in an inability to focus at all…

Ultimately, students may not have been able to fully absorb the programming processes demonstrated in the live-coding examples because of the pace and difficulty in focusing.

The authors speculate that “there may be an additional step between students observing effective programming processes and being able to apply these processes in their own work.” They then cite methods from Cognitive Apprenticeship beyond modeling (e.g., scaffolding, coaching) as possible ways to supplement traditional lectures.

Closing thoughts

My hunch is that the most effective CS1 lectures combine live coding with static code, perhaps in the following way: the instructor writes the code while verbalizing their thought process, then (ideally seamlessly, to reduce context switching) annotates and explains the code as if it had been written ahead of time. (Or maybe it’d be better to flip the order.) It sounds repetitive, but I think it’s useful to repeat concepts, especially when they are new. Even slight changes in presentation can provide additional insight.

Results like the ones in this paper highlight the importance of education research. Live coding might have intuitive appeal, but research allows us to check our intuitions.

• Learning is vital for programmers, but the human mind works quite differently than a computer.

• Understanding how humans learn can help you learn more effectively.

• The Internet and LLMs have not made learning obsolete; learning is essential and takes time.

• Expertise changes how you think, letting you solve problems more easily but also potentially hindering your ability to teach.

Then they list and explain their “10 Things.” In this post, I’ll highlight a few of those things and share some thoughts.

Intrinsic vs. extraneous load

Learning places demand, also known as cognitive load, on our working memories. There are two types of cognitive load: intrinsic load is the information inherently necessary to whatever’s being learned, while extraneous load is everything else. Beginners often cannot distinguish between the two types of load, which makes it harder for them to learn.

For example, suppose a developer is trying to create a database, and they need to learn the database schema. The schema can be described using a diagram, with rows and columns, or as a paragraph (see the figure below). For most people, the diagram is likely easier to process because both the format of a paragraph (i.e., a block of text) and its non-essential content (e.g., punctuation, capitalization) increase extraneous load.

This example reminds me of the difference between pseudocode and executable code: conceptually, pseudocode and executable code convey the same ideas, but the latter often contains more visual elements and distracting details. (For more about this, see this post.)

Experts recognize

Memorization is often viewed as a poor method of learning (e.g., “Don’t just memorize — try to understand!”), but the authors point out that experts have memorized a lot of things, and this “frees up their cognition” when reasoning about new problems. For example, “[a]n expert may immediately recognize that a particular piece of code is carrying out a sorting algorithm, while a beginner might read line by line to try to understand the workings of the code without recognizing the bigger picture.” I think this is particularly clear in sports: when most NBA players shoot the ball, I doubt they’re thinking about “BEEF” (balance, elbow in, eyes on target, follow through) — proper shooting form is deeply stored in their muscle memory.

Experts typically don’t sit down and deliberately try to memorize things. Instead, they accumulate knowledge in their long-term memory through experience. For beginners, this points to a way of improving at, say, recognizing code: simply read and understand a lot of it. An analogy here is machine learning: modern algorithms require vast amounts of training data.

Finally, even though the Internet is huge and AI tools keep improving, the authors argue that memorizing is still important because it allows us to quickly access information while minimizing the cost of context switching. For example, it’s hard for anyone to write complex software if they constantly need to look up syntax.

Spaced practice vs. cramming

According to the authors, the spacing effect is “one of the most predictable and persistent effects in cognitive psychology.” It states that for long-term learning, cramming is less effective than spacing practice sessions across multiple sessions (ideally over the span of weeks). Whenever a learner solves a problem, they match the problem to a corresponding concept in their long-term memory. By keeping this neural connection active over a longer time horizon, learners improve at selecting the correct concept when solving new problems.

But, like muscles, our brains require rest — they need time to develop and consolidate new information. In fact, the authors recommend keeping learning bouts to 90 minutes or less and resting for at least 20 minutes after each bout. They emphasize that “rest” means walking or sitting quietly; they don’t consider chatting, working on other tasks, or browsing the Internet as rest.

Spatial skills seem to transfer

For the most part, there’s no generic “problem-solving” skill that generalizes across disciplines. For example, when a child receives chess or music instruction, their other cognitive skills rarely improve. However, spatial skills (e.g., rotating a Tetris shape in one’s mind) seem to be an exception: according to the authors, “spatial training improves performance on a range of non-verbal skills regardless of initial ability, age, or type of training task.” They note that “[t]his phenomenon is so unusual that it has caused much consternation in cognitive and learning sciences.” But spatial reasoning is not a magic bullet or perfect predictor of success in programming: the authors claim that “these weak-to-moderate correlations largely disappear with increased experience for various reasons.”

Here’s a 2020 paper by Jack Parkinson and Quintin Cutts that suggests that training spatial skills improves outcomes on computing assessments. It also provides an example of a spatial reasoning problem; see the figure below.

Experts are not necessarily the best teachers

Experts sometimes experience a blind-spot problem, in which they fail to see the difficulty of a task because their knowledge has become so automated. Moreover, they might be unable to articulate their reasoning and resort to phrases like, “I just know.” To address these situations, the authors suggest that “a knowledgeable (but still relatively novice) peer” (e.g., a teaching assistant, perhaps) can help “bridge the gap between beginners and experts.”

I think another remedy for the blind-spot problem is having the teacher recall the feeling of being stumped or confused. Many professors likely encounter this in their research, but perhaps pursuing an unrelated endeavor (e.g., shooting a basketball) would be even more effective since that likely matches the students’ experience more closely. Feeling stumped can be demoralizing, but if the professor can relate, perhaps they can help the student learn more effectively.

Pseudocode is more widely accessible.

Let’s say you’re trying to solve some programming problem in Java, and you’d like to implement an algorithm that you found in some research paper. Now suppose the paper uses Python, and you haven’t seen any Python in a while. In this situation, you’ll probably need to look up various idiosyncrasies. For example, what exactly does “for i in range(10):” mean? Does it include 0? Does it include 10? Resolving questions like these requires that you spend additional time and energy on tasks only tangentially related to the main problem, and this detour would be even costlier if you’ve never seen any Python in your life. On the other hand, if the paper states, “for i = 0, 1, …, 9,” you can translate that into Java much more easily. (Perhaps ideally, the research paper includes both pseudocode and executable code — Steven Skiena’s textbook does this.)

This might sound like a superficial advantage of pseudocode, but I believe there’s a deeper phenomenon here. Pseudocode is not just a time-saving device; it helps us grasp the essence of an algorithm by eliminating clutter and highlighting its core ideas. In other words:

Pseudocode better expresses algorithmic ideas.

Here’s an algorithm, similar to Breadth-First Search, implemented straightforwardly in Python. Note that Python is already more pseudocode-like than most programming languages, yet it still provides an illustrative example:

And here’s the same algorithm in pseudocode:

At a glance, we can see that the Python version (1) is longer and denser, (2) assumes that the graph is given as an adjacency matrix, and (3) requires the reader to know (or look up) Pythonic quirks such as “len(G)” and “[False] * n”. These mental hurdles obstruct the path toward understanding the algorithm (i.e., what it’s really doing) by submerging its description in the details of one particular implementation.

More generally, Python demands conformity to a rigid syntax, while the flexibility of pseudocode allows us to directly express the core ideas. For example, “for u in range(n): for v in range(n): if G[u][v] == 1” is a long-winded way of saying “find an edge (u,v).” Similarly, “marked[v] = True” is dry and technical, while “mark v” conjures a vivid image of an action that the reader can visualize themselves doing (e.g., placing a green checkmark on a circle).2 In fact, a picture might sometimes be even better than pseudocode:

Compressing ideas in this manner (executable code to pseudocode and/or picture) frees up space in our mental “working memory,” which is especially helpful when we’re dealing with new information (see, e.g., here and here). It’s like substituting “sodium chloride” with “salt” — the former is more appropriate if we’re concerned about a chemical reaction in a science experiment, but the latter is better for everyday cooking purposes. Similarly, executable code is essential if we want to modify the output of a computer program, but if we’re reasoning about algorithms at a higher level, pseudocode is generally more fitting.

1. Communication: In the short term, grades provide a succinct way for teachers to communicate to students (and potentially their parents) about their progress. In the long term, colleges, graduate schools, and employers rely on grades to make admissions and hiring decisions.

2. Motivation: Learning isn’t easy; as the book puts it, “Learning is an active process that involves invisible work and considerable effort.” Thus, grades “have been used as a way of getting students to take school seriously.”

3. Synchronization: As a whole, the education system has various moving parts: students change schools, transfer to other districts, apply to colleges around the world, etc. Grades help tie the system together by providing schools with a means of synchronization. For example, transfer students typically receive credit for a course if they took the equivalent at their previous school, even if the two offerings aren’t perfectly equivalent.

The book proposes some improvements to the standard grading practices that, it argues, simultaneously preserve the purposes listed above while addressing negative aspects of the status quo (e.g., cheating, teaching to the test, avoiding challenges). I’ll describe some of the proposals below along with my thoughts on how I’ve considered implementing them in an undergraduate CS course.

1: Provide grade-less feedback

Teachers should provide narratives, or “thick descriptions of student abilities,” in addition to regular grades. Unfortunately (like many alternative grading methods), this sounds nice but seems difficult to scale to hundreds of students, since giving good feedback takes time and experience. It’s also quite demoralizing when teachers realize that many students ignore it. Of course these days, there are AI tutors, but I’m not convinced that they can solve the core problems with grading. They’re still (depending on the context) unreliable, and personally, I’m less likely to take an AI tutor’s feedback seriously than an instructor’s. (Of course, AI has been developing quite rapidly, so it’s hard to rule out AI-based solutions completely.)

One thing I’ve done, which is relatively easy to implement, is assigning homework problems that are worth an extremely small (possibly zero) number of points. My hope is that, because these problems essentially don’t affect students’ final grades, students can use them as opportunities to test their true abilities and receive feedback. Usually, when every problem matters for their grade, students are more likely to rely on external assistance, which obfuscates the communication of their true abilities to their instructors.

2: Create digital portfolios

When someone sees a letter grade on a student’s transcript, it’s usually hard for them to tell what exactly that letter grade means. To remedy this, the book proposes that the work itself, not just a grade, should be visible to future audiences via a digital portfolio. The book argues that these “double-clickable” transcripts should include larger projects, skills, and competencies, which can be “interpreted through holistic human judgment” and minimize the negative parts of grading. Teaching to the test might be fine if the “test” is actually a series of interesting projects.

I don’t know what a digital portfolio looks like in an algorithms course, but I think it’d be motivating for students to leave the course with something substantial beyond a grade and a bunch of small, independent assignments. One idea I’ve considered is having students create a video in which they explain the solution to a problem. After all, teaching is a great way to learn, and creating an educational video could be useful career preparation. (According to a 2019 poll, “YouTuber” and “teacher” are the top two career aspirations for children in the US and UK!)

3: Foster intrinsic motivation

As I’ve written about before, external motivation (e.g., grades) can crowd out intrinsic motivation, which many argue is a negative effect of the education system. However, as the authors readily acknowledge, fostering intrinsic motivation is difficult because motivation involves many complex elements such as school culture and family support. Still, they believe schools should try to create assignments that are tied to authentic competencies and lead to longer-term goals. (The opposite would be pure busywork.) Unfortunately, it’s much easier to administer and consistently grade, say, multiple-choice quizzes than open-ended prompts. But I definitely see their point: if a student only engaged with a subject via multiple-choice quizzes, their interest in the subject would probably stagnate (or even decrease).

Instead of focusing on the exchange-value of education (i.e., do these tasks in exchange for an A), educators should focus on its use-value. By “use,” the authors don’t mean “useful for getting a job” but rather, they mean something like “useful for living a good life.” (I suspect that they think “useful for getting a job,” still sounds too transactional.) They use art as an example: for most people, drawing and painting aren’t useful for enhancing one’s résumé, but they can certainly be useful for living a good life.

Similarly, in my opinion, developing an appreciation for theoretical computer science is similar to developing an appreciation for art, jazz, physics, nature, etc. For most people, there’s no obvious quantifiable reason to appreciate any of these things. But, without getting too philosophical, I think that part of “living a good life” is cultivating a sense of curiosity and awe.

4: Use common performance tasks

Almost every school has a course called, say, “Algebra 1,” but these courses vary in content and difficulty. This especially becomes a problem when students change schools. To preserve synchronization across institutions, the authors “envision common performance tasks” that allow us to shift our focus “from the current formalisms to what students can actually do.” Even though some subjects (e.g., history?) seem harder to break into concrete skills than others (e.g., math), I think common performance tasks would be useful for every subject.

In computer science, a few major organizations (ACM/IEEE/AAAI) released curricular guidelines in 2023; here is the report for “Algorithmic Foundations.” In my opinion, their list of topics is quite long, and it’s not clear what the learning outcomes entail. For example, one of the outcomes is “Evaluate whether a greedy approach leads to an optimal solution.” This can be either extremely easy or very difficult, depending on the details of the situation! But still, for the sake of synchronization, I think it’s worth it for CS programs to at least be aware of the guidelines, if not try to follow them.

5: Make grades overwritable

The last proposal I’ll discuss sounds the most radical. The authors use the following analogy: if someone can ride a bike, it basically doesn’t matter how long it took them to learn. Similarly, they argue:

Instead of permanent marks, we can imagine assessment systems that update constantly to reflect what students know and can do today. Can the person in question ride a bike now? If so, previous struggles are suddenly no longer relevant information — they are archival noise that should be scrubbed from the data.

In computer science, there was a paper from SIGCSE 2023 that explores this very idea: “A’s for All (As Time and Interest Allow).” Both the book and the paper make a strong case for overwritable grades, but I worry about the negative consequences (many of which could be unforeseen). For example, the book notes that one potential downside is that students become even more obsessed with getting straight A’s, possibly to the detriment of their learning and mental health. Still, the idea is interesting, and maybe even useful at a smaller scale — I’ve written about one of my own experiments with alternative grading, which included reattempts without penalty.

Conclusion

Grades are deeply embedded in our education system, so if we want to improve grading practices, it’s important to know their history and the purposes they serve. Off the Mark provides a great overview in addition to multiple proposals for improvement. Some of them are pretty radical, but others are relatively easier to implement. I discussed their paper in my last post, and it ends rather pessimistically, so I’m glad that this book provides a nice sense of hope.

Yale is often cited (and cites itself) as one of the original adopters of grading in the American colonies. In 1837, the school indeed tracked grades in individual courses, but interestingly, those grades “were often kept secret from students –– an effort to minimize the day to day competition among them.”

In the early- to mid-nineteenth century, K-12 schools also began issuing grades. One overtly competitive system was called the Lancasterian model: each day, students “were literally repositioned in the classroom –– the top students moving to the front of the class and the less capable students moving to the back.” Proponents of the system argued that the competition was commendable since they believed it motivated students to work hard.

But not everyone felt this way. As educational reformer Horace Mann wrote, “if superior rank at recitation be the object, then, as soon as that superiority is obtained, the spring of desire and of effort for that occasion relaxes.” (In another post, I described some psychology research that supports this claim.) Instead, he favored monthly report cards, which could reduce competition, track students’ progress, and keep parents informed.

Between 1870 and 1910, as a result of compulsory schooling enforcement, child labor laws, and the increased use of education for social mobility, K-12 enrollments almost tripled in size. As a result, administrators needed to develop more efficient managerial systems. For example, professors used to conduct college entrance exams at individual high schools, but that became seen as unfair and required tons of effort. Instead, colleges began to use an accreditation system for high schools. This growth in standardization paralleled similar moves in other industries. For example, as the market for grain expanded, grades and standards were required because personal inspection by the buyer became too difficult.

Despite the efforts at standardization, the meaning of individual grades was (and is) unclear. Teachers assign grades in different ways, schools have different norms, and many scores are near the boundary of two grades (e.g., 89 might be a B+ while 90 is an A-). Many educators, influenced by reformers like John Dewey, denounced the negative effects of grading. But as the authors put it, “grades were not going to be dropped,” because they “were too useful as extrinsic motivation, even if such motivation was seen as having serious consequences.”

Eventually, the grading system became fairly standardized, at least on the surface: by 1971, over 80% of schools used letter grades. However, many teachers opposed letter grades; according to a 1971 survey, only 16% of teachers believed they were the best reporting method for elementary students. Furthermore, many students began gaming the system by shopping around for “easy A” courses, and colleges’ reliance on students’ evaluations of instructors (among many other factors) incentivized grade inflation.1

The paper ends on a rather pessimistic note:

The upshot of all of this is that educators are stuck in a bind… They must find a way to work within a system that is universally accepted –– one essential for national movement, seamless coordination, and seemingly standard communication to parents and outsiders. And, at the same time, they must find a way to keep students focused on learning and not merely on a set of measurable outcomes loosely connected to the process of education.

Fortunately, the authors’ book Off the Mark was published last year, and according to the official blurb, it offers “a range of practical reforms.” I haven’t read it yet, but I probably will, and I suspect I’ll write about it afterward.

I plan, for as long as this newsletter lives, to make the last post of every year a 100-word (or less) summary of each post published earlier that year. Thanks for reading!

Subscribe now

In chronological order:

1: Bloom’s 2 sigma problem

Tutoring is highly effective for students’ learning, but it’s not easily scalable, so Benjamin Bloom sought alternative ideas. For example, Mastery Learning (the idea that a student shouldn’t learn Unit 2 before they’ve mastered Unit 1) is pretty effective, but it’s tricky to implement since students progress at different rates. He also demonstrates the effectiveness of teaching “higher mental processes,” which is often found in non-US countries (see #8 below).

2: Drawing for rewards

Education systems often use grades and other extrinsic rewards (e.g., stickers, pizza) to boost student motivation. However, results from studies in psychology suggest that extrinsic rewards can “crowd out” intrinsic motivation; as a result, many believe that extrinsic rewards are required for motivation. Once a reward system is in place, it’s hard to break out of this cycle — Alfie Kohn has written quite a lot about this.

3: Lessons from dog agility

People who pay to learn, outside of traditional classroom settings, often enjoy their experiences while learning. One reason could be that they have intrinsic motivation (see #2), but there could be external, pedagogical reasons as well. For example, dog agility classes contain breaks, social interactions, and physical activity — there’s some evidence that these components are beneficial for learning in general. More broadly, I wonder what aspects of non-traditional learning settings could benefit traditional classroom settings.

4: Alternative grading in algorithms

For better or worse, grading schemes heavily influence students’ experiences, so it’s worthwhile to consider various approaches. This post describes a few things I tried at Elon University that seemed to work reasonably well. One idea I liked is reducing the number of possible outcomes for open-ended problems (e.g., each problem is worth 3 points rather than 20). In my experience, this makes grading more efficient and consistent.

5: Delbanco's three cases for college

“What is the purpose of college?” is a big question, and Andrew Delbanco gives three answers in his book, College: economic investment (both for individuals and nations), democracy (e.g., the Founding Fathers were big fans of education), and liberal arts (e.g., developing one’s mind). In my opinion, it’s nice to have stated goals, even if there’s tension within them.

6: Problem solving vs. career training

In the 1970s, two computer scientists gave two (somewhat competing) visions of an introductory programming course. (Both papers won a SIGCSE “Top Ten Papers of All Time” Award.) David Gries thought CS1 should emphasize problem solving, describing algorithms, and verifying correctness, while G. Michael Schneider was more in favor of teaching students in “realistic programming environments.” Today, many schools have two versions of CS1: one for future CS majors, and one for non-CS majors.

7: Phonics and systematic instruction

CS education hasn’t been debated as much as reading — the “reading wars” have been going on since at least the 1960s. Basically, team “phonics” claims that teachers should explicitly teach the connection between letters and sounds, while team “whole language” claims that teachers should facilitate the discovery of words in picture books and other contexts. My understanding is that phonics has “won,” and I wonder if there are analogous approaches that are similarly effective for teaching other subjects, such as math and CS.

8: The Teaching Gap

This book, published in 1999, describes the differences in math education in the United States, Germany, and Japan. One finding is that the classrooms they studied in Japan tended to focus on “structured problem solving,” while those in the US focused on “learning terms and practicing procedures” (see #1). The differences are roughly analogous to the debate on reading (see #7). My favorite part is their claim that teaching is a cultural activity — I definitely agree, but unfortunately, it makes substantial reform appear difficult.

9: Parsons problems

In a 2008 paper (which also won a SIGCSE award), Paul Denny, Andrew Luxton-Reilly, and Beth Simon proposed adding Parsons problems to CS1 exams. A Parsons problem gives students a programming problem and a set of code fragments, and it asks them to solve the problem by selecting and ordering a subset of the code fragments. The paper suggests that Parsons problems and code writing require the same cognitive skills, but the former are easier to objectively grade and give partial credit.

The paper proposes adding Parsons problems — named after Dale Parsons, who co-authored this 2006 paper with Patricia Haden — to CS1 exams. A Parsons problem is the following: the student is given a set of code fragments and a problem, and the student must select and order a subset of the fragments into a correct solution for the problem. (People have studied various variants of Parsons problems; here is a survey from 2020.)

Denny et al. report on two studies about Parsons problems: the first was an interview of 13 students who recently took CS1, and the second was a quantitative analysis of 74 students’ performance on three tasks: code writing, code tracing, and a Parsons problem.

Study 1: Interviews

The 13 students including 6 women, were North American undergraduates “approximately four weeks into a second computing course following a Java, objects-first CS1.” One group wrote code while thinking aloud, solved Parsons problems, and wrote code again; the other did the same except instead of writing code (at the beginning and end), they traced code.

Clearly, there wasn’t a ton of data, but the authors still had some observations. I won’t describe them all; here are a few that stuck out to me:

1. Many students, especially the ones who performed better, strongly preferred writing their own code to solving Parsons problems. They felt that Parsons problems reduced their “freedom” by forcing them down a path toward one particular solution.

2. Students had “a surprisingly strong uniformity of dislike towards tracing questions on exams” because tracing code gets “annoying” and “tedious.” To them, Parsons problems are “not so bad” compared to tracing code. Perhaps relatedly, they weren’t great at the code tracing problems, and the correlation between performance on the those and Parsons problems was quite low. One student even “refused to trace the code.”

3. Some students, even after correctly solving a Parsons problem, did not fully understand the solution to the problem. For example, they knew that “(r+c)%2==0” is a line in the correct solution, but only because they traced an iteration or two, not because they truly understood what the line does.

So far, the case for Parsons problems does not strike me as extremely strong: students thought they were were worse than code writing and better than code tracing, but that’s clearing a low bar. (One student said, “In the real world, you wouldn't trace the numbers, you'd run the program.”) And based on the authors’ observations, the Parsons problems were not effective at imparting “a full comprehension” of the code.

Study 2: CS1 Exam

As you can probably guess, the second study paints a rosier picture of Parsons problems. This one had 74 students in CS1 at the University of Auckland take a final exam worth 60% of their final grade.1 The exam had 11 questions, including one on code writing, one on code tracing, and a Parsons problem. The three questions were conceptually related and all of them involved looping through an array. Figure 1 from the paper is the Parsons problem from the exam.

One benefit of Parsons problems is that they help instructors categorize errors by type. For example, in the Parsons problem above, some pairs (2, 4, 6) contain a syntax error while others (1, 3, 5) contain a logic error. If instructors observe that logic errors are more common than syntax errors, then that can inform their teaching. In contrast, in a solution to a code writing problem, it’s often unclear if an error is a minor typo or something more significant.

Parsons problems also allow students who struggle with the code writing problem to demonstrate some understanding of the material. In Figure 9 from the paper, we see that students in the lower quartile (according to full final exam score) scored higher in the Parsons problem than in the code writing.

Furthermore, among (1) code writing, (2) code tracing, and (3) the Parsons problem, the two that had the highest correlation in performance were (1) and (3). To the authors, the r-squared values “suggest that code writing and Parsons problems require the application of similar skills, whereas code tracing requires a quite different set of skills.” If that’s true, then Parsons problems would be a reasonable substitute (or supplement) for code writing on exams, with the benefit that grading the former is easier and less subjective. Indeed, the authors' rubric for the Parsons problems was much simpler than their rubric for code writing, and it resulted in fewer discrepancies among the three authors after they individually graded the submissions.

The authors acknowledge that students can guess their way through a Parsons problem, but state that this effect can be ameliorated by asking multiple Parsons problems. They also point out that in their interviews, the students who couldn’t begin answering the code writing question performed better on the Parsons problem than they would by purely guessing, which again suggests that Parsons problems allow students who struggle with code writing to demonstrate some knowledge of the material.

Thoughts

The challenges addressed by Parsons problems are familiar to me. In my experience, it’s unusual for a student to make partial progress when solving a problem in mathematics or a related subject like algorithms.2 Instead, it often feels like “you either get it or you don’t,” which could discourage students and reduce their self-efficacy. Parsons problems could help mitigate these effects by giving students an additional opportunity students to display some understanding.

Additionally, I’ve often felt that grading open-ended problems (e.g., “Design and analyze an algorithm that solves the following problem…”) can take quite some time and energy, and it’s impossible to make a completely objective rubric. Since Parsons problems have a smaller number of discrete outcomes, they’re easier to grade and are less prone to subjectivity.

So I think it’d be beneficial to incorporate Parsons problems in other CS courses such as algorithms. In fact, the paper suggests mathematical proofs as an “area of possible potential similarity.” Additionally, in a 2023 paper by Erickson et al. that documents their effort to “develop auto-graded scaffolding exercises for an upper-division theoretical computer science class,” the authors describe one of their components as “similar to Parsons programming problems but at the level of pseudocode instead of executable code.”

Finally, I find results on Parsons problems interesting from a more psychological level because they question the intuition that the best way to measure students’ ability at X (e.g., writing code) is to ask them to do X (e.g., on an exam). It makes me wonder if we can similarly question the intuition that the best way to improve at X is to do X.