Day 2
Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks
Paul Denny and many other authorsIn this study, Paul and colleagues characterized prompt-based exercises and students interactions with them. This type of exercise could be an introduction to programming, learning to interpret and read code before having to write it yourself. They found that students felt prompt-based coding was simpler than pure coding, and that it helped them focus on problem-solving. However, students were not super good at it: their prompts were incomplete, ambiguous or incorrect (as they misunderstood the problem). Luckily, there seemed to be a learning effect, where students who made a certain mistake (primarily omission) were able to internalize this and make that mistake less in later exercises.
Paul presenting their design implications.
A subset of these authors have also worked on another paper on these exercises, in which they injected errors if the student got the code prompt correct on the first try, in order to try to get students to take over the problem solving.
Poster session
Then came the poster session with the work I was presenting. I had purposefully left one third of my poster empty with space for feedback. Luckily, I had some people show up and share ideas with me, although not as many or as varied as I was hoping. However, I do now have some new ideas for how to tackle quantitative evaluation of constructive alignment.
Snapshot from the session. Photo courtesy of Sophia Krause-Levy.
"Why Put In This Much Effort?": How AI Availability Shapes Students' Motivation in Introductory Programming
Keith Tran, Colton Harper, and Thomas PriceIn his presentation, Keith explained the constant temptation of genAI use for homework that our students face every night. But a student is not binary for or against its use: each student has motivated and unmotivated moments. In this work, the authors studied motivation using the framework of situated expenctancy-value theory, interviewing 13 students in a semi-structured manner. They found that 12 of those students are primarily motivated by learning, but even they struggle at times to be motivated enough to do the work themselves.
Students in this study mentioned that effort is not only devalued by the student's use of AI themselves, but also by other people using it (as it does not seem difficult anymore, and they can still get a high grade without). Interestingly, of the 12, 5 students admitted to using AI in ways that clashed with their stated values. The paper mentions that students justify this to themselves by describing the applicable policies as grey areas. The analysis also shows that these five are students with less programming experience. My question then is, are the students' values not fully internalized? I wonder how long students were holding the perspectives they shared in the interview.
I fully recommend reading the whole paper if you are interested, the qualitative analysis surfaces a lot of interesting insights.
"Both a powerful tool and also a trash fire": investigating the outlooks, practices, challenges, and debates of computing education in the Digital Humanities
Anna Sollazzo and Quintin CuttsThis paper won the best paper award!
In this work, Anna presented a characterization of computing education in the Digital Humanities, by means of 27 instructor interviews. The interesting aspect here was that instructors are not quite aligned on contents or methods. There is a broad scope, for example, a lesson could include project-based learning, contextual learning, process-focused assessment, etc. The underlying principles could focuse on tools or first principles, could be vocationally-driven or exploratory, could be machine-agnostic or machine-oriented. Overall, this paper reminded me a bit of my first explorations in my PhD: so little we know and so much to uncover! I hope that, like my work, this paper brings an increase in interest to CSEd for Digital Humanities.
Code as Anchor, Memory and Metaphor as Support: Learner Experiences with Multi-View Visualizations
Naaz Sibia and many collaboratorsThis excellent presentation by Naaz described work in which students use three representations for code comprehension. The task is a think-aloud, and the students are offered the code, a memory representation, and a metaphorical representation. Results by means of eye-tracking show that students spent the least time looking at the metaphor image, and the students without experience looked even less. The students mentioned that they felt the metaphorical representation was childish or insufficiently rigorous, but I'm wondering whether another part of the equation is that it was just easy to intepret? Once you understand what something says, why would you look at it again extensively? Vice versa, perhaps the code was the hardest thing to understand, and that is why it was looked at most? It would be interesting to study this in more detail, as the eye-tracking information can be interpreted in multiple ways.
Naaz presenting their takeaways.
Planning on Paper: Problem Decomposition with Diagrams in Introductory Computing
Annapurna Vadaparty, Devamardeep Hayatpur, Gerald Soosairaj, Leo Porter, and Dan ZingaroThis was one of my favorite papers of the conference, as it closely aligns with my interest in terms of type of work and concepts under study.
In this study, Annapurna and her colleagues utilized a complicated programming problem, an evil wordguesser task (check the paper, it's fascinating), as a decomposition problem for their students. The students were given a description of the problem in words and had to draw a diagram on paper that represents how they would implement it. The advantage of this method is the freedom of paper, which does not offer or impose structure.
In terms of diagram contents, they coded node types, node purposes, and relation types. Nodes could be functions, fragments, data or annotations, and the authors found 21 different purposes for them, such as prompting the user or checking the win condition. Relations were either hierarchical or sequential.
There were sixteen different types of issues in the decomposition. The most common issue was missing functionality, where the decomposition was incomplete. Other issues included clashing notations, hardcoded functionality, poor encapsulation, and vague language. Annapurna hypothesized that one of the major causes of these issues might be competing mental models. Students also seemed to not trace their diagrams for evaluation, as that action could have resolved some of the issues.
This paper is a really interesting contribution as it argues teaching decomposition and problem solving as a skill separate from implementation. Given the reliance of students on generative AI for code generation, perhaps we can instead ask our students for such decompositions more often, assessing understanding of the required structure, rather than the exact code snippets? The 'process over product' school of thought aligns well with this work.
Impression from the banquet. Photo courtesy of Sophia Krause-Levy.
Thursday ended with my birthdayparty at the banquet. For this, we all moved over to the castle, which is a pink building that overlooks all of Uppsala. The banquet was well-organised, with three courses broken up with entertainment in the form of singing, musical entertainment, speeches and a mentalist. I think this is one of the best banquets I've participated in so far, really enjoyable. Thank you to all contributors, and specifically to Johan and the Marmelade Orchestra.
Day 3
Live Coding in the CS Classroom: An Interview Study of Instructor Practices
Daniel Manesh, Vee Pettit, Yuhang Zheng, Yan Chen, David Smith, and Sang Won LeeDanny presented a characterization of live coding practices based on interviews with 22 instructors. Live coding is very versatile, and as such, adopted by instructors in many different ways. They use it primarily to either introduce new concepts, or as integrative practice of concepts taught previously in some other manner. There are also plenty of participation opportunities that teachers can facilitate, but active participation remains a struggle for most of the lecturers in this study. There is also a juxtaposition of teachers who like the students to type along (for staying on task and encouraging experimentation), while others are against this (due to cognitive overload and the possibility fo transcription errors). These teachers might or might not upload the code solutions to the LMS.
The table discussion I participated in revolved around how live coding is not (always) a good pedagogical tool. As a teacher, you have to make really clear to your students what you want them to do and what they should take away from the practice. Also, shout-out to this paper by Christopher Perdriau from day 2. Christopher discussed how live coding is not feasible for Deaf students, as there is a split attention effect and there can be significant lag from ASL interpretation.
I have personally not used live coding in my classes, but one of the things about it that I like, is that it shows students that everyone makes mistakes. I'll have to see if it is something that could have its place in one of my lab sessions.
Talk, Tech, and Togetherness: Ethnographic Insights into Siding in Introductory Undergraduate Computer Science
Kristin Fasiang, Melissa Chen, Darren Gergle, and Eleanor O'RourkeMy second favorite paper from ICER this year.
This paper was about siding: interactions in the classroom that are off to the side of the main learning activity, but still support learning, such as side talk, gestures, and passing notes. Siding is another characterization of how students learn collaboratively.
Kristin performed ethnographic observations during CS1 lectures. The lectures specifically offered time for collaborative working, as this was not facilitated or possible outside of lecture times. This meant students would split up and sit across the room in groups, and Kristin would join and observe them. This paper gave me a new-found appreciation of siding, as Kristin's analysis is showing some very important social behaviors. Unfortunately, students often do not appreciate other students' siding during lectures. Besides the peer siding, which is direct student-to-student, there is also sociotechnical siding (which includes an interaction with a technical resource such as a news website article or a meme), and digital siding (for example with an LLM). As siding is facilitated by the classroom setup, maybe building in some space for this during teaching would be good for engagement and connection between students.
Kristin discussing how to facilitate siding.
Fast and Forgettable: A Controlled Study of Novices’ Performance, Learning, Workload, and Emotion in AI-Assisted and Human Pair Programming Paradigms
Nicholas Gardella, James Prather, Juho Leinonen, Paul Denny, Raymond Pettit, Sara RiggsIn this work, Nicholas compared human-human pair programming to human-AI pair programming, a reality for many of our students nowadays. They set up 11 pairs of students who did not know one another, and had them complete programming tasks under time pressure. The participants completed tasks both human-human as well as human-copilot, in a counterbalanced setup. The authors measures performance scores through unit test performance, and workload and emotion as self-reported measures.
One month later, the same participants were invited once again, to perform the same programming tasks (but the participants did not know beforehand that they were the same tasks). A learning effect was then calculated based on a comparison of score of the first session. The authors found that copilot reduced workload, but had a workse emotional effect. Copilot also helped in performance, but maybe not for learning, although the latter was not a statistically significant effect.
One interesting discussion question that came up at some other table (sorry, I do not know who to attribute it to), is whether it would be less socially acceptable to reproduce the code created through the human collaboration? This could have an effect on the performance in the second session, that could potentially partially explain the lack of learning effect.
Nicholas presenting results on emotion in pair programming.
Evaluating LLM-Generated Contextualized Algorithm Design Problems
Erica Goodwin, Katherine Braught, Jonathan Liu, Dip Kiran Pradhan Newar, Yael Gertner, Seth Poulsen, Diana FranklinThe final paper I'd like to discuss is this work on algorithm assignment contextualization. The idea is that algorithm education can easily be contextualized as the problems can read as little stories. The authors were hoping to do this generation with LLMs. The study was done across multiple stages. First, they worked with simple prompts, but this was not feasible: 10% of the problems were unsolvable, 48% had a difference leading it to test a different concept than instructed, and 20% contained ambiguities.
Moving on to more structured prompts in phase two, contextualized to student themes and using the knowledge gained from phase one, the authors found that the problems were now of high quality, but of mixed thematic quality. So, to do this better, they introduced a third phase, in which they introduced culturally and locally relevant themes as generated by a diverse set of contributors to this paper. All members contributed these descriptions in their own words, in the categories places, holidays, community events, and foods. They found that now the contextualization was applied more deeply but less realistically (only 53% is realistic) than in phase 2. Member checks could mitigate this, but that always requires manual evaluation. On the other hand, this could result in a question bank that is better than the current assignments.
In the discussion session we debated the risks of employing unrealistic, or superfically contextualized assignments. They could lead to negative experiences, which could be worse than the (hopefully) neutral experiences associated with less targeted contextualization. Perhaps we could make some vignettes for our classrooms that allow to substitute different contexts in the 'fill in the gaps'.
Closing
In the closing we learnt that next year, ICER will be going to California, specifically to Sacramento. I am very unlikely to go, something significant has to change in the US for me to consider it. Therefore, I enjoyed myself twice as much this year, to make up for having to miss this great conference once again in 2027. Calkin and all the other organisers did a really great job making us all feel at home in Uppsala, many thanks!
(Almost) all attendees. Photo courtesy of Sophia Krause-Levy.