Day 2! We started bright and early with the first paper session at 8:30. Katherine, who I had met on Monday, was to kick-off the day in paper session 4.
You can find the work of Katherine and her colleagues here.
The title of the work is relatively self-explanatory, the idea was to find out how students who worked on research projects in groups, experienced their group dynamics. These were second year students, mentored in groups of 4. The experiences of the students were analyzed through a questionnaire with open questions and Likert scale questions, taken at two points during the course. 106 students filled in at least one questionnaire.
From the quantitative analysis, they found that group experiences were overwhelmingly positive. The qualitative analysis found that these experiences centered around three themes: group fit and belonging, emotional and academic support, and logistics. This last one includes division of labor, scheduling of meetings and communication.
In conclusion, group-based undergraduate research experiences seem feasible and have a positive impact.
The second paper by Atira, Dustin and their colleagues can be found here.
The previous paper focused on the students' perspective on undergraduate research experiences, this paper is exploring the faculty perspective. The quote in the paper title was a good introduction, one that made several people in the room laugh and agree. It tells us much of the story that is held within!
So, to summarize the contribution: research experiences are valuable for the students' development, as well as for student retention. However, what do faculty think of it if they can just consider their own workload and efforts? To explore this, the authors ran an interview with 12 faculty, who each supported at least 1-5 students per period. They identified three misalignment during these experiences, between:
This last point specifically reminded me of the paper “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI by Nithya Sambasivan et al. We have used this paper in our Knowledge Engineering course, to show our Data Science students that, although the model is important, the data is maybe even more important. And focusing on the data is essential in order to create systems that work correctly. Data work is perhaps more boring, according to some, but it is the absolute core to any conclusion being drawn from the data.
Then it was time for the final lightning talk and poster sessions, this time by the attendees of the doctoral symposium. There were two that specifically caught my attention.
The first was by Mohammed Hassan of University of Illinois, with work titled How do we Help Students "See the Forest from the Trees?". His proposal is to work on exploring the mental characteristics that come into play when learning to read code. He states that we are aware of several theories and techniques for reading and interpreting code, such as transitive inference, conservation and reversibility, but we do not know how to teach these things to students.
Mohammad's previous work has been very interesting as well, with a paper on reverse-tracing at ICER21 and a paper on debugging code at SIGCSE22. Unfortunately I was not able to speak with Muhammad at the poster, as it was incredibly busy and I joined the line too late. I should probably reach out to Mohammed and Craig to see what they think of applying these techniques to SQL!
The second talk and poster of interest was by Sanaa Algaraibeh of University of Idaho, titled Techniques for Enhancing Compiler Error Messages. She has analyzed learners' experiences with error messages, and has proposed a technique to parse source code in three phases (functions, function bodies, and line-by-line), such that the compiler can generate better syntax error messages. The example error message looks very nice, and made me curious to see more. Their next step is to evaluate this technique in the classroom. I was also not able to talk to Sanaa, and I should reach out to her too, given our current work on SQL parsing and error messages.
Our next paper session, just before lunch, was on how groups and teams (don't) work in the context of computer science courses.
This paper by Lauren and her colleagues can be found here.
The work focused on evaluating whether current models of self-regulation and co-regulation match students' expression of their learning strategies, and how these learning strategies relate to student performance. To do this, they used a repeated questionnaire (midpoint and end of the course) containing 8 questions, containing 3 quantitative and 1 qualitative question for both self-regulation and co-regulation.
From the qualitative questions, they identified a couple of themes that describe students experiences regarding co-regulation:
This work reminds me of many things I have learnt during teacher training courses. They key idea is that active learning elements, where students communicate and reflect, may help them learn better. This seems to be represented from the students perspective as well in this paper.
This work by Kai and his colleagues can be found here.
The work is similar in nature to the work by Lauren and her colleagues, with as major differences that Kai and colleagues used interviews over questionnaires, and that Kai and colleagues analyzed a group project, instead of individual assignments worked on in a group. So, although both were examining how students worked in groups, the focus was different due to the different nature of the projects.
The idea behind this study is to analyze how students ran their teams, what their problems were, and how they were able to overcome these (or not).
They found there were three main challenges faced by their student teams: communication difficulties, time management and task planning. Teams were able to (partially) overcome such challenges through the application of self-reflection, such as identifying the need for more communication, or through external pressure such as deadlines and grade reduction. Successful teams were those who communicated, collaborated and held themselves accountable.
It seems that one of the most important characteristics of well-functioning teams is reflection. This is extra important in case of challenges. As such, Kai suggested that group projects should use tooling that can help teams track their issues and communication. This can help them give insights, and as such requires less reflection (which we know can be difficult for undergrads!)
After another plate of lovely Italian pasta for lunch, it was time for more paper presentations! The next session contained two papers again, both on notional machines.
The title of this paper is a bit of a mouthful, but the core of it is that Gayithri examined notional machines that were used by highschool teachers in a course on electronic textiles. How do teachers apply these notional machines within the classroom, to help students learn?
To identify notional machines and their impact on learning, Gayithri observed 37 class periods at a public charter school. (One of the attendees helpfully shared the wikipedia article on Charter Schools for more information) Of the 37 class periods, she analyzed 8 periods that were heavy on discussion of programming concepts.
She found that students actively engage with the teacher's notional machine, questioning them, adopting them, and giving suggestions for similar mechanisms. Notional machines seem to have a mediating role in the classroom, and they help the class to make sense of the concepts together.
There was lots of discussion at the tables on whether these metaphors should be called notional machines, or that a notional machine is the model of the 'whole', and not just the parts. Via discord, we were presented with alternative terms from other areas of study as well, such as didactic reduction and boundary objects. Nevertheless, I found this methodology inspiring. Notional machines are still a very vague concept, but by identifying the metaphors used in the classroom, we can get closer to a concept representation that is accurate. I'd love to run a study like this for SQL commands, to see how various teachers explain these concepts!
In the paper, they examine to what extent students in upper-level courses understand the concept of runtime stacks. To this end, they asked students to draw representations of pieces of code at some point during execution. They find that students have a very fragile understanding of stacks, and the knowledge they do have often does not generalize. One example of a problem they identified is in the characteristics of a stack: it can push and pop. However, in their explanations, the students never used pop!
After another coffee break, it was time for one of the big events of the conference: the Keynote! It was planned in the middle of the conference, such that it was available to most of our virtual attendees as well.
The keynote was on Manu's key research direction, which is productive failure. To introduce the topic, he gave us a nice example. Suppose we have two groups of kids, in two different rooms. We're introducing a new toy to them, to play with. For the one group, we demonstrate how to play with the toy, for the other we will not. Which of these two groups will be more creative?
Of course, telling kids how to play with a toy boxes them in. They will rarely go beyond what you have shown them. However, this is exactly how we teach our students! This fails to develop creativity and flexibility in our student populations.
So, what is the alternative? We can design for students to fail through the concept of Productive Failure (PF). The opposite of this is our typical way of instructing: Direct Instruction (DI). Research has shown that Productive Failure is better than Direct Instruction for transfer and conceptual knowledge. If it is done correctly, it can be three times as effective as normal teaching. This can be reached with even a small intervention: a recent study of a 7 hour intervention (out of 200 hours in the course), improved passing rates significantly (a paper on this is on its way!).
However, you cannot randomly give students a task and let them fail. PF is about more than a difficult task. If a task is not engaging, students will not want to try it. The four keywords Manu used were activation, awareness, affect and assembly.
An interesting question from the audience, that I was also wondering about, is how to scale Productive Failure to more complex topics? Manu referred to the meta-analysis they had undertaken, where they found that PF does not work well for domain-general skills. What does this mean? Wikipedia tells me that domain-general skills involve memory development (association, generalization, recognition and recall), executive functions (working memory) and language (attention, perception, remembering). I'm not sure what to conclude from this.
On the other hand, the work by Phil Steinhorst, on Investigating Productive Failure in Computer Science, part of the Doctoral Consortium, did show the application of PF in more advanced courses. I talked to him a little bit at his poster on Monday, and they introduced single PF exercises in courses such as Operating Systems, and Pattern Recognition. Neither the poster, nor the abstract, had much information on what these exercises looked like, but perhaps I should dive a bit deeper in Phil's work to see if this could be applied for SQL as well. The biggest question I have is, (how) can you apply PF for language-related questions with a specific syntax?
The final session of the day was on Teachers and their backgrounds and emotions surrounding teaching.
You can find the work by Xiaohua and Felienne here.
In China, since a few years, programming languages and AI have been adopted as a core curriculum activity in primary and secondary school. There are no dedicated CS teachers, so STEM teachers are responsible for teaching these subjects. As they are not specifically educated for this job, it is interesting to examine what these teachers' background is, and which teaching practices they apply.
They found that close to forty percent of their participants had majored in CS, and 23 percent had prior experience with Scratch as language of instruction. But none of these had a statistically significant effect on teaching practices.
With regard to which teaching practices they applied, most practices were applied often or always by more than 50% of participants. The only practice that ran below 50% was I let students practice similar tasks until I know that every student has understood the subject matter. The most often applied practice was I set goals at the beginning of instruction.
This work by Jayne and Amy can be found here.
The question Jayne and Amy set out to answer is, what makes teachers decide to (not) become CS teachers? To this end they interviewed candidates in their Masters in Teaching program. Jayne herself was also in this program, which helped with recruiting and rapport, leading to high-quality data.
Ten candidates were interviewed, of whom five were interested to become a CS teacher, and the other five were not. In the interviews, they identified four groups of factors that influenced these decisions:
An interesting question from the audience was regarding burnout. In the US, apparently there is typically only one (part-time) CS teacher per school. Being the only teacher likely also has an effect on burnout. How do they support for this? Amy mentioned the Computer Science Teachers Association (CSTA) that they try to interest the students for. Plus, the aim is to embed the students in their cohort, such that they can rely on one another.
After such an interesting close of the day, it was time for a little break, after which we had the banquet. I managed to quickly drop off my stuff at the hotel before we went up the mountain. It was incredibly hot, especially in the funicolare, but we were rewarded with nice food and a great view!