I've been trying to move more toward Human-Computer Interaction work, and this week marked a renewed interest in this move with events from different special interest groups. Most interestingly, on Tuesday I attended a symposium coorganised by ICT-Research Platform Netherlands (IPN) and the Dutch CHI chapter.
I was pointed toward the event by my collaborator Laura Koesten, who also gave a talk there and kicked off the event.
Laura Koesten - A Human-Centered Perspective on Data-Centric Sensemaking.
The central thread in Laura's work is that data do not speak for themselves, we have to process them in some way to make meaning. For example, we can organise them into summaries, models, charts, clusters, or annotate them.
One central tenet of her research is on dataset summaries. These are an essential way to study data sensemaking, as summaries require multiple steps toward interpretation that are then expressed and can be evaluated. In the first studies, they did this in a written manner. They felt this was a heavy cognitive task, so later studies worked with verbal explanations instead. This introduced a personal experience dimension into the expressions, which they found also shaped how participants engaged with the data.
Another tenet is the use of visualizations for sensemaking. These present an entrypoint to the data, an imposed organisation to help users understand a set. However, are the messages that we intend to communicate with the visualizations also the messages that the plot's users arrive at? Laura and her collaborators studied many aspects of visualizations across separate studies. One study showed that meaning does not only arise from the chart, but also from the reader and the take-away message. Another study showed how important titles and other annotations are for interpreting charts. In a study with journalists and their visualizations from news articles, the researchers found that -although journalists might like to pretend so- a general purpose audience does not exist. However, this was a nice learning process for the journalists, as returning users' interpretations of the plot led them to new insights and a full redesign for the plot.
Laura presenting on journalists' visualizations.
Finally, it was really fun to see Laura present our work on collaborative conceptual modeling. I had missed our CHI presentation this year in Barcelona, but now I had the opportunity to see her highlight what she thought were the primary outcomes. She highlighted the following two findings:
- There were two roads to model emergence: either our participants followed traditional data modeling steps, or they followed a process closer to storytelling. The latter behavior mostly occured for participants with little computer science experience.
- We identified many differen collaborative patterns. Even within one theme, such as affirmations, there were loads of different applications.
David Ayman Shamma
The second talk was presented by Ayman, and I unfortunately did not manage to catch its title. The overall theme of the talk was to highlight many different studies he and his colleagues have done surrounding climate data. Some examples include interactions for carbon reduction, vehicle optimization including eco mode use and EV batteries, and designs for ride green for Uber.
Overall, it seems like the general user is not aware of how to interpret many such greener measures. Instead, it seems they anchor onto anything that presents numbers. For example, there was a study on whether participants would choose a greener alternative based on how the environmental savings were presented. Elements such as number of trees, bags of waste or home energy seem hard to understand. More directly comparable metrics such as pounds of CO2 made it much easier for them to choose.
Heloisa Candello - Invisible interactions: Designing for context, conversations, and technology experiences.
The third talk was by Heloisa, who works in and studies the Brazilian population. Her work centers around the invisible parts of interaction that might take place but are missed in observation. Some parts of interaction are really tangible: clicks, taps, queries, system output, logs and session data, for example. Other elements might present the metadata perspective and are more easily missed. Some examples include why it happened, who was there, what approaches were tried and what were some of the workarounds, and most importantly who and/or what is not there?
Heloisa presenting on news articles describing incidents caused by disregarding the invisible.
Heloisa presented three layers to categorize this information:
- Context: where, when and with whom
- Practices: sequences, workarounds, coordination, routines, negotiation
- Exclusion: what is left out, what assumptions were made, which access barriers exist, who has agency, what kinds of invisible labor are involved?
She then presented three case studies, each of which centered around data interactions and what turned out to be invisible.
The first case study involved a museum in which they set up an interactive exhibition in which visitors could chat with characters from a famous novella. They studied engagement with the chat, especially in cases where the chat would break (in which case it would say something like "time for more coffee"). They found that users who were attending with their family would continue engaging after failure, as would users who would be called by name. However, the intersection of those groups would actually stop interacting. It turned out that the way the experiment was set up did not take into account group interaction, and only allowed the input of a single name, leading to disengagement of the whole group when one person's name was called.
The second case study was about content curators for chatbots, who would be called upon when a bot would not know the answer to a question. They studied how these curators developed democratic practices, and found that they had a lack of direct access and not the right tools to investigate the context of the chat. The invisible elements are large in this case: the workers themselves are invisible, only seen as question answerers. The practice is invisible, there was no way to see when and how often various answers were updated, by whom, and for waht reason. And the interactionw as invisible, it was seen as background work.
The last case study was on the use of micro credits and their use in Brazil, and the impact of AI on these communities. Applying for microcredits can be challenging and scary. Potential difficulties might arise from small business practices such as informal employment situations, outstanding debts, or lack of financial guarantor. To support these small business owners, Heloisa and her colleagues investigated whether conversational AI could support them by evaluating business health, which the owners could then take to the bank to help them communicate the promise of their business. Ih this case, the invisibilities are communicated by means of design guidelines, with factors including identity, coercion, and accountability.
I really appreciated this talk because of the parallels I see to what we try to do in the Information Organisation course. As Laura discussed in the first talk, data require context for interpretation. The fact that there is so much data for us to work with hanging around on the internet, does not mean that we can do with it whatever we want without any consequences. Making people more aware of what we do not know, even in this information age, is essential. Hopefully I can learn something from Heloisa's work in terms of working with laypeople.
Piotr Mirowski - Making things up: LLMs on the theatre stage
The final talk was by Piotr, who is a researcher as well as an improv comedian. He was first introduced to LLMs in the contexts of weather forecasts. In those situations LLMs are great, as they naturally introduce pertubations that, in normal weather models, have to be introduced artificially. This also plays a role in the glitch aesthetic of the use of genAI in theatre and film.
As Piotr is also an improv comedian, this provided a nice segue into LLMs for theatre. For example, the team studied whether LLMs can be used for screen/playwriting, but is mostly seen as useful for circles that do not host writing workshops/groups. Participants also mentioned that LLMs should come up with things (ideas, contexts, sentences) humans could not. Another participant said that such systems could never write anything great, but would be fine for applications such as Netflix.
Even later, Piotr and his colleagues took LLMs onto the stage as a live actor. The shows of Improbotics evolved through multiple modalities, including earpieces, external microphones with speaker recognition, and smart glasses. The latter even allow the actors to curate between activities, which returns some agency to them. Overall, the work by Piotr returns some excitement to AI for creativity, as it explicitly requires improvisers to work around its limitations.
Piotr presenting about ethics. When AI and creativity come up, many people have opinions on the ethics of that, which they want to share.
For me, the AI and ethics perspectives are the most interesting things about the talk. I personally lean heavily toward the genAI-sceptic side of the scale, for many reasons. However, the perspective of artists becoming more creative thanks to what genAI can(not) bring to the stage was a new one to me. This is something I really respect Piotr for exploring.
Reflection
After the event, we all went out for dinner together in the center of Delft. We had lots of interesting discussions, including trying to characterize the different CHI events. Then, they tried to sell me on the CHI chapter parties and its powers of networking. I must say I am quite convinced. Unfortunately, CHI is held in the US of A next year, which I do not want to travel to under the current circumstances. Nonetheless, with the newer work I am doing that has less or even no educational focus, I have to choose what will be my second (or third) home. The breadth of CHI was something I always was hesitant of. With so many papers and tracks on different topics, I do not know if my work will reach an appropriate audience. But maybe such breadth also has its beauty.
Many thanks to Himanshu, Ujwal, Alessandro and Pablo for organising!