The Digital Methods Initiative (DMI), Amsterdam, is holding its annual Winter School on 'Chatbots for Internet Research?'. The format is that of a (social media and web) data sprint, with tutorials as well as hands-on work for telling stories with data. There is also a programme of keynote speakers. It is intended for advanced Master's students, PhD candidates and motivated scholars who would like to work on (and complete) a digital methods project in an intensive workshop setting.
Positions now are increasingly staked out in the debate concerning the application of chatbots and LLMs to social and cultural research. On the one hand there is the question of ‘automating’ methods and shifting some additional part of the epistemological burden to machines. On the other hand there is the rejoinder that chatbots may well be adequate research buddies, assisting with (among other things) burdensome and repetitive tasks such as coding and annotating data sets. They seem to be continually improving, or at least growing in size and apparent promise.
Researcher experiences are now widely reported: chatbots have outperformed human coders, ‘understanding’ rather nuanced stance-taking language and correctly labeling it better than average coders. But other work has found that the LLM labeling also has the tendency to be bland, given how the filters and safety guardrails (particularly in US-based chatbots) tend to depoliticise or otherwise soften their responses. As researcher experience with LLMs becomes more widely reported, there are user guides and best practices designed to make LLM findings more robust. Models should be carefully chosen, persona’s should be well developed, prompting should be conversational and so forth. LLM critique is also developing apace, with (comparative) audits interrogating underlying discrimination and bias that are only papered over by filters.
At this year’s Digital Methods Winter School we will explore these research practices with chatbots and LLMs for internet research, with an emphasis on bringing them together. How to deploy and critique chatbots and LLMs at the same time, in a form of reflexive usage?
For all details about this Winter Course, please visit the Digital Methods website below.
For a preview of what the event is like, you can view short video clips from previous editions of the Winter School on YouTube.