In what ways do judges use AI?

‘It ranges from very mundane, administrative tasks to high-stakes decisions. AI sorts documents or helps to search case law. But there are also systems that profile people and conclude that someone has a 70 percent chance of reoffending. Other AI systems draft judgments and take over the judge’s work. AI can even ruin lives in cases concerning, for example, custody, prison sentences and social welfare.’ 

Is AI also used by judges in the Netherlands?

‘In the Netherlands, there have been pilot schemes in which AI is used to draft judgments in straightforward cases, but it hasn’t gone much further than that here yet. Profiling systems are, however, used extensively, for example, by the probation service. With the emergence of large language models, we are seeing AI play a much greater role in the judiciary. In other countries, AI is already used much more extensively in the judiciary. China and Brazil are frontrunners in that regard.’

Are China and Brazil model countries?

‘China not really. That country is a frontrunner in the use of AI, but not in its regulation. Brazil is more of an example. They have comprehensive, detailed legislation specifically for the use of AI in the courtroom. Now, I don’t think judges will start writing their judgments using AI en masse. But it is interesting to see how more tasks are being outsourced. In my research, I look at how litigants might have all sorts of questions about a judge’s use of AI: why did an AI-system produce this output in my case? Why did it write this judgment? Why do I have a certain percentage chance of reoffending? And why does AI use these cases for comparison, and not others? People need to understand the logic behind these outputs.’

Is it very difficult to understand AI systems?

‘It can sometimes be difficult or even impossible to provide an explanation, depending on the model. One of the main reasons is technical. The largest AI models can behave in ways that are illogical and non-intuitive to humans.’

Why is it important to know how AI arrives at a conclusion?

‘It is dangerous and harmful if we do not know this, because of the importance of “procedural justice”, that is say, the extent to which people perceive the judicial process and decision-making as fair. You cannot detect errors in a system you do not understand. That means you cannot contest those errors either, even though it may be a life-changing matter. For instance, in a case concerning social welfare, AI could reduce you to a set of factors that you are unaware of. But you would not be able to contest that profile, because you do not know how the system views you. If those errors go undetected, AI can ruin lives on a massive scale. The second problem is that citizens are not respected in a process in which they cannot participate. In fact, it demonstrates a profound lack of respect. Citizens deserve to be able to observe the process and to contest decisions.’

How can AI in the judicial system be better explained?

‘Even during the AI development process, we need to consider how it can be explained to the public. This would also allow the judiciary, for example, to reflect on whether it should be using a more explainable type of AI model.’ 

Is it realistic to expect AI to become explainable?

‘Yes, I think these measures are practically feasible. At the same time, it is true that AI systems such as large language models can perform a great deal of work very cheaply, but they are not faithfully explainable. So, if you want to replace those opaque models with explainable models, it can cost a lot of time and money in some cases. The same applies to profiling systems. Sometimes we use non-explainable systems because they are more powerful and identify patterns that would otherwise remain hidden. It is also quite understandable that large language models, such as ChatGPT, are used for low-impact tasks, such as checking for grammatical errors. But when is the impact of a particular task significant and when it is not? That is quite difficult to determine.’

Who currently oversees the use of AI in the judiciary?

‘The judiciary itself oversees the use of AI. This is given careful consideration. The output of an AI system is certainly not blindly accepted by a judge. But if things go wrong, the citizen must still be able to defend themselves. That is why the explainability of AI is so important. It is a final check. But as far as I’m concerned, we should also consider in advance which citizens we are burdening with that “task”. After all, it takes time, knowledge and financial resources to delve into the use of AI in a case, even if the AI system is explainable. I argue we should rather use AI in a case against a large company suspected of fraud than in a case concerning social welfare. We don’t need to use AI in every court case.’

Isn’t the right to an explanation ultimately just too expensive?

‘Developing explainable AI and producing an explanation takes time and money. I understand the perspective that legal proceedings must also be affordable for society. But what is so special about the administration of justice is that you have the right to make your voice heard – regardless of whether that is financially viable. We must not simply brush aside the challenge posed by judicial AI. Even if AI were flawless – which it certainly isn’t – you, as a person, deserve to be able to scrutinise the system. You deserve to be heard and respected. You cannot reduce that to a matter of cost. Yes, we must consider the finances and practicalities of everything, but people have an inviolable dignity. Even in a digital society.’