Seeking advice from chatbots: how responsible is it?

21/08/2024 – In a recent verdict, a Dutch judge indicated that it had used ChatGPT as a source. This led to a storm of outraged reactions among lawyers and AI experts. This real-life example raises the question of how responsible the deployment of smart chatbots actually is and what organisations should take into account in that case. But are there not also important lessons to be learned from this interesting choice?

Since the launch of ChatGPT in November 2022, AI and smart chatbots have been hot topics. In the first six months of 2024 alone, the biggest tech companies ('Big Tech') together invested 100 billion euros in AI. Nobody seems to want to fall behind in the race. At the same time, there is also a lot of research on the possibilities and dangers of AI. A spring report by the AP (Dutch supervisory authority) concludes that AI risk management is not yet progressing at the same pace as AI development. In addition, in the legal landscape, the AI Act has finally officially come into force, but it is still going to take some time before the provisions regarding high-risk AI systems actually take effect and can be enforced by regulators for compliance.

AI as a source of information

In short: getting to grips with AI remains a challenge for now. It is therefore not surprising that many organisations and experts are reluctant to allow the use of AI chatbots within their day-to-day operations. Perhaps that is why experts have reacted with such surprise to the judge's use of ChatGPT.  The aforementioned court case involved a dispute between two homeowners over a roof structure that would interfere with certain solar panels. The judge then consulted ChatGPT to estimate the average lifetime of solar panels and the current average price per kilowatt-hour of electricity, as reflected in the judgment. This information was used by the judge as a resource for determining damages.

This choice by the court is questionable. For instance, the information produced by smart chatbots is far from always up-to-date, and sometimes this information never even existed. The New York Times found that smart chatbots make up at least 3% of their answers entirely by themselves. In some cases, this is as high as 27%. It is also not always entirely clear exactly where smart chatbots get their data from. Reason enough to use these tools with great caution. Other risks when using chatbots include leaking sensitive data or infringing someone else's copyright.

Opportunities

So should there be a ban on the use of chatbots in a legal context? That conclusion probably stretches too far. After all, using chatbots can bring many benefits. For instance, it can be time-saving for structuring complex information. In addition, there are now even chatbots that specialise in law, such as Harvey, which is used by several law firms. Many lawyers therefore assume that the use of such AI tools will only increase over time.

Responsible use of chatbots

If you want to start using a chatbot within your organisation, keep the following in mind:

• Establish policies for generative AI within your organisation that define what employees may or may not use the chatbot for.
• Train staff who work with it a lot. Indeed, the AI Act is also going to mandate this.
• Does the chatbot regularly receive personal data as input? Have a DPIA carried out. Not sure if the chatbot is operating ethically enough? Have an IAMA carried out.
• Be transparent: inform data subjects (such as customers) that their data may be processed by an AI-driven chatbot.

Also check out this earlier blog on using ChatGPT responsibly within your organisation.