Public sector due diligence on AI systems needs to improve in order to protect vulnerable individuals

The US is starting to see legal cases being brought where AI chatbots are alleged to have caused significant harm to individuals. It’s inevitable that these kinds of actions will also be brought in other jurisdictions.  Meanwhile, the stretched UK public sector is investing in chatbots to provide advice to vulnerable individuals, including carers who are dealing with seriously ill relatives.

Some of the advice is undoubtedly helpful.  The chatbots can point individuals to the benefits to which they are entitled and to phone numbers so that they can call for further assistance where they need it. This is clearly more user-friendly than trawling through numerous government or local authority websites for information which may be hard to find. 

But there are also deeply worrying aspects to the use of AI chatbots for this sort of advice.   One particular provider which appears to be used by both central government and numerous local authorities asked users for their personal details and their sick relative’s name and medical conditions. It was not clear why this information was required.  The company’s privacy notice was riddled with legal errors and set out no proper legal basis for the processing of incredibly sensitive personal data.  Any privacy professional would quickly have been able to ascertain that this company had received no credible advice on data protection.

We often explain to clients that a privacy notice is their “shop front” when it comes to data protection compliance.  If the privacy notice is deficient, then it’s very likely that the organisation in question will have failed to do its due diligence including conducting a data protection impact assessment to ensure that any risks to individuals are surfaced and mitigated.  Failing to do these mandatory assessments where the people using the chatbot are vulnerable is not only unlawful but deeply unethical.  

There was another telling aspect in the privacy notice.  One of the most prominent paragraphs was a disclaimer, stating that the company took no responsibility for the advice it was giving to these individuals. Disclaimers have no place in a privacy notice.  These notices are meant to give people clarity about how and why their data is being processed. 

The key here is that officials procuring these sorts of systems for the public sector need to upskill. Contracting with companies which appear to take no account of the law and no responsibility for the service they provide is deeply worrying.  No doubt AI can be used to improve public services, but only if the contracts are awarded to organisations which are ethically sound and legally compliant. 

Subscribe to receive more articles like this here.