AI and Automated Decisions: GDPR and the Evolving Decision-Making Process

AI and Automated Decision-Making

Broadly speaking, artificial intelligence (AI) is the term given to the ability of computer software to simulate human cognition, allowing machine systems to learn, reason, and interact. Over time, algorithmic software can be “trained” to make connections – from random inputs to revealing outputs – which allow it to “learn”, and “make” decisions.

This decision-making process, when conducted without any human involvement, is known as automated decision making (ADM). Common examples of ADM are:

• An autonomous vehicle stopping so it doesn’t drive through a red light

• A sales company using an algorithm to monitor someone’s online behaviour to build a user profile they can use to send specifically targeted advertisements

• In the health space, a machine using visual cognition and perception to detect cancerous skin cells and provide patients with a decision on a diagnosis on foot of that result

ADM has many potential benefits to society such as saving time, costs efficiency, and reducing human labour. However, providers of these technologies need to be aware of and comply with a number of data protection requirements.

What does the GDPR say about ADM?

Article 22 of the GDPR states that data subjects have the right not to be subject to a decision based solely on automated processing. Therefore, individuals may object to any processing of their data which is carried out without any human oversight or involvement. For instance, you wouldn’t have to agree to a credit institution only using an online algorithm and chatbot to determine your credit rating for the purposes of deciding whether or not you can take out a loan.

Importantly too, Articles 13 - 15 of the GDPR require that any organisation planning to feed data into an algorithm that will then make a decision which affects an individual, must inform the individual that processing of this nature will take place.

Are there any exceptions?

While ADM using an individual’s personal data is not subject to total prohibition under the GDPR, it is only permitted in instances where:

1. The individual has explicitly consented to the automated processing, or

2. The ADM is necessary for the performance of a contract, or

3. It is legal to do so under EU or member state law

Even in cases based on one of the three exceptions above, the business employing ADM must implement suitable measures to safeguard the individual’s rights, freedoms and legitimate interests. The individual must at least have the right to obtain human intervention on the part of the company and also the ability to express his or her point of view and to contest the ultimate decision made.

Profiling

Article 22 also prescribes that data subjects have the right not to be subject to any ADM process which includes profiling.

The GDPR defines profiling as any use of an individual’s personal data by a computer system to evaluate, analyse or predict aspects about them, and particularly references their performance at work, economic situation, health, personal preferences, interests, reliability, behaviour, location or movements.

Restrictions on profiling apply if the decision in question will produce legal or similarly significant effects on a data subject. By way of example, if seeking insurance or a bank loan, or if you are a patient seeking medical treatment, profiling would likely result in a significant impact on you.

Special category data

Where special category (i.e. particularly sensitive) data is concerned, the threshold of protection (and restrictions on ADM) set by the GDPR are much higher.

Special category data (SCD) is defined in Article 9 of the GDPR as any data concerning an individual’s racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership. It also refers to an individual’s genetic data, biometric data (for the purpose of unique identification), health data or information relating to a person's sex life or sexual orientation.

While the processing of SCD is already subject to special conditions under the GDPR, any automated processing of SCD (including profiling) is generally prohibited, unless certain conditions apply which allow it to take place. ADM can be used to process this special class of protected data if:

1. The individual has given their explicit consent to the processing, or

2. There is a “substantial public interest” to justify the processing in question

What steps should organisations be taking?

The potential of AI and ADM technologies remains significant across a number of sectors, whether retail, manufacturing, healthcare or education. Given all of the rules around automated decision making, organisations have to be mindful of a number of things:

1. AI technologies must be fed data (often a lot of it) to thrive, and so special attention must be afforded to this fact

2. ADM and profiling using personal data can, and often does, result in legal or similarly significant effects for individuals

3. The GDPR contains strict measures that seek to mitigate the risks for individuals of any unchecked ADM

An organisation that is alive to these requirements as they develop, test and refine their AI technology will be well placed on their journey to regulatory readiness.

Conclusion

With AI and efficient automation becoming more readily available in the market, organisations must be mindful of the protections that GDPR affords to individuals with respect to these technologies.

An organisation relying on ADM technology needs to ensure that it is aware of the salient provisions of the GDPR. This includes determining when and where an automated process will have a “legal” or “similarly significant” effect on an individual, and paying due attention and care if the ADM is processing special categories of data.

There are also uncertainties ahead on the application of the technology itself. It remains to be seen to what extent complex decisions can be fully automated, and then, where human intervention is needed, to what extent can this be measured? Will full human integration into the decision-making process be required, or will mere oversight be sufficient? We will be watching the developments in this area with interest.

For more information, please contact Mark Adair, Rory O’Sullivan-Hennessy and Conor Califf, or a member of our Technology team.

---

The content of this article is provided for information purposes only and does not constitute legal or other advice.