Avoiding “baked in” bias in AI healthcare applications

“ARTIFICIAL intelligence (AI) holds enormous potential for the NHS, if we can use it right.”

So wrote Matthew Gould in a blog back in 2020. Gould is the National Director for Digital Transformation in NHS England.

“It [AI] can reduce the burden on the system by taking on the tasks that can be converted into an algorithm. Many of these are in areas of greatest pressure, like radiography and pathology. It could improve patient outcomes, and increase productivity across the system, freeing up clinicians’ time so they can focus on the parts of the job where they add the most value.”

Part of the challenge of “using AI right” is avoiding unintended and potentially harmful consequences when employing AI algorithms in practical applications.

Back in 2019, a research paper published in Science examined a commercial algorithm used by hospitals and other health systems in the US to predict those patients most likely to need follow-up care.

The system falsely classified Black patients as being healthier than white patients even when they were just as ill. The researchers found that the bias arose because the algorithm relied on care costs as a proxy for health needs, rather than actual illness. Unequal access to care in the US means that less money is spent caring for Black patients and this resulted in large racial biases in the algorithm.

The NHS is certainly keen to harness the enormous potential of digital technology to analyse patient information, as a succession of health and social care secretaries have made clear. But it is also aware of the risks that AI could exacerbate existing health inequalities if concerns over algorithmic bias are ignored.

To that end, NHS England recently launched a pilot into the use of algorithmic impact assessments (AIAs) to help eradicate such bias. An AIA designed by the Ada Lovelace Institute is now being piloted to help developers consider the risks and biases of any potential AI system before being allowed access to vast stores of NHS data.

Assessing the “black box”

A computer algorithm can be described in very simple terms as the preset coded instructions by which a system operates, whereas an AI algorithm is “an extended subset of machine learning that tells the computer how to learn to operate on its own”. Many of these software systems are mind-bogglingly complex – effectively “black boxes”, opaque to meaningful public scrutiny and accountability.

AIAs are an emerging approach for holding AI developers accountable for the potential societal impacts of algorithmic systems before they are used, and for ongoing monitoring once a technology has been implemented. The Ada Lovelace Institute has led research in the UK on the use of AIAs.

The NHS plans to trial AIAs across a number of initiatives and more immediately as part of a precondition to access data from the National Covid-19 Chest Imaging Database (NCCID) and a proposed National Medical Imaging Platform (NMIP).

The NCCID is a centralised database that supports researchers to better understand Covid-19 and develop technology to improve care for patients hospitalised with severe infection. The proposed NMIP will expand on the NCCID and make it possible to collect imaging data on a national scale, providing this data for the development and testing of safe AI technologies applied to imaging in general, such as for cancer screening.

Head of AI Research & Ethics at the NHS AI Lab, Brhmie Balaram said: “The algorithmic impact assessment will prompt developers to explore and address the legal, social and ethical implications of their proposed AI systems as a condition of accessing NHS data.

“We anticipate that this will lead to improvements in AI systems and assure patients that their data is being used responsibly and for the public good.”

Unintended subjectivities

The Ada Lovelace Institute highlighted its approach to AIAs in a recent paper. The approach challenges developers to ask questions at the earliest stages of an AI system’s development. Will this system affect certain individuals disproportionately more than others? What kinds of socio-environmental factors – such as stable internet connectivity or a reliance on existing hospital infrastructure – will determine its success or failure?

This will then allow early mitigation and help prevent unintended “subjectivities”, such as biased or potentially racist attitudes or behaviours, to become “baked-in” to AI systems.

For example, research published in the Lancet in 2021 identified that some AI systems being developed to diagnose skin cancer run the risk of being less accurate for people with dark skin. The researchers identified 21 open access datasets containing 106,950 skin lesion images of which only 2,436 images had skin type recorded and only 11 images were from people recorded as having brown skin or black skin.

The Ada Lovelace paper states: “Potential harmful impacts arising from use of these systems are myriad, from both healthcare-specific concerns around violating patient consent over the use of their data, to more generic risks such as creating public mistrust of AI systems and the institutions that develop or deploy them.”

Seven-step process

The AIA pilot will involve developers engaging in a seven-step process before being allowed access to the National Medical Imaging Platform. This will involve first undertaking an impact-identification exercise to be submitted to a Data Access Committee.

Here, the purpose, scope and intended use of the proposed system, model or research must be detailed along with who will be affected. It is intended to provoke reflexive thinking about ethical concerns, intended and unintended consequences, and measures to mitigate any harms.

An interactive workshop for suitable applicants will then pose further questions on harm/benefit scenarios and other potential impacts before the committee makes a decision about whether to grant data access. The successful AIA will then be published and revised on an ongoing (iterative) basis by the project team.

Octavia Reeve of Ada Lovelace Institute and interim lead of the project, said: “Algorithmic impact assessments have the potential to create greater accountability for the design and deployment of AI systems in healthcare, which can in turn build public trust in the use of these systems, mitigate risks of harm to people and groups, and maximise their potential for benefit.”