AI Safety: Bias

Historical data may contain outdated information or information skewed by historical inequalities.

Example: Based on the gender pay gap, generative AI systems have suggested lower starting salaries for women.

The underrepresentation of certain groups in training data can mean they are falsely seen as statistically 'less important' and so can be disregarded in the outputs.

Example: Facial recognition software often has a higher success rate for people with lighter skin, indicating an imbalanced data or time allocation during the tool’s training stage.

Many generative AI tools show a tendency to align with cultural norms from English-speaking and European countries, likely through an imbalanced amount of training data from these countries. The decision which data to include may be financial (obtaining English-speaking information may seem less resource-heavy) or the people making the data selection may favour English-speaking sources due to their own bias.

Cultural bias does not only relate to different regions of the world. Within a culture, dominant ‘in-groups’ may be represented differently from marginalised groups. The latter regularly experience harmful and derogative stereotyping in generative AI outputs.

Example: Despite the prompt for an AI image-generation tool to generate an image of a visibly disabled person to lead a meeting, the disabled person was depicted in a listening-only role in the AI’s output.

Algorithms process information in different ways, depending on how they are designed to organise, select, and present their results. This could introduce biased favouritisms of solutions over others.

Example: The instruction to a generative AI to always give an answer can be in conflict to always give correct answers. If the AI algorithm weighs the first instruction as more important, generated outputs may be incorrect (see hallucinations). 

While a generative AI tool may work well in isolation during testing, once it is deployed in the real-world, outputs may be distorted by people using it for tasks the system was not intended for. Another reason can be overreliance on AI within an automated system. Essentially, an AI system can be developed for one task but then used for something else in practice, which can lead to bias.

Example: A CV-summarising system may work well during testing, and its training data is demographically well balanced. When deployed, employers start using it for hiring decisions, and outputs are biased because the system was not trained or tested for this task.

The way people use an AI system can slowly shape how it behaves. Human interactions with generative AI systems often provide a source of additional training data for the tool. In turn, AI outputs affect people’s preconceptions, which results in a feedback loop of reinforced bias. Here, bias does not come from the original training data alone, but from repeated user behaviour and feedback over time.

Example: If you consistently ask a generative AI model to check grammar only on documents written by people with non-English sounding names, this kind of interaction can start reinforcing the idea that people with a non-English background are more likely to have poor language skills.

If the generative AI is not continually trained, for example for financial reasons, it may fail to reflect changing (societal) perspectives and new discoveries.

Example: If an AI system’s training data was from before 2020, outputs would not reflect the Covid-19 pandemic and its impacts. Answers to prompts asking for business, career, or policy advice will not include important reference to remote work, or related social, health and economic developments. 

Checklist for AI Auditing | European Data Protection Board