SWITCHED ON
The daily technology series nobody asked for but everyone needed
The Invisible Judge: AI in Hiring, Lending, and Criminal Justice
Algorithms are already deciding who gets jobs, loans, and bail. The question of whether they are doing it fairly has an answer. The answer is not reassuring.
A human hiring manager who discriminates can be held accountable. An algorithm that discriminates at scale, embedded in a system nobody fully understands, producing decisions that nobody has to individually justify — that is a considerably harder problem. It is also the problem we have built and deployed without adequately solving it first.
Yesterday we covered AI art and copyright — the training data dispute, the lawsuits working their way through courts, the fair use argument and its limits, the Glaze and Nightshade tools that artists are using to fight back technically, and the harder question about what the creative economy looks like when the marginal cost of generating competent visual art approaches zero. Today we are moving from the philosophical to the immediately personal. Algorithms are already making or informing decisions about whether you get a job interview, whether you get a mortgage, and, in the criminal justice context, how long you might spend in prison or whether you get bail. The systems doing this are opaque, inconsistently audited, and in several documented cases demonstrably biased in ways that replicate and amplify the inequalities of the historical data they were trained on. This is not a hypothetical future risk. It is the current state of play.
01 — Hiring: The Resume That Never Gets Read
Automated resume screening is now standard practice at large employers. The majority of applications submitted to Fortune 500 companies are processed by applicant tracking systems before any human sees them. These systems filter, rank, and in some cases score candidates based on keyword matching, work history patterns, educational credentials, and — in more sophisticated systems — inferences drawn from language use, social media profiles, and video interview analysis.
Amazon built and then scrapped an AI hiring tool in 2018, after internal audits revealed it was systematically downgrading resumes that included the word "women's" — as in "women's chess club" or "women's university" — and penalising graduates of all-women's colleges. The system had been trained on a decade of Amazon hiring decisions, made predominantly by and in favour of men, and had faithfully learned to replicate that pattern. The tool was never used to make actual hiring decisions, but it illustrates the mechanism with unusual clarity: a model trained on historically biased outcomes will learn those biases as signal rather than noise.
Video interview analysis tools — which score candidates on factors including facial expressions, word choice, and vocal tone — have been adopted by thousands of companies and scrutinised by researchers who have found evidence of bias correlated with race, gender, and disability status. The companies offering these tools dispute the findings. The auditing methodologies are contested. The tools continue to be used at scale while the argument proceeds, generating hiring decisions for real candidates whose outcomes are shaped by systems whose fairness has not been established to any rigorous standard.
A biased human recruiter affects the candidates they personally review. A biased algorithm deployed at scale affects every candidate who passes through the system — potentially millions of people, all experiencing the same systematic disadvantage, with no individual decision-maker to hold accountable.
02 — Lending: When Your Postcode Is Your Credit Score
Credit scoring has been algorithmic for decades — the FICO score, introduced in 1989, is itself a model. What has changed is the expansion of the data inputs that lenders use, the opacity of the models consuming them, and the precision with which those models can proxy for protected characteristics without explicitly using them.
The problem of proxy discrimination is central to understanding algorithmic lending bias. It is illegal in most jurisdictions to use race, gender, or religion as inputs to a lending decision. It is not illegal to use postcode, employment type, shopping patterns, smartphone model, or the time of day at which you submit your application — all of which correlate with race and other protected characteristics to degrees that allow a sufficiently sophisticated model to effectively discriminate on protected grounds while using only ostensibly neutral inputs. The model never sees race. It sees a constellation of proxies that reconstruct it with uncomfortable accuracy.
A 2021 investigation by The Markup found that mortgage algorithms in the United States denied home loans to Black applicants at significantly higher rates than white applicants with similar financial profiles, even after controlling for factors like income, loan amount, and neighbourhood. The lenders attributed the disparities to legitimate risk factors. The investigation could not independently audit the models because the models are proprietary. This opacity is not incidental to the problem — it is a structural feature of a system in which consequential decisions are made by black boxes that their owners are not required to open for scrutiny.
03 — Criminal Justice: The COMPAS Affair
The most extensively documented and debated case of algorithmic decision-making in high-stakes contexts is COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — a risk assessment tool used in the US criminal justice system to predict the likelihood of reoffending. COMPAS scores are used in some jurisdictions to inform bail decisions, sentencing recommendations, and parole hearings. They carry significant weight in decisions that determine whether a person is free or imprisoned.
In 2016, ProPublica published an analysis of COMPAS scores for defendants in Broward County, Florida, and found that the tool was nearly twice as likely to falsely flag Black defendants as future criminals compared to white defendants, and nearly twice as likely to falsely flag white defendants as low risk when they went on to reoffend. The company that produces COMPAS, Equivant, disputed the methodology. A subsequent academic debate about the appropriate definition of fairness in risk assessment tools produced an uncomfortable mathematical finding: several common definitions of fairness are mutually incompatible, meaning a tool cannot simultaneously satisfy all of them. The choice of which fairness criterion to optimise is itself a values choice, not a technical one — and it has been made by private companies without democratic deliberation.
The opacity problem is, if anything, more acute in criminal justice than in lending. At least in lending, the Equal Credit Opportunity Act provides some recourse for discrimination claims. In criminal justice, the use of proprietary risk assessment tools has been challenged on due process grounds — defendants and their lawyers have argued that they have a right to understand and challenge the basis of a risk score used against them in court. Courts have split on this. Some have upheld the use of proprietary tools. Others have found that the inability to audit or challenge the algorithm raises constitutional concerns. The legal framework has not settled.
04 — Predictive Policing and the Pre-Crime Problem
Predictive policing tools — systems that analyse historical crime data to predict where crimes are likely to occur or who is likely to commit them — have been adopted by police departments across the United States and in other countries, and have been abandoned by others following evidence of the biases they introduce.
The fundamental problem with predictive policing is a feedback loop that is almost impossible to break. Police departments historically over-police certain neighbourhoods and communities. Historical crime data therefore shows higher crime rates in those areas, not necessarily because more crime occurs there but because more policing occurs there and more arrests are made. A model trained on that historical data learns that those areas and communities are higher risk and directs more policing toward them. More policing produces more arrests, which feeds back into the training data, which reinforces the prediction. The model is not predicting crime. It is predicting where the police have historically looked for crime, and directing them to look there again.
Los Angeles, Santa Cruz, and a number of other jurisdictions have banned predictive policing tools after community pressure and internal reviews found the evidence for their effectiveness weak and the evidence for their discriminatory impact substantial. Others continue to use them. The pattern is consistent with the broader landscape of algorithmic decision-making in high-stakes contexts: rapid adoption, slow auditing, inconsistent regulatory response, and the people most harmed by the systems having the least power to challenge them.
The promise of algorithmic decision-making was objectivity — removing human prejudice from consequential judgements. The reality is that algorithms encode the prejudices of their training data, execute them at scale, and add a layer of technical complexity that makes them harder to challenge than the human bias they replaced.
05 — What Accountability Actually Requires
The regulatory response to algorithmic bias is, like most AI regulation, fragmented and developing. The EU AI Act classifies AI systems used in hiring, credit scoring, and criminal justice as high-risk, imposing requirements for transparency, human oversight, data quality, and the ability of affected individuals to seek explanation and redress. These are meaningful requirements, assuming they are enforced, which requires the auditing capacity that most regulators do not yet have.
In the US, the Equal Employment Opportunity Commission and the Consumer Financial Protection Bureau have both issued guidance applying existing anti-discrimination law to algorithmic systems. The Federal Trade Commission has pursued enforcement actions in related areas. New York City passed a law in 2021 requiring bias audits of automated employment decision tools used by city employers, with audit results published. It is the most concrete piece of legislation in this space in the US and, by most assessments, a floor rather than a ceiling.
What meaningful algorithmic accountability actually requires goes beyond what any current regulatory framework fully delivers: independent auditing by parties with full access to models and training data; the right of affected individuals to meaningful explanation of decisions made about them; the ability to contest those decisions through processes that can actually change outcomes; and the removal of systems whose bias cannot be adequately corrected. This last requirement — actually stopping the use of systems that demonstrably discriminate — is where the gap between stated principle and actual practice is widest. The systems remain in use. The auditing continues. The people affected by the biased decisions absorb the consequences while the regulatory process catches up.
Tomorrow we are going somewhere that has been building in the background of this entire series — tech monopolies and antitrust. Google, Apple, Meta, Amazon, Microsoft: how they got so large, what regulators on both sides of the Atlantic are trying to do about it, and whether the antitrust tools designed for industrial-era monopolies are the right instruments for platform-era dominance. See you then.
Switched On is a daily technology series covering AI, social media, data privacy, and the digital forces reshaping modern life — with no corporate spin, no false comfort, and absolutely no mercy for buzzwords.
