Liable, but Not in Control? Ensuring Meaningful Human Agency in Automated Decision‐Making Systems

In: Policy & internet, Band 11, Heft 1, S. 104-122

Abstract

Automated decision making is becoming the norm across large parts of society, which raises interesting liability challenges when human control over technical systems becomes increasingly limited. This article defines "quasi‐automation" as inclusion of humans as a basic rubber‐stamping mechanism in an otherwise completely automated decision‐making system. Three cases of quasi‐automation are examined, where human agency in decision making is currently debatable: self‐driving cars, border searches based on passenger name records, and content moderation on social media. While there are specific regulatory mechanisms for purely automated decision making, these regulatory mechanisms do not apply if human beings are (rubber‐stamping) automated decisions. More broadly, most regulatory mechanisms follow a pattern of binary liability in attempting to regulate human or machine agency, rather than looking to regulate both. This results in regulatory gray areas where the regulatory mechanisms do not apply, harming human rights by preventing meaningful liability for socio‐technical decision making. The article concludes by proposing criteria to ensure meaningful agency when humans are included in automated decision‐making systems, and relates this to the ongoing debate on enabling human rights in Internet infrastructure.