Automated Moderation Must Keep Pace with Its Own Flaws

Whistleblower Frances Haugen’s leaked documents from Meta in 2020 revealed a disturbing statistic: the company’s algorithms designed to detect terrorist content incorrectly deleted nonviolent Arabic-language content at an alarming rate of 77 percent. This was not an isolated incident, as Meta’s own transparency report later that year showed similar findings. Five years on, researchers in the region confirm that overzealous moderation remains a persistent problem, with few solutions in sight.

The issue is particularly pronounced in less-resourced languages, where automated systems are struggling to keep pace. A 2025 report from the Center for Democracy and Technology found that labeled datasets in certain languages and dialects, such as Maghrebi Arabic and Kiswahili, contain inconsistencies, bias, and inaccuracies due to a lack of annotators who speak these languages fluently. An investigation into ChatGPT’s outputs in several low-resource languages highlights the depth of this problem.

Language disparities are just one concern among many when it comes to automated moderation. The systemic suppression of content from Palestine is another example, as well as the repeated misclassification of LGBTQ+ content as adult or explicit material. These varied examples demonstrate the risks of relying too heavily on automated systems and highlight the need for stronger safeguards.

Automated systems can process vast amounts of content at a scale that humans cannot match, potentially enabling better moderation and alleviating the psychological load on moderators who must view disturbing content. However, these systems also reproduce existing biases, struggle to understand context, and often make mistakes that disproportionately affect vulnerable communities. As Rachel Griffin noted in 2023, ‘Perfectly accurate moderation is not only technically out of reach but intrinsically impossible.’

Despite the intrinsic flaws of automated systems, there are steps companies, policymakers, and civil society can take to ensure they operate in ways that respect human rights and minimize predictable harms. If companies continue to rely on automation for content moderation – as seems likely – then accountability must evolve alongside these technologies.

One way this evolution can begin is by committing to the Santa Clara Principles 2.0. These principles, first outlined in 2020 and re-launched in 2021 after international input, reflect the needs and expectations of the global community and specifically address automation. The Foundational Principle states that companies should integrate human rights and due process considerations at all stages of content moderation, publishing information on how this integration is made.

Companies should only use automated processes to identify or remove content when there is sufficient confidence in their quality and accuracy. Users must also have clear and accessible methods for obtaining support if they believe their content has been wrongfully removed. Drawing on the Santa Clara Principles 2.0, international human rights standards, and years of research documenting the shortcomings of automated moderation, we propose eight recommendations for policymakers thinking about regulation and companies deploying AI-assisted content moderation systems.

These recommendations include ensuring that automated technologies complement, rather than replace, human moderators. Companies must be transparent about when and how automation is used in content decisions, regularly auditing their systems for bias – particularly in low-resource languages and conflict zones. Users should have the ability to appeal moderation decisions, providing context if necessary, with appeals promptly evaluated by human moderators.

Companies should also assess the human rights impact of their moderation decisions, issuing public statements on the results. If they rely on third-party vendors, companies must carefully audit those vendors for compliance with these principles. Lawmakers and policymakers should avoid promoting legislation that effectively or explicitly mandates automated moderation systems, instead focusing on creating a regulatory framework that prioritizes transparency and accountability.

The design and oversight of content moderation systems must respond to the concerns of policymakers, civil society, independent researchers, and affected communities. This is not just a technical problem for engineers and product teams to solve; it requires a comprehensive approach that addresses the complex issues at play.