When a comment disappears on Instagram, a video is flagged as potentially harmful on YouTube, or a complaint is automatically responded to on X, more and more often it is not a human but an AI system that is behind it. Such systems, which act autonomously – that is, evaluate content, make decisions, and engage in dialogue with users – are known as “social AI agents.” Public debates are thus increasingly shaped by systems that are not publicly accountable themselves.

This development has implications on two levels: On the one hand, generative AI systems produce texts, images, and videos at a pace that makes human review practically impossible, while agentic systems automatically evaluate, downrank, or remove this content. Authorities and regulatory bodies, on the other hand, continue to rely on procedures that depend on case-by-case review and transparent justification. While the EU AI Act and the Digital Services Act have established new legal requirements for transparency and accountability, verifiable criteria that link these requirements to the practice of automated moderation are still lacking.

The collaborative research project AGORA-KI addresses this gap. Four partners from the fields of social research, technology assessment, legal and policy analysis, and technology development combine empirical research, participatory reflection, and regulatory operationalization to understand the consequences of automated moderation and translate them into robust governance instruments.

The work of the ITAS team forms the empirical core of the project. Through guided interviews with moderators, platform developers, and regulatory actors, the team investigates the logic behind automated moderation decisions and how the actors involved attribute responsibility. The project also gives a voice to people affected by hate speech as well as users whose content has been wrongly removed. Both groups experience a loss of control vis-à-vis algorithmic systems: the former because hate speech directed at them remains online; the latter because their posts are incorrectly classified as violations. The researchers are interested, for example, in whether people targeted by hate speech feel that platforms are increasingly delegating protection to machines, or whether users whose posts have been removed lose trust in the platform as a space for public discourse.

Building on these findings, ITAS designs and realizes participatory workshops in which these groups work alongside experts and regulatory authorities to determine requirements for transparency, fairness, and human oversight that future moderation systems must meet. Marginalized communities, such as LGBTQIA+, BIPoC, and neurodivergent individuals, are deliberately included in this process. The team also uses scenario workshops (foresight) to explore how moderation might function under changing technological conditions, with the aim of developing governance criteria that remain effective even as the technological basis shifts.

ITAS synthesizes the results into a structured governance matrix: a tool that consolidates requirements from different perspectives, makes tensions visible, and defines indicators against which compliance can be assessed. The matrix thus creates an empirically grounded bridge between what regulation demands and what can actually be verified in practice.

AGORA-KI thereby establishes a basis on which to assess whether automated moderation systems meet the requirements of democratic accountability.

Tel.: +49 721 608-22329
E-mail