The speed and scale of agentic action make it all but impossible to keep a well-informed and attentive human in the loop. Much more feasible is to delegate the monitoring of agent interactions to other AI systems. But this presents some challenges. How should this work in practice if we want to maintain accountability, and adhere to relevant regulatory frameworks for oversight? A recent working paper by Matthias Holweg and Philipp Hacker tackles these issues.
One of the challenges they raise is the self-preference bias of LLMs: “When an LLM evaluator shares architectural or training foundations with the evaluated model, the assessment risks systematic distortion”. They propose a principle of independence drawing on the International Ethics Standards Board for Accountants (IESBA) standards which include (1) independence of mind and (2) independence in appearance. Moreover, common law says that an entity with an interest in the outcome of a determination should be disqualified from making it. They argue from these ideas that the AI system used for monitoring should not come from the same model (or model family) as the AI being monitored, and that there should be financial independence in the assessing entity.
The authors further consider what the law in the EU demands in terms of human oversight and monitoring and what that entails for automated oversight. Full human oversight of all interactions is not economically viable, but they argue that a risk-based sampling approach would help fulfill the legal requirements for high-risk use cases. In other words, they still think it would be consistent with the law if a person didn’t oversee and sign-off on every interaction of the agent. This opens the door to a scenario where AI both proactively samples a subset of cases for human attention and also monitors, flags, and escalates problem cases to a human overseer. They argue that this process should be risk-cognizant. So, for instance, in a situation where an action is irreversible or where harm occurs more quickly than monitoring could help intervene, this might require prospective sign-off by a person. On the other hand, “A retrospective regime is adequate where the monitored actions are reversible or where their residual risk to health, safety, or fundamental rights remains acceptable without prior review.”
AI watching AI also means making automated decisions about what to log and retain that might be relevant to an accountability process later. Logging everything isn’t viable from a resource perspective (do we really want data centers everywhere?) nor from a privacy perspective (the more you log, the more likely you sweep in personal data). As Holweg and Hacker write, “a balance needs to be found that enables selective audits but disables surveillance-style universal recording”. Logs need to be retained longer if a potential harm is irreversible or where it would go undetected or take an impacted person longer to recognize the harm. Logging similarly needs to be grounded in an understanding of risk.
All of this means that we shouldn’t expect a one-size-fits-all approach to AI systems monitoring AI agents. Even if a human can’t look at everything (i.e. human on-the-loop rather than in-the-loop), the degree of human involvement in monitoring should align with the risk of the agent’s actions. Oversight of low-risk agent interactions (e.g. reading information from a trusted website) can be more fully automated than high-risk agent interactions (e.g. writing data in a medical context). The use of AI for oversight of AI systems is yet another layer of AI governance that needs more explicit guidance in policy.
References
Holweg M and Hacker P (2026) Should We Allow AI to Police Itself? Assessing the Technical Feasibility and Legal Implications of LLM-Based Agent Evaluation. DOI: 10.2139/ssrn.7378338.