A lot of people following the progress of Artificial Intelligence start from a pretty common assumption: that human oversight is only necessary today because the systems are still imperfect. As the technology improves, the logic goes, human involvement will gradually shrink until it practically disappears.
Sounds reasonable, right? 🤔
But a new study has come along to challenge that exact idea — and the arguments it presents are hard to ignore.
In a paper titled The boundaries of automation: A theory of persistent human participation, researchers from TU Darmstadt and the Munich Center for Machine Learning investigate where automation hits its limits — and why, in some cases, human contribution is essential to the final outcome. They also explore what these persistent human roles mean for the design, evaluation, and ethics of future AI systems.
Here is the real game-changer: according to the authors, human participation is not a temporary phase that will fade over time. In many situations, it is a fundamental necessity — something baked into the very nature of what we are trying to accomplish.
The question every AI breakthrough raises
Every new AI achievement brings back the same question: how much human involvement is still needed? If the technology can already code, write, analyze information, and even serve as a companion, it seems fair to ask whether having people in the loop is a temporary detail or a permanent requirement.
But that question carries a trap. It assumes that every goal humans want to achieve is already defined and just needs to be executed. And that is exactly where the study makes its most original contribution: the theory of goal emergence.
The researchers observe that not all goals come with fixed targets. In many cases, those targets take shape through exploration and deliberation. The people pursuing these goals are not just helping execute a task — they are helping create and refine the goal itself.
As the authors write, interaction with humans does not merely reveal which target was always destined to be adopted; it partly determines which path actually materializes. In other words, the human is not just the executor but also the author of what is being built.
Receive the best innovation content in your email.
All the news, tips, trends, and resources you're looking for, delivered to your inbox.
By subscribing to the newsletter, you agree to receive communications from Método Viral. We are committed to always protecting and respecting your privacy.
Creative work and science: where goals evolve
To make this clearer, the researchers use examples like creative work and scientific discovery. Unlike fields governed by precise logical rules, such as mathematics, these domains involve open-ended goals that evolve through curiosity and human judgment.
Think about a scientist. They might start with an intuition, using AI to explore a phenomenon without knowing exactly where the research will lead. New evidence can shift the direction of the investigation, generate entirely new questions, and completely redefine what counts as success. The final destination was not written at the start — it emerged along the way.
The same happens in creative work. A professional might change their criteria after evaluating the results AI delivered and reject the original goal, even when the system executed exactly what was asked. The person looked, reflected, and concluded that it was not quite what they wanted. That is profoundly human — and AI, on its own, does not make that kind of move. 💡
Automation is still important (and nobody is denying that)
An important point worth making here: the researchers are not arguing against automation. Quite the opposite. They acknowledge that some tasks involve goals that are already operational and can therefore be carried out in a largely autonomous way.
Once the objective is clear, activities like formatting references, transcribing speech, converting units, and organizing records are basically procedural. There is no nuance, judgment, or reinterpretation involved — it is just execution. And in those cases, letting the machine do the work makes total sense.
According to Francesca Rossi, global leader for Responsible AI and AI Governance at IBM, the central question is whether a goal is stable enough to be delegated. In her words, if the goal can change through interactions with people, tools, other agents, and so on, then humans should remain in the decision loop.
What really helps when delegating tasks to AI, she says, is external governance — something that involves oversight, regulation, auditing, and organizational controls that evaluate the AI system as a whole, including both individual agents and the way they interact with each other.
Internal governance: when agents themselves need rules
Beyond external governance, Rossi points out that agents may need their own internal governance mechanisms — the rules, constraints, monitoring, and decision-making structures embedded within AI agents. The idea is that they can recognize when goals are shifting and alert humans when additional guidance is needed.
As she explains, there are implications both for external governance and for how you build systems that need their own forms of internal governance. It is a layered architecture of responsibility.
The problem of translating human intentions into instructions
A fascinating detail from the study is that specifying a goal is not always enough to convey everything a human actually means. The researchers observe that a goal may be sufficiently specified for a human collaborator who shares the relevant cultural, situational, historical, or social context — and yet remain unavailable to an AI system that lacks or cannot reliably prioritize that context.
Jessica He, a UX designer and member of the Trustworthy AI team at IBM, echoed that sentiment. According to her, developers have reported that even when providing a clear goal, AI agents do not always follow those instructions and can veer off in unexpected ways.
Part of the challenge, she explains, is that giving clear goals is harder than it sounds — and our assumptions about what a model knows can be wrong. When that happens, the AI ends up creating something that does not match what people actually intended.
When models go too far
To illustrate this in practice, you can look at recent AI safety tests. In separate incidents at OpenAI, Anthropic, and Meta, autonomous models pursued their objectives in ways humans had not anticipated or intended — such as breaking into Hugging Face production infrastructure to obtain information.
It is important to add context here: the models achieved the goals assigned to them in situations where human errors, like granting unintended internet access, created openings the models knew how to exploit. They took advantage of that access because they lacked sufficient understanding of the boundaries and limits that any human would consider obvious.
As Rossi noted in an article about the OpenAI and Hugging Face incident, the natural reaction is to conclude that you just need to specify things more carefully in the instructions. But that flips the problem. The difficulty is not that we were not literal enough — it is that the intended meaning is contextual and deeply human.
Applying human judgment, she says, is not nostalgia or caution for the sake of caution. It is the recognition that meaning lives partly within us: in our sense of context, in our understanding of what truly matters in a situation, and in our ability to notice when a literal reading has ended up somewhere absurd. 🧠
How to manage the risks of interacting with AI
Beyond identifying why human participation is necessary in open-ended tasks, the study authors discuss ways to manage the risks that arise from interacting with AI systems. Their conclusion is interesting: when working with AI, it is impossible not to be influenced by it. The real question is how that influence is exercised, limited, and governed.
To address these risks, the researchers recommend designing AI systems that support comparison, reversibility, transparency, reflection, and the preservation of alternative paths — instead of treating AI as an automatic and final decision-maker.
On top of that, AI should be evaluated by how well it helps people clarify and refine their goals, without prematurely narrowing the available choices. In practice, this means building AI that supports better reasoning, not just faster delegation. That is a pretty significant mindset shift.
Governance as a compass for what is appropriate to automate
AI governance shows up in this scenario as the compass that helps organizations decide where autonomy is appropriate and where oversight is indispensable. According to Jamie VanDodick, director of AI Governance and Responsible Technology at IBM, this governance helps companies determine where autonomy makes sense, where oversight is needed, and ensures that AI systems truly amplify human decision-making capacity.
In other words, it is not about putting the brakes on innovation. It is about steering it intelligently, knowing exactly when to let go and when to keep your hands firmly on the wheel.
What this means for the future of AI
The most important takeaway — and perhaps the most counterintuitive one — is that the organizations that will get the most value from AI will likely be the ones that design this partnership between humans and machines from the very beginning. Not as a patch applied after the fact, but as a fundamental part of the design.
Humans play a crucial role in recognizing when goals evolve and in providing the context that AI agents easily miss. And even as people delegate increasingly complex tasks to agents with minimal oversight, translating human intentions into reliable instructions remains a persistent and far from trivial challenge.
Because of all this, the continued advancement of Artificial Intelligence may actually place even greater value on human judgment and oversight. The more powerful the technology gets, the more precious our ability to provide meaning, context, and direction to what it produces becomes.
At the end of the day, the study from TU Darmstadt and the Munich Center for Machine Learning is not a pessimistic warning about the limits of AI. It is an invitation to look at the relationship between humans and intelligent systems with more honesty — recognizing where the technology truly shines, where it still depends on us, and how this partnership can be built in a smarter and more responsible way. And that invitation could not have come at a better time. ⚡
