Discussion Board 7.5: Think & Share💡Trusting AI Agents at Work

My overall view is that trust in AI should be earned through evidence, not assumed because the technology is capable.

I am seeing AI automation in work at few places. e.g, "Client Sentiment Analysis and drafting response based on client sentiment".  Humans review the draft email before hitting "Send" .

I would be comfortable giving an agent more autonomy for low-risk, reversible and well-defined tasks, especially where the inputs have already been validated by humans.e.g, create a ticket with all the inputs from product/Business SME discussions. This could be corrected if anything goes wrong.

For higher-value decisions, I would always want a pause-and-review mechanism, where the agent provides the evidence and recommendation but the appropriate human makes the final call. e.g, a Pull Request approval - I would want to see the evidence before allowing the agent to commit the code.

My biggest concern would be confidence in the correctness of the decision -AI is fundamentally predictive. Even when it performs extremely well, its outcomes are not guaranteed to be 100% correct. That becomes particularly important when the consequences of an incorrect decision are high or irreversible,

From my experience, I'm already seeing AI move beyond simple productivity tools into business and technology workflows. For example, I have seen AI used to read documents, interpret the information and translate it into business-process workflows that ultimately result in transaction processing. Importantly, human approval remains in place before the transaction is executed. Similarly, Agentic AI is being used in areas such as threat modelling, where activities that previously took weeks can potentially be completed within hours. However, the quality of the outcome remains highly dependent on the quality and completeness of the inputs and context provided to the agent.

From a Risk and Controls perspective, I see a significant opportunity in regulatory compliance for new-market launches. Identifying all applicable regulatory requirements across jurisdictions can be complex and time-consuming. A potential future model could involve specialised legal and compliance agents working together to identify applicable regulatory obligations, map them against internal policies and controls, identify potential gaps and provide recommendations to Risk and Compliance professionals.  However, the judgement and the authority to make and execute decisions should still be with the business owners. 

Trust should be progressive and proportional to the potential impact of an incorrect decision. An agent summarising regulations or identifying potential requirements could operate with greater autonomy, whereas decisions affecting regulatory compliance, customer outcomes, financial transactions or market-entry approval should continue to have appropriate human oversight.

Before increasing an agent's autonomy, I would expect several conditions to be met.

1. The agent must operate on trusted, complete and appropriately governed data, with clear data ownership, lineage and access controls. This is particularly important in large organisations that have grown through mergers and acquisitions, where data and processes may be fragmented across multiple systems.
2. Appropriate governance must be established. This includes clearly defined decision rights, human accountability, audit trails, explainability, escalation mechanisms and the ability to override or suspend the agent.  An AI agent should also recognise when it does not have sufficient information or confidence to make a recommendation and escalate the matter rather than making an unsupported decision. 
3. Privacy, security and regulatory requirements must be embedded into the design rather than addressed after deployment. This is particularly important for financial institutions operating across multiple jurisdictions with different data-protection and regulatory requirements.
4. I would want evidence of sustained performance before progressively increasing an agent's authority. This means testing not only normal scenarios but also exceptions, edge cases and failure conditions, followed by continuous monitoring in production. I can relate to a scenario , that a business workflow was completely automated through agentic AI workflow and was running without any issues for almost six months hit an edge case scenario with no sufficient human to reverse the actions taken as the team was disbanded. Therefore , it is key that the management decide when is the time to increase the authority 

I would not view trust in AI as a binary choice between human and machine decision-making. I would adopt a progressive model. 

1. AI Recommend and human approve 
2. AI acts with defined boundaries 
3. AI acts autonomously with monitoring and exception escalations. 

The level of autonomy should ultimately be determined by the impact of a potential failure, confidence in the agent's performance and the reversibility of its actions or  executing the kill switch when the unexpected agents activities are noticed. In highly regulated financial organisations, accountability may remain with humans even as AI agents increasingly perform and execute significant parts of the underlying workflow.