2.1 million approvals research visual

When teams deploy AI agents into critical business operations, their first instinct is almost always binary: either the agent operates with complete autonomy, or every single action requires manual review. Both extremes fail in production.

Over the past eighteen months, Synthora processed over 2.1 million human signoffs across 42 enterprise engineering, finance, and customer service teams. In this paper, we analyze the conditions under which human operators approve, modify, or reject agent actions — and share concrete guidelines for configuring approval policies that stay safe without burning out your team.

Key Empirical Takeaway

Approval accuracy plummets by 44% when operators are asked to review more than 30 routine actions per hour. True governance requires selective, high-confidence gating rather than blanket rubber-stamping.

1. The Rubber-Stamp Paradox

The most dangerous failure mode of human-in-the-loop systems is not an agent making a mistake — it is a fatigued human clicking "Approve" without reading the evidence.

In our analysis of accounts payable runs below $500, when operators received more than 15 approval notifications per morning, their average dwell time before clicking "Approve" dropped from 14.2 seconds down to 1.8 seconds. At 1.8 seconds, humans are not auditing invoices; they are simply acting as an inefficient human CAPTCHA.

"The goal of automation is not to eliminate human judgment, but to amplify human scrutiny where financial and legal risk actually lives."

2. Where Humans Intervene Most

Across the 2.1 million data points, the overall override rate was 4.8%. However, the distribution was heavily skewed across transaction value and data source reliability:

Workflow Type Volume Examined Avg. Review Time Human Override %
AP Invoices under $1,000840,000 runs4.2s1.2%
AP Invoices $1,000 – $10,000620,000 runs18.6s6.4%
AP Invoices over $10,000110,000 runs44.1s28.9%
Customer Refunds (Support)380,000 runs8.5s5.1%
Vendor Contract Onboarding150,000 runs62.0s34.2%

3. The 94% Confidence Cliff

Modern LLMs frequently generate confidence scores, but raw model confidence is notoriously miscalibrated on tabular ERP entries. An agent might report "98% confidence" simply because the prompt syntax looks standard, even when a quantity mismatch exists between a purchase order and warehouse receiving slip.

Synthora solved this by replacing singular model confidence with triangulated source verification:

  • Direct primary key match: Did the invoice ID exist in NetSuite before the run started?
  • Cross-system parity check: Does the Slack receiving log signed by the warehouse foreman match the physical pallets billed?
  • Historical vendor variance: Has this vendor ever submitted invoices with pricing deviations in the past 12 months?

4. Designing Better Policies: Programmable YAML Gates

Instead of hardcoded rules, top-performing operations teams encode governance as plain, auditable YAML playbooks:

policy-rule-ap-high-value.yaml 
policy:
  name: "High-Value Invoice Exception Gate"
  version: "2.1"
  triggers:
    - event: "netsuite.invoice.received"
  rules:
    - if: invoice.amount > 10000 or vendor.flagged == true
      then:
        require_approval:
          role: "ap_lead"
          channels: ["slack", "email"]
          timeout: "4h"
          on_timeout: "escalate_to_cfo"
    - else:
      auto_approve_if:
        confidence_gte: 0.96
        po_match: "exact"

5. Five Practical Rules for Ops Leaders

  1. Never gate on routine parity: If an invoice perfectly matches an approved purchase order and receiving slip, let it execute autonomously.
  2. Gate on asymmetry: Escalate immediately when quantity, payment terms, or remit-to bank routing numbers deviate.
  3. Provide one-click context: Never send an alert that says "Run needs approval." Send the delta: "Billed 5 pallets ($18,420), warehouse logged 3 ($11,052)."
  4. Always preserve reverse capability: Ensure every automated write can be rolled back without requiring database administrators.
  5. Audit the approvers: Periodically measure how long operators spend reviewing runs to detect fatigue before compliance gaps occur.
ML

Dr. Maya Lin

Head of Research at Synthora. Previously leading systems safety at Stripe and distributed data systems at Palantir. PhD in Computer Science from Stanford University.