Agentic AI for Fraud Detection in Banking: From Rules Engines to Real-Time Agents

Agentic AI fraud detection replaces the static rules engine's flood of alerts with a two-layer system — machine-learning models that score transactions in real time, plus LLM investigation agents that assemble evidence into audit-ready case files — while a human analyst keeps the final say on every disposition and SAR filing.

Fraud and AML leaders are not searching for another dashboard. They are searching for a way out of alert fatigue: queues that grow faster than teams, investigators spending forty minutes per alert on cases that were never fraud, and examiners asking why backlogs keep aging. The shift now underway in banking is architectural. Rules engines are not being ripped out — they are being demoted to one signal among many, with detection models and investigation agents doing the triage work that used to consume entire departments. This article walks through that architecture, where regulation draws the line on automation, and how banks are staffing these builds.

Why static rules engines drown fraud teams in false positives

A rules engine flags transactions that match hand-written conditions: cash deposits just under $10,000, velocity spikes, transfers to high-risk geographies. Every rule is legible and auditable — which is why regulators historically liked them — but rules cannot distinguish a criminal structuring deposits from a restaurant owner with a legitimately cash-heavy week. So banks tune rules broadly to avoid missing anything, and the queue explodes.

The numbers are brutal. Industry benchmarking consistently places false positives at 90-95% of all alerts generated by rules-based transaction monitoring systems, with only roughly 2-4% of alerts actionable enough to justify escalation or a suspicious activity report. Some purely rules-based programs run false-positive rates approaching 98%. Meanwhile, the LexisNexis Risk Solutions True Cost of Financial Crime Compliance research put global compliance spend above $270 billion, with alert-triage inefficiency cited as a leading driver of waste. Banks commonly devote 10-15% of total headcount to KYC and AML activity — and the industry still detects only a small fraction of global financial crime flows.

When nineteen of every twenty alerts are noise, the alert queue stops being a control and becomes a liability: it burns out investigators, ages real cases past regulatory deadlines, and trains everyone to expect that the next alert is nothing.

The false-positive problem is not just cost. Every hour spent clearing a noise alert is an hour a genuinely suspicious pattern sits uninvestigated. Alert fatigue is a detection failure wearing an efficiency costume.

What an agentic fraud detection architecture actually looks like

An agentic architecture splits the work rules engines were never designed to do into three layers, each doing what it is best at:

• Detection layer — ML risk models. Supervised models trained on confirmed fraud outcomes and unsupervised anomaly detection score transactions in milliseconds. Rules still run — they encode regulatory expectations and known typologies — but their output becomes a feature feeding the risk score, not an automatic case.
• Investigation layer — LLM agents. When a transaction or pattern crosses the risk threshold, an investigation agent does what a human analyst does in the first forty minutes of a case: pulls KYC records and beneficial ownership data, reconstructs the counterparty graph, summarizes 90 days of account behavior, checks sanctions and adverse-media hits, and compares the activity against the customer's stated profile. The output is a structured case file with cited evidence and a recommended disposition — close as false positive, continue monitoring, or escalate.
• Disposition layer — human analysts. An investigator reviews the case file, agrees or disagrees, and makes the call. Every agent action, retrieved document, and reasoning step is logged so the file stands up to audit.

This is no longer speculative. Nasdaq Verafin announced agentic AML and fraud analyst rollouts for the second half of 2026 that triage alerts "similarly to how experienced human investigators manage cases," and FIS is building a Financial Crimes AI Agent with Anthropic — with BMO and Amalgamated Bank in development — aimed at compressing AML investigations from days to minutes while improving SAR narrative quality. Whether a bank buys a platform or builds on its own case-management stack, the pattern is the same: models detect, agents investigate, humans decide. It is a specific instance of the general agentic deployment playbook — and like every enterprise agent, fraud agents need scoped tool access, deterministic audit trails, and human-in-the-loop controls designed in from day one, not bolted on for the exam.

The false-positive payoff comes from two places. The detection layer suppresses alerts a rules engine would have fired on behaviorally normal activity. The investigation layer then clears surviving noise in minutes instead of half-hours, with consistent documentation a tired human rarely matches at alert forty of the day. Banks running these layers in parallel with legacy rules typically report false-positive reductions in the 40-70% range — real, but earned through months of tuning and back-testing, not switched on.

Where BSA/AML and FinCEN SAR rules draw the line on automation

The regulatory boundary is clear, and well-designed agentic systems respect it: AI can investigate; it cannot decide.

Under the Bank Secrecy Act, a bank must file a Suspicious Activity Report with FinCEN no later than 30 calendar days after the initial detection of facts that may constitute a basis for filing — extendable to 60 days only when no suspect has been identified. The bank must retain the SAR and supporting documentation for five years. The FFIEC BSA/AML examination manual makes clear that examiners will test the integrity of the entire alert-to-SAR pipeline: how alerts are generated, how dispositions are reached, and whether the reasoning is documented.

Three implications for agentic systems:

1. The filing decision is human. An agent can draft a SAR narrative — often better-structured than a rushed analyst's — but a qualified BSA officer or designated analyst must review the case file, own the judgment, and sign off. "The model closed it" is not an answer an examiner accepts.
2. Closure logic must be explainable. Auto-closing low-risk alerts is where regulators probe hardest. If an agent recommends closure, the case file must show exactly what was checked and why the activity was consistent with the customer's profile — evidence, not a confidence score.
3. The clock does not care about your architecture. The 30-day window starts at initial detection. Agentic triage actually helps here — it collapses the investigation phase that eats most of the window — but banks must show that automation never delayed detection or filing.

Model risk management applies too. Federal Reserve SR 11-7 expectations — validation, documentation, ongoing monitoring — extend to detection models and, in most examiners' current reading, to the LLM agents acting on their output. Banks that already run mature compliance programs for AI under FINRA and SEC oversight will find the BSA-side governance familiar; banks that do not should build it before the first agent touches a live alert.

Evals, drift monitoring, and adversarial testing for fraud agents

Fraud is the rare domain where your system's errors are studied by a motivated adversary. That makes production evaluation non-negotiable, and it looks different from a chatbot eval program:

• Golden-case eval suites. Build regression sets from historical alerts with known ground truth — confirmed fraud, confirmed clean, and the ambiguous middle. Every agent or model change must replay against these before release, measuring not just disposition accuracy but evidence completeness and narrative quality.
• Drift monitoring. Transaction mixes shift with seasons, product launches, and fraud fashion. Monitor detection-score distributions, alert volumes by typology, agent recommendation rates, and — critically — human override rates. A rising override rate is the earliest signal that the agent's judgment has drifted from your analysts'.
• Adversarial testing. Red-team the pipeline the way a fraud ring would: structuring patterns tuned to sit under model thresholds, synthetic identities engineered to look profile-consistent, and prompt-injection attempts hidden in transaction memos or counterparty names that could steer an LLM agent's summary. The last category is unique to agentic systems and routinely missed.
• Feedback loops. Every human disposition is a labeled example. Feed confirmed outcomes back into detection-model retraining and agent eval sets, so the system sharpens on your fraud, not a vendor's generic corpus.

A fraud agent that passed evals in March and hasn't been re-tested since is not the agent you deployed — the adversary has been iterating even if you haven't.

This discipline is the fraud-specific application of agent evals in production: the same checks that gate a release should run continuously against live behavior, with drift alarms treated as seriously as detection misses.

Human-in-the-loop disposition and audit-ready case files

The design goal is not to remove analysts — it is to move them up the value chain. In a mature agentic program, investigators stop doing data gathering and start doing judgment: reviewing assembled case files, handling the genuinely ambiguous cases, dispositioning escalations, and quality-checking a sample of agent-recommended closures.

An audit-ready case file from an investigation agent should contain, at minimum: the triggering alert and risk score with contributing features; every data source the agent queried, with timestamps; the evidence retrieved, cited inline in the narrative; the agent's recommended disposition and its reasoning; the human reviewer's identity, decision, and any disagreement with the recommendation; and the complete action log. Disagreements are gold — they are simultaneously an audit trail, a training signal, and the honest measure of whether the agent is earning trust.

Sampling matters as much as review. Even for alerts the agent recommends closing, a defined percentage should get full human re-investigation, with results tracked over time. That sample rate can fall as demonstrated accuracy rises — but it should never reach zero, because it is the control that makes the whole architecture defensible to an examiner.

Team shape: fraud SMEs plus contract AI engineers, not a multi-year internal build

The banks getting these systems into production are not the ones that spent eighteen months hiring an internal AI research team. They are the ones that paired the expertise they already had with the engineering they did not.

The internal side is non-negotiable: BSA officers and fraud investigators who know the institution's typologies, risk appetite, and examiner history must own detection policy, disposition standards, and the SAR decision. What most banks lack is the other half — engineers who have built LLM agent pipelines, retrieval over case-management systems, eval harnesses, and the observability layer that makes agent behavior auditable. That talent is scarce, expensive to hire permanently, and — for a build-then-operate project — arguably wrong to hire permanently at all. The build phase needs four to six specialized engineers; steady-state operations needs one or two.

This is where the staff-augmentation-versus-hiring math favors contract talent decisively. Gain America deploys contract AI engineers who have shipped agentic systems in regulated environments — engineers who embed with the bank's fraud and compliance teams, work inside the bank's data perimeter, and hand off a documented, eval-gated system the internal team can run. Our financial services AI practice pairs these forward-deployed engineers with the model-risk and audit-documentation workstreams from day one, because in banking a fraud agent that works but cannot survive an exam does not actually work.

A realistic sequence for a mid-size institution: two to three months building the detection-plus-agent pipeline against historical alert data and standing up the eval suite; two to three months running in shadow mode alongside the existing rules queue, measuring agreement with human dispositions; then a staged cutover where the agent takes first-pass triage on low-risk typologies while humans keep every final call. Twelve months in, the typical outcome is a materially smaller queue, faster SAR cycle times, investigators working real cases instead of noise — and a case-file archive that makes the next exam easier, not harder.

The rules engine got banking's fraud program this far. The queue it created is now the biggest obstacle to catching what it was built to catch. The way out is not better rules — it is agents that investigate like your best analyst, evals that keep them honest, and humans who still make every call that matters.