AI Fraud Detection Pilot Due Diligence: 12 Questions Banks Should Ask

Quick answer. Before approving an AI fraud detection pilot, a bank or regulated fintech should be able to answer twelve practical questions about the decision, baseline, data, comparison, authority and evidence. A compelling demo or high model score is not enough. The aim is to decide whether one proposed change merits a controlled historical evaluation—without silently granting it production authority.

This checklist is written for Fraud Risk, Credit Risk, Model Risk, AI Governance, operations and procurement teams assessing a vendor or internal AI proposal. It is a due-diligence aid, not a substitute for the institution’s own validation, security, legal or regulatory processes.

First, define what the pilot is supposed to decide

1. Which operational decision will change?

Specify the workflow, the exact decision point and the accountable owner. “Improve fraud detection” is too broad. “Test whether one proposed signal changes the highest-priority cases investigators receive at the current daily review capacity” is a question that can be evaluated.

2. What is the current authorised baseline?

Document the existing models, rules, queue construction, analyst process and intervention policy. A candidate that is compared only with a weak or artificial baseline can look better without being useful. Record the baseline version and the time period it represents.

3. What exactly is the candidate change?

Is the vendor proposing a new model, one additional signal, a threshold change, a relationship/context layer, or a workflow change? These are different interventions. Freeze the proposed treatment before inspecting evaluation outcomes, and separate model development from comparative evaluation.

Then test whether the evidence can support the claim

4. Is the historical population fit for this decision?

Agree the cohort, time window, exclusions, product mix and intended use. Note changes in policy, seasonality, channel or fraud patterns. The result directly describes that bounded population; it is not automatically a forecast for all future customers.

5. Are outcomes and signal availability trustworthy enough?

Ask when labels became final, how disputed or missing outcomes are represented, which signals were actually available at the decision time, and who owns each source. An accurate-looking result can still contain hindsight leakage or incomplete outcomes. Cryptographic integrity of a file does not establish that its business content is true.

6. Is the comparison fair under real operating constraints?

Keep the authorised baseline and candidate on the same eligible population, outcome definition and, where applicable, the same investigator capacity. If a team can review only K cases, compare which cases enter those same K slots. Do not claim an improvement merely because the treatment sends more cases to review.

7. Which metrics would change the institution’s decision?

Consider case overlap and turnover at fixed capacity, true-positive counts under a pre-agreed label definition, false-positive burden, escalation volume, latency and uncertainty. AUC or a model-level metric alone may not answer a queue-level or operational question. Predefine what would justify Continue, Refine or Stop.

8. What happens when evidence is missing or inconsistent?

Test late or missing labels, stale inputs, conflicts between risk domains, unsupported claims, tampered artifacts and unavailable systems. Ask whether the candidate is contained, rejected or escalated, and whether a reviewer can see why. Failure behaviour is part of the evaluation, not an implementation footnote.

Finally, establish authority and reviewability

9. Can the result be reproduced against a pinned scope?

Require a record of the population, candidate and baseline identities, code/configuration versions, selected metrics and material assumptions. Can another permitted reviewer check that the result and the evidence package belong to the same run? Reproducibility is a separate question from the real-world correctness of labels.

10. Who may review, approve, stop and deploy?

Model output is a recommendation, not permission to modify policy, contact a customer or deploy a system. Name the human and institutional authority at each stage. Record stop conditions and ensure later shadow or production activity requires its own approval.

11. What will the buyer actually receive?

Ask for a bounded decision package: comparison results, control findings, evidence inventory, reproducibility context, known limitations, unresolved exceptions and an explicit recommendation to Continue, Refine or Stop. Avoid accepting a dashboard screenshot as the complete record.

12. What is the next gate—and what has not been proven?

Separate internal synthetic testing, public-dataset benchmarks, institution-controlled historical replay, shadow evaluation and production performance. A vendor should identify which stage is complete and which remains subject to data permission, human acceptance, independent assurance, security checks or production approval.

A compact vendor due-diligence request

A risk team can send the following request before any confidential data exchange:

For one proposed AI-assisted Fraud or Credit workflow change, please identify the exact decision, current baseline, candidate treatment, required outcome evidence, operating constraints, failure controls, reviewer authority and deliverable decision package. Distinguish completed synthetic or public-benchmark evidence from institution-specific validation. Describe the minimum public-safe scoping information needed for a first discussion, and the approvals required before any historical data is used.

A credible response should make it easy to say not yet when the labels, authority, baseline or technical controls are insufficient.

What AEGI can show today—and what remains open

AEGI Shield is being developed as a governed review and evaluation layer around proposed Credit and Fraud workflow changes. The commercial starting point is one Controlled Evaluation: one workflow, one baseline, one proposed change and one decision. The institution retains ownership of policy, customer-impacting actions and production progression.

AEGI has documented a bounded October 2026 synthetic engineering integration spanning Shield, a research-scope Core metric recheck and an Ops staff review/evidence handover. It also publishes the BAF-003 public-synthetic fixed-capacity benchmark, in which queue composition changed from 494 to 544 fraud-labelled applications within the same 2,417 review slots. Those are different evidence classes. Neither demonstrates bank production performance, real fraud-loss reduction or a validated outcome in a prospective institution’s population.

AEGI’s evidence hierarchy makes the distinction explicit. The material next step is an institution-controlled comparison—subject to scope, data feasibility and approval—not a claim that a simulated workflow has already passed external bank qualification.

How to start without sending customer data

The first exchange needs four public-safe elements:

1. Workflow: which bounded Fraud or Credit process is under consideration?
2. Baseline: what is currently authorised and how is it measured?
3. Proposed change: what signal, model, rule, threshold or context is being considered?
4. Decision: what evidence would make the institution Continue, Refine or Stop?

Do not email customer records, credentials, production datasets or confidential bank material through a public contact channel. Data custody and access are a later, separately approved discussion.

Explore the Controlled Evaluation approach or request a scoping conversation.

Frequently asked questions

Is a synthetic demo enough to approve an AI fraud pilot?

No. It can demonstrate bounded software behaviour and control paths, but not performance on a bank’s historical cases or the validity of real-world outcomes. A bank-specific question requires bank-authorised evidence.

Must a bank replace its fraud engine to evaluate a new signal?

Not necessarily. A proposed signal or review-context treatment can be compared with the existing authorised baseline before an institution decides whether deeper integration is warranted.

Does a first meeting require sensitive data?

No. Workflow, baseline, candidate and decision scope can be discussed without customer records. Whether a subsequent controlled evaluation is feasible depends on lawful permitted data, outcomes, owners and operating constraints.

Does a successful historical replay authorise production?

No. It informs the next decision. Shadow operation, customer-impacting action and production deployment each require the institution’s separate approval and applicable controls.

Scope and non-endorsement: This article provides a practical evaluation framework, not legal, regulatory or model-validation advice. AEGI’s public supporting results are synthetic or public-synthetic and do not constitute independent bank validation, production readiness, financial ROI, fraud-loss reduction or regulatory endorsement.