Insight AEGI Shield

Fraud Evaluation with Delayed Labels: When to Report Insufficient Evidence

A worked synthetic example explains why a positive missing-label range can still produce Insufficient Evidence, and how to preserve the comparison's scope.

Quick answer

Delayed fraud labels can leave a historical evaluation with an incomplete answer, even when the observed comparison looks favourable. Keep unknown outcomes visible, distinguish missing-label bounds from statistical uncertainty, and apply the completeness rule agreed before results were inspected.

In AEGI’s current offline supplied-score adapter, any unknown or immature label in the frozen cohort produces Insufficient Evidence. This is an explicit rule of this adapter, not a universal requirement for every fraud evaluation method.

Why a score can be ready before its outcome label

A fraud-review system assigns priority at a decision point. The outcome used to evaluate that priority may become available later. Research by Dal Pozzolo and colleagues distinguishes early investigator feedback from delayed supervised labels in credit-card fraud detection and shows why those sources cannot simply be treated as immediate, equivalent feedback.[1]

For a historical comparison, that creates two different timing questions:

  1. Was the score, and the information used to produce it, available at the decision point?
  2. Was the outcome known and eligible under the evaluation’s label cutoff and maturity rule?

A score-availability timestamp addresses the first question only when supported by provenance. A label timestamp addresses the second under the agreed contract. Neither should be used as a substitute for the other.

Unknown is not a negative label

A label of 0 means a case meets the agreed negative-outcome definition. UNKNOWN means the evaluation lacks an eligible known outcome for that case. Replacing UNKNOWN with 0 changes the question being measured.

For example, suppose the baseline and candidate each select four cases. If a candidate-selected case has no mature label, recording it as negative reduces the candidate’s counted positives without establishing its actual outcome. Dropping the case creates a different problem: the comparison no longer uses the frozen cohort and selection basis.

The safer reporting approach for this adapter is to preserve the case, report what is known and show the missing-label possibilities. The agreed decision rule then determines whether the available evidence is sufficient.

Worked example: a positive range with an incomplete answer

AEGI’s reporting rehearsal uses 12 deliberately authored cases and K=4. The incomplete-label fixture selects these candidate cases:

Case Candidate rank Label available to the evaluation
EX-001 1 1
EX-005 2 UNKNOWN
EX-006 3 1
EX-007 4 1

The baseline selects one known positive among its four cases. The candidate selects three known positives and one unknown. If the unknown outcome is 0, the candidate has three positives; if it is 1, the candidate has four.

Quantity Result
Baseline positives at K 1
Candidate positives at K Between 3 and 4
Paired difference Between +2 and +3
Known labels across the cohort 11 of 12
Proposed outcome under the frozen rule Insufficient Evidence

The positive lower bound does not satisfy this adapter’s separate requirement for complete, mature cohort labels. The report preserves both facts: the bounded comparison is favourable, and the declared evidence requirement has not been met.

The example was designed to exercise this branch and was reproduced with a successful replay check. It is not a sample of customer outcomes or evidence of model performance.

A missing-label bound is not a confidence interval

The range above enumerates possibilities for one unknown binary label. It does not estimate sampling uncertainty, future performance or the probability that the candidate is better. Calling it a “95% confidence interval” would add a statistical meaning that was never calculated.

Paired bounds also account for overlap between the two queues. If an unknown case appears in both, its contribution to the candidate-minus-baseline difference cancels. Unknown cases selected only by the candidate can increase the upper bound; unknown cases selected only by the baseline can decrease the lower bound.

That arithmetic does not remove a contract’s completeness requirement. Under the current adapter, an unknown case outside both top-K queues also yields Insufficient Evidence, even though it does not change those queues’ count difference. This deliberately conservative rule is visible and fixed before execution. A different missing-label method requires its own scoped design and review.

Freeze the timing rule before comparing results

A usable evaluation brief should record:

  • the cohort and why those cases were included;
  • the baseline and candidate score identities and availability;
  • the common decision time supported by this adapter;
  • the positive and negative outcome definitions;
  • the label cutoff and maturity rule;
  • K, the improvement threshold and the treatment of unknown labels;
  • the permitted response if labels arrive later.

The authored example uses a decision time of 1 October 2026 at 09:00 UTC, an outcome cutoff of 10 October at 09:00 UTC and a seven-day maturity setting. These values exist to make the demonstration reproducible. They are not a recommended waiting period for a bank’s labels.

An actual engagement must check that its timestamps and maturity semantics fit the workflow. The current adapter uses one common decision time. A workflow requiring per-case times or multiple observation windows needs an additional fit assessment.

Distinguish four findings from an input refusal

For the current supplied-score adapter, the descriptive proposals are:

Proposal When it is produced
Continue All cohort labels are eligible and known, and the positive-count increase at K meets the predeclared threshold.
Refine Labels meet that rule, and the difference is nonnegative but below the threshold.
Stop Labels meet that rule, and the candidate selects fewer positives at K.
Insufficient Evidence Any cohort label is unknown or ineligible under the maturity or cutoff rule.

A malformed input is handled separately. A missing score, duplicate case identifier or invalid timestamp can cause a refusal before a result is generated. It does not establish that the candidate performs worse.

The generated proposal also remains separate from the owner’s decision. An owner may halt a project because the needed data cannot be obtained, but the record should preserve “insufficient evidence” as the analytical reason rather than inventing a negative performance finding.

What to do after an insufficient result

First, identify the exact gap: unavailable labels, immature observations, a cutoff mismatch or a cohort that does not fit the agreed method. Preserve the report and frozen inputs so the limitation remains visible.

Then agree the next action. It may be appropriate to wait for an already-defined observation point, correct a mapping problem, narrow a future evaluation scope or stop because the decision cannot be supported within the available evidence. The action depends on the workflow and its owner.

If input, cutoff, K or methodology changes, create a new frozen revision and record why. A later result can supersede an earlier finding for the new scope without erasing the original evaluation history.

Questions buyers should ask

How long should we wait for fraud labels?

There is no waiting period prescribed by this adapter for every workflow. Agree the outcome definition, observation process and maturity rule with the institution before execution.

Can we assess only cases with known labels?

That would change the evaluated population and may introduce selection effects. It requires an explicitly designed comparison; the current adapter does not silently drop unknown cases.

Does Continue mean the candidate can go live?

Continue proposes review of the next step for the declared comparison. The owner still considers provenance, operating constraints and the evidence required for any subsequent stage.

Can we start without transferring customer records?

Yes. The first discussion can describe the queue, baseline, candidate, review capacity and label availability. Data handling is established before records are shared or processed.

Inspect the deliverable before scoping a run

See all four synthetic package examples, check the current input contract, or scope a Fraud Review Evaluation.

Method and source

The worked example and decision rules describe AEGI’s versioned supplied-score adapter and separately authored reporting fixtures as checked on 7 November 2026. They do not establish customer performance, statistical significance or independent verification of the source data.

[1] Dal Pozzolo, A., Boracchi, G., Caelen, O., Alippi, C. and Bontempi, G. (2015), Credit Card Fraud Detection and Concept-Drift Adaptation with Delayed Supervised Information, IJCNN. This source provides background on delayed labels in credit-card fraud detection; it does not endorse AEGI’s contract or establish an application-review result.

NEXT STEP

Bring one proposed change.

Controlled Evaluation Bring One Question