The Forensic Black Swan Problem: How Should Science Handle Evidence From Events That Have Never Been Seen Before?

Novelty, open-set recognition, and the limits of validation when forensic evidence falls outside every reference population science has ever built.

Contents

• Executive Summary
• Key Findings
• 1. The Evidence Outside the Map
• 2. What Does "Novel" Actually Mean?
• 3. Unknown vs. Unclassified
• 4. The Boundary of Validation
• 5. The Reference Dataset Problem
• 6. The Ground Truth Problem
• 7. The Unknown Error Rate Problem
• 8. Open-Set Forensic Science
• 9. When Anomaly Is Mistaken for Crime
• 10. Novel Psychoactive Substances
• 11. Emerging Cybercrime
• 12. Synthetic Media Detection
• 13. Unusual Trace Evidence
• 14. The False Novelty Problem
• 15. The Novelty Confirmation Framework
• 16. The Right to Abstain
• 17. Bayesian Reasoning and Unknown Hypotheses
• 18. The Unknown Unknown Problem
• 19. AI and the Acceleration of Novelty
• 20. How Should Laboratories Respond?
• 21. The Forensic Novelty Response Framework
• 22. The Forensic Novelty Confidence Gradient
• 23. Cross-Domain Comparison
• 24. Major Research Gaps
• 25. Future Research Agenda
• Infographic Concept
• Conclusion
• References

Executive Summary

A mass spectrometer produces a clean, reproducible signal. The chromatography is sound, the calibration is current, and the peak does not match anything in the laboratory's spectral library. A deepfake-detection classifier returns a confident "authentic" verdict on a video generated by an architecture that did not exist when the classifier was trained. A digital forensic examiner finds a process behaviour on a compromised server that resembles nothing in any prior case file. In each instance, the underlying scientific question is the same, and it is harder than it first appears: is the laboratory looking at something genuinely new, or is it looking at a familiar phenomenon that its own methods are not equipped to recognise?

This review develops that question as a distinct methodological problem, which it terms — as a proposed conceptual framework rather than an established doctrine — the forensic black swan problem: the situation in which evidence falls materially outside the validated representational domain of available reference populations, analytical models, or interpretive frameworks (Taleb, 2007; Geng et al., 2021). The term borrows its imagery, but not its full philosophical apparatus, from Nassim Nicholas Taleb's account of statistically extreme, retrospectively-rationalised outlier events (Taleb, 2007). Forensic evidence rarely meets Taleb's strict criteria of unpredictability and civilisational consequence; what forensic science more often confronts is a narrower, more tractable cousin of that problem — evidence that a specific validated method, database, or classifier was never built to recognise, which is a statement about the boundaries of a knowledge system rather than a metaphysical claim about unpredictability itself.

The review's central argument is that forensic science's demonstrated strength — high accuracy within well-characterised, densely populated reference domains such as human fingerprint comparison, controlled-substance identification against curated spectral libraries, and closed-set digital forgery detection — is inseparable from a corresponding vulnerability at the domain's edge. Validation studies, by construction, describe performance under the conditions the study created. When evidence originates from a class of phenomena that no validation study, reference database, or training corpus has ever represented, the demonstrated accuracy of the method does not automatically travel with it (Cui & Wang, 2022; Lu et al., 2024).

Yet the review resists the temptation to romanticise novelty. Drawing on open-set recognition research (Geng et al., 2021; Chen et al., 2021), out-of-distribution detection (Cui & Wang, 2022), classical novelty-detection theory (Markou & Singh, 2003a, 2003b; Chandola et al., 2009), and the forensic cognitive-bias literature (Dror et al., 2006; Kassin et al., 2013), it develops a taxonomy distinguishing at least seven ways an observation can appear unprecedented without being genuinely new — including analytical artefact, instrument failure, database gaps, and misclassification. The central methodological hazard the review identifies is not that forensic science will fail to notice a genuinely novel phenomenon; it is that forensic science will notice something anomalous and resolve the discomfort of "I don't know" too quickly, by forcing an unfamiliar observation into a familiar category that happens to be available.

The review examines this hazard across four evidentiary domains — novel psychoactive substances and non-targeted toxicological screening (Pasin et al., 2017; National Institute of Justice, 2021), emerging cybercrime and zero-day digital artefacts, synthetic media authentication under adversarial and generative drift (Villari et al., 2025; Soni, 2024), and unusual physical trace evidence — and finds a structurally similar pattern in each: reference libraries and training corpora lag the phenomena they are meant to characterise, closed-set assumptions quietly persist inside systems described as general-purpose, and the resulting gap is bridged, in practice, by human judgement that is itself vulnerable to contextual and confirmation bias (Dror et al., 2006).

Drawing on the philosophy-of-science literature on the "problem of unconceived alternatives" (Stanford, 2006; Ruhmkorff, 2011) and on open-set machine learning's formalisation of a classifier's "right to reject" (Geng et al., 2021), the review proposes two original conceptual instruments intended for critical use rather than adoption as standards: a seven-stage Forensic Novelty Confirmation Framework for testing whether an anomaly is genuinely novel, and a Forensic Novelty Response Framework (Detect–Verify–Characterise–Compare–Challenge–Replicate–Classify or Abstain) for laboratory practice. It argues that scientific abstention — the documented, reasoned refusal to force a classification the evidence does not support — deserves recognition as a marker of scientific maturity rather than professional failure, a position consistent with the logic, though not the specific procedures, of open-set recognition research (Geng et al., 2021) and with PCAST's (2016) insistence that categorical conclusions require demonstrated foundational validity.

India-specific material is included only where the literature and reporting directly support it: the rapid emergence of new psychoactive substances documented in Indian clinical literature, and reporting from the 2026 India AI Impact Summit in which cybersecurity practitioners stated plainly that no model can yet reliably detect a well-made deepfake, alongside documented gaps in police training for AI-enabled financial crime (Medianama, 2026). No broader claim about Indian forensic capacity is made beyond what these sources state.

Key Findings

• Absence of a reference-library or database match is evidence of absent knowledge, not evidence of novelty; the two are frequently conflated in both casework and public communication of forensic results.
• Closed-set assumptions persist inside systems marketed as general-purpose. Deepfake detectors trained and evaluated on known manipulation families show measurable, sometimes severe, accuracy loss against manipulation techniques absent from training data (Villari et al., 2025).
• Demonstrated error rates are properties of a validation population, not of a method in the abstract. PCAST's (2016) black-box-study criteria were explicitly scoped to the populations and conditions each study sampled, and subsequent statistical work has shown that even those figures are sensitive to how missing or "inconclusive" examiner responses are handled (Khan & Carriquiry, 2023).
• Non-targeted, library-independent analytical strategies exist and are actively used in forensic toxicology precisely because targeted, library-dependent methods structurally cannot find what they were not built to look for (Pasin et al., 2017).
• Open-set recognition research provides a formal, non-forensic-specific vocabulary — "known known," "unknown unknown," rejection thresholds, open-space risk — for a problem forensic science has so far mostly discussed informally (Geng et al., 2021).
• Cognitive bias research demonstrates that the temptation to resolve an anomaly by classification is not merely a database problem but a documented human decision-making vulnerability, most acute precisely where no independent ground truth exists to check the conclusion against (Dror et al., 2006; Kassin et al., 2013).
• The philosophical "problem of unconceived alternatives" (Stanford, 2006) offers a caution against overconfidence in any framework — including the ones proposed in this review — since a framework designed to catch known categories of interpretive failure cannot, by definition, guarantee it has anticipated the categories it has not yet conceived.

1. The Evidence Outside the Map

Validation, in the ordinary practice of forensic science, means comparing a method's output against known answers across a defined population of samples, cases, or conditions. The resulting statement of accuracy or error is therefore always a statement about a territory the validation study charted (National Research Council, 2009; PCAST, 2016). Casework, however, does not confine itself to charted territory. An analytical instrument, a comparison algorithm, or a trained examiner can be asked to interpret evidence that originates from outside every population the underlying validation ever sampled — a new precursor chemistry, an unfamiliar file-system artefact, a generative architecture that postdates every training corpus in existence.

The map metaphor matters because it locates the problem correctly: the issue is not that the method is unreliable within its charted domain, nor that the evidence is inherently unknowable. It is that scientific confidence, once established inside a domain, does not automatically extend to the domain's exterior, and the boundary between "inside" and "outside" is frequently invisible from the perspective of a working analyst who has no principled way to know, in the moment, whether the case in front of them lies on the map or past its edge.

2. What Does "Novel" Actually Mean?

"This has never been seen before" collapses several distinct scientific situations that require different responses. The review proposes the following working taxonomy, adapted from general anomaly- and novelty-detection theory (Markou & Singh, 2003a; Chandola et al., 2009) and from forensic error-source literature (Dror et al., 2006), for critical use rather than as an established classification system.

Type	Description	Typical Resolution
I — Previously known, locally unrecognised	The phenomenon is documented in the wider literature but unfamiliar to the specific analyst or laboratory.	Literature search; consultation; reference-library update.
II — Rare variant of a known phenomenon	An unusual expression of an established class (e.g., an atypical drug metabolite, a rare allele).	Comparison against expanded population data.
III — Known phenomenon outside the reference database	The phenomenon exists and may be documented elsewhere but is absent from the database actually consulted.	Reference expansion; non-targeted screening (Pasin et al., 2017).
IV — Analytical artefact	An instrumental or procedural by-product mimicking a real signal.	Artefact-exclusion protocols; replicate analysis.
V — Instrument or software failure	A fault condition rather than a true observation.	Instrument diagnostics; software version audit.
VI — Misinterpretation of known evidence	The evidence is ordinary; the interpretive framework applied to it is wrong.	Independent re-examination; blind verification (Dror et al., 2006).
VII — Genuinely novel phenomenon	The evidence corresponds to no adequately characterised prior class.	Structured novelty confirmation (see Section 15); possible scientific abstention.

These categories are analytically distinct but empirically difficult to separate in the moment of discovery, which is precisely why the distinction matters. A laboratory that treats every unmatched result as Type VII risks manufacturing false discoveries; a laboratory that reflexively treats every unmatched result as Type IV or V risks suppressing genuine findings — including, in toxicology, the earliest signals of a new substance entering circulation (National Institute of Justice, 2021).

3. The Difference Between Unknown and Unclassified

Open-set recognition research draws a sharp technical line that forensic practice often blurs: a classifier's inability to assign a confident label is not the same statement as "this belongs to no known class." Geng et al. (2021) formalise this as the distinction between "known known classes" — categories represented in training or validation data — and "unknown unknown classes," for which the system has no representation at all; a system operating under closed-set assumptions will, by construction, force every input into one of its known categories, often with high stated confidence, regardless of whether the input actually belongs there.

The forensic corollary is direct: a system's failure to classify, or its confident misclassification, is evidence about the system's coverage, not a scientific finding about the evidence's ontological status. Absence of a match is not proof of absence of a match — it may simply be proof that the comparison space searched was too small.

4. The Boundary of Validation

PCAST's (2016) framework for foundational validity rests on empirical black-box studies conducted under specific, documented conditions — sample types, examiner populations, comparison protocols. Monson et al. (2022), for instance, report false-positive rates of 0.656% and 0.933% for firearms comparisons across bullets and cartridge cases respectively, drawn from 173 examiners performing 8,640 comparisons of specific ammunition types. These figures are real, useful, and specific to the conditions the study created; PCAST's own report was explicit that validity claims cannot be extended past the populations a study actually examined (President's Council of Advisors on Science and Technology, 2016).

External validity — whether a result generalises beyond the conditions that produced it — is therefore not a peripheral statistical footnote but the central question a forensic laboratory must ask before applying any validated method to evidence that differs materially from the validation population in substance, technology, degradation state, or provenance.

5. The Reference Dataset Problem

Every comparison-based forensic discipline depends on a reference population: a spectral library, an allele-frequency database, a corpus of known-manipulation training images. Population-genetics research on forensic DNA databases demonstrates the practical stakes of reference incompleteness directly — the statistical weight assigned to a DNA match depends materially on which reference population is used to estimate allele frequency, and mismatched or incomplete reference populations distort that weight in ways that are not always conservative (Fazl-Ersi et al., 2022, discussing distributional shift generally; see also population-genetics literature on reference-database representativeness).

In toxicology, the same structural gap is explicit and well documented: certified reference standards are frequently unavailable for new psychoactive substances at the point they first appear, and library-matching approaches are, by design, unable to identify a compound the library does not contain (Pasin et al., 2017). The absence of a reference match in such cases reflects the pace of the reference library's construction, not the pace of the phenomenon's emergence.

6. The Ground Truth Problem

Forensic validation depends on comparing a method's conclusion against an independently known correct answer — but casework exists precisely because that answer is not independently known. This structural tension has been examined in a companion review published on this platform, which distinguishes experimentally constructed ground truth, simulated ground truth, corroborated operational truth, and partially observable historical truth, arguing that these categories are not interchangeable even though courtroom and laboratory discourse frequently treats "ground truth" as a single portable concept (Budding Forensic Expert, 2026).

The novelty problem sharpens this tension further. Even where ground truth is available for validating a method against known classes, no ground truth exists — by definition — for a class the validation study never sampled. A black-box study can tell a laboratory how often examiners are correct about samples resembling those in the study; it cannot tell them how often they will be correct about a sample from outside the study's population, because no independently verified answer for that population currently exists to check against.

7. The Unknown Error Rate Problem

Khan and Carriquiry (2023) demonstrate that even error rates computed within a validated population are more fragile than commonly reported: applying hierarchical Bayesian models that properly account for examiner non-response and "inconclusive" answers, they show that error rates publicly reported as low as 0.4% could plausibly be as high as 8.4%, or over 28%, depending on how missing and inconclusive responses are treated statistically. If a rate this carefully studied is this sensitive to methodological choices within its own validated domain, the assumption that the same rate applies unchanged to evidence from an entirely unvalidated domain is unsupported by the data that produced the original figure.

The scientifically defensible position is narrower than either extreme: a validated error rate describes performance under validated conditions, uncertainty about performance necessarily grows as evidence departs from those conditions, and that growing uncertainty should be reported rather than absorbed silently into an unqualified number carried into court.

8. Open-Set Forensic Science

Open-set recognition treats the ability to say "this does not belong to any class I am equipped to identify" as a core design requirement rather than a system failure (Geng et al., 2021; Chen et al., 2021). Chen et al. (2021) formalise this through the concept of "open space risk" — the risk incurred whenever a classifier extends confident labels into regions of feature space that no training data actually occupied — and propose minimising that risk directly rather than treating it as an unavoidable cost of forced classification.

Proposed principle — The Right to Abstain:

A scientifically responsible forensic system — human or computational — must be capable of concluding: "this evidence does not correspond to any class I am currently validated to identify." This is a conceptual proposal drawn from the logic of open-set recognition research, not an established forensic standard, and its practical implementation would need discipline-specific development, peer review, and institutional buy-in before adoption.

9. When Anomaly Is Mistaken for Crime

Unusual is not synonymous with criminal, and unfamiliar is not synonymous with malicious. A rare but entirely benign software update behaviour, an uncommon but lawful chemical compound, or a legitimate outlier in a population sample can present exactly the same surface signature as evidence of wrongdoing: a result the system was not built to recognise. Anomaly-detection theory has long distinguished statistical rarity from meaningful deviation (Chandola et al., 2009); forensic practice must make the same distinction explicit, because the cost of collapsing "I have not seen this before" into "this indicates wrongdoing" falls directly on the person whose evidence produced the anomaly.

10. Novel Psychoactive Substances: A Model of Scientific Catch-Up

Forensic toxicology offers the most mature working example of institutionalised novelty management. New psychoactive substances are deliberately engineered, in many cases, to sit just outside existing legal and analytical definitions, and their chemical structures change fast enough that certified reference standards routinely postdate a substance's street appearance (National Institute of Justice, 2021; Pasin et al., 2017). The field's response has been methodological rather than purely regulatory: non-targeted, high-resolution mass spectrometry screening approaches are explicitly designed to detect and tentatively characterise compounds without requiring a pre-existing certified standard or a complete spectral library entry, using diagnostic fragmentation patterns and accurate-mass data instead of a direct library match (Pasin et al., 2017).

This is, in effect, a working instance of the open-set principle applied outside machine learning: an analytical strategy built on the explicit assumption that the target of interest may not yet exist in any reference collection. Indian clinical literature documents the same underlying pressure — new psychoactive substances arriving faster than clinical and laboratory familiarity can track them, with treating clinicians and, in under-resourced settings, laboratories themselves sometimes unaware such substances exist before a case presents.

11. Emerging Cybercrime: The Moving Baseline Problem

Digital forensics faces a distinctive version of the novelty problem: the definition of "normal" system behaviour is itself a moving target, revised by every software update, platform migration, and firmware release. Zero-day detection research has responded by shifting away from signature-based matching — which is structurally incapable of catching an attack pattern that has never been catalogued — toward behavioural-baseline and anomaly-based approaches that flag deviation from learned normal activity rather than matching against known threat signatures.

The methodological cost of this shift is that the baseline itself must be continuously re-established as legitimate software evolves, or the system will drown in false positives generated by ordinary platform change rather than malicious activity — a direct digital-forensic analogue of the false-novelty problem discussed in Section 14.

12. Synthetic Media: Detection After the Unknown Manipulation

Synthetic-media authentication is, empirically, one of the clearest documented cases of closed-set fragility in an operational forensic-adjacent field. Multiple independent studies report that deepfake detectors trained on a given set of manipulation techniques experience measurable accuracy degradation when evaluated against manipulation techniques absent from their training data — a pattern described consistently across CNN-based, frequency-domain, and hybrid detection architectures (Villari et al., 2025). Cross-manipulation evaluation protocols, in which a detector is deliberately trained on some manipulation families and tested on others, exist specifically because in-distribution performance figures were found to be poor predictors of real-world reliability.

This has direct, documented consequences outside the research literature: at the 2026 India AI Impact Summit, cybersecurity practitioners stated plainly that no current model can reliably detect a well-produced deepfake, and separately identified gaps in police training for AI-enabled financial crime investigation as a compounding factor (Medianama, 2026). The provenance-based alternative to detection-after-the-fact — cryptographically signed content credentials under the C2PA standard — records how content was created and edited but was explicitly not designed to determine truthfulness or authenticity of the underlying events depicted, and independent security analysis has identified structural limits in how far its guarantees extend even for content that carries a credential (Coalition for Content Provenance and Authenticity, 2024; Golaszewski et al., 2026). Provenance and detection therefore address different halves of the same problem, and neither alone resolves the unknown-manipulation gap.

13. Unusual Trace Evidence and the Problem of Scientific Silence

Not every forensic domain has the luxury of a large, continuously updated literature. For rare materials, unusual transfer mechanisms, or atypical environmental degradation, the relevant experimental literature may simply be sparse or absent. In such cases, the scientifically responsible position is not to extrapolate confidently from adjacent, better-studied phenomena, but to state plainly what the current evidence base does and does not support — a discipline of communicating the limits of knowledge that is easier to state than to practise under the institutional pressure to produce a conclusive answer.

14. The False Novelty Problem

The excitement of an apparent discovery is itself a documented source of interpretive risk. Cognitive-bias research in forensic science shows that confirmation bias and contextual bias operate most powerfully in precisely the conditions a novelty claim creates: no independent ground truth to check the conclusion against, an examiner motivated to find something scientifically noteworthy, and often institutional or reputational incentive attached to a positive finding (Dror et al., 2006; Kassin et al., 2013). Apparent novelty can equally arise from contamination, instrument drift, software versioning artefacts, or a database gap mistaken for a true absence — the Type IV and Type V categories described in Section 2.

15. The Novelty Confirmation Framework

Building on the general anomaly-to-novelty literature (Chandola et al., 2009; Pimentel et al., 2014) and on the artefact-exclusion logic already implicit in analytical chemistry practice, the review proposes the following seven-stage sequence as a conceptual framework for evaluating a claim of forensic novelty — explicitly a proposed heuristic for critical use, not an established international standard.

1

Observation. An unusual result is recorded and documented in detail before any interpretation is attempted.

2

Analytical Verification. The measurement itself is independently confirmed — repeat analysis, alternate instrument, alternate operator.

3

Artefact Exclusion. Instrumental, procedural, and software-version explanations are actively investigated and ruled out, not merely assumed absent.

4

Reference Expansion. Broader, non-proprietary, and international reference sources are consulted beyond the laboratory's default library.

5

Alternative Explanation Testing. Known, mundane explanations are deliberately tested against the observation before novelty is entertained further.

6

Independent Replication. Where feasible, the observation is reproduced by an independent party without knowledge of the original finding.

7

Novelty Assessment. Only after Stages 1–6 does the laboratory assess the remaining evidence for genuine novelty — and even then, with an explicit statement of remaining uncertainty.

16. The Right to Abstain: A Scientific Principle for Unknown Evidence

Scientific maturity is not always demonstrated by reaching a conclusion. It can equally be demonstrated by a documented, reasoned decision that available methods do not currently justify one. This principle draws directly on the logic of open-set recognition's rejection option (Geng et al., 2021) and aligns with PCAST's (2016) insistence that a categorical conclusion requires demonstrated foundational validity — a requirement that, read carefully, implies its own converse: where foundational validity for the specific evidence class has not been demonstrated, a categorical conclusion is not scientifically supportable.

Abstention should be distinguished carefully from adjacent but different states: "inconclusive" (the method was applied but produced no determinate answer), "insufficient data" (more material or measurement would resolve the question), "method outside scope" (a different method might resolve it), and "no result" (a procedural failure occurred). Each implies a different next step, and collapsing them into an undifferentiated "unknown" discards information a court or investigator needs.

17. Bayesian Reasoning and Unknown Hypotheses

Bayesian inference updates belief across an enumerated set of hypotheses in light of evidence — but the framework says nothing about hypotheses that have not yet been formulated. This is not a minor technical gap; it is the specific subject of Kyle Stanford's "problem of unconceived alternatives," which argues from the historical record of science that investigators — individually and, Stanford contends, even collectively as scientific communities — have repeatedly failed to conceive of relevant alternative explanations that later evidence supported (Stanford, 2006). Ruhmkorff (2011) pushes back on the strength of this claim, arguing that a documented historical pattern of individual failure does not establish that science as a corporate enterprise is structurally unable to generate the missing alternative eventually — a caution the review takes seriously rather than treating Stanford's thesis as settled.

Applied to forensic reasoning, the practical implication is modest but important: a Bayesian analysis is only as complete as its hypothesis space, and no explicit accounting exists within standard Bayesian machinery for the possibility that the true explanation lies outside every hypothesis currently on the list. Some form of residual "catch-all" probability mass — acknowledged explicitly rather than implicitly assumed away — is the most honest way current statistical practice has of representing this gap, imperfect as that solution is.

18. The Unknown Unknown Problem

The familiar distinction between known knowns, known unknowns, and unknown unknowns maps directly onto Stanford's (2006) philosophical argument and onto open-set recognition's formal vocabulary of "known known classes" and "unknown unknown classes" (Geng et al., 2021) — three independent literatures converging on the same structural point. A validation framework can be designed to test rigorously for failure modes its designers have already imagined; by definition, it cannot be designed to test for failure modes no one involved in its construction has yet conceived. This is not a solvable engineering problem so much as a permanent structural feature of any validation exercise, and the honest response is not to claim the gap has been closed but to build institutional habits — external audit, adversarial testing, cross-domain review — that increase the odds of surfacing what the original designers missed.

19. AI and the Acceleration of Forensic Novelty

Generative AI systems, automated cyber-attack tooling, and algorithmically produced synthetic documents share a common property relevant to this review: their rate of technical change now plausibly outpaces the multi-year cycle typically required to conduct, publish, and peer-review a forensic validation study. The documented degradation of deepfake detectors against unseen manipulation architectures (Villari et al., 2025) is one measured instance of this dynamic; the broader question the review raises, without claiming a settled answer, is whether forensic science is entering a period in which evidential novelty is generated faster than traditional validation science can characterise it — and if so, whether continuous or rolling validation models, rather than periodic ones, become a methodological necessity rather than an optional refinement.

20. How Should Forensic Laboratories Respond?

The review's recommendations are deliberately framed at the level of scientific practice rather than operational procedure, consistent with the requirement that this discussion not function as a technical manual for evading detection or facilitating misuse. Laboratories can reasonably be expected to treat method scope as an explicit, documented boundary rather than an implicit assumption; to invest in reference-library expansion and non-targeted screening capability precisely because targeted methods cannot find what is absent from their target list (Pasin et al., 2017); to document anomalies systematically even when — especially when — no immediate explanation is available; to report uncertainty honestly rather than defaulting to categorical language; and to build independent review into precisely the cases where no ground truth exists to catch an error, since that is where cognitive bias research shows the risk concentrates (Dror et al., 2006; Kassin et al., 2013).

21. A Proposed Forensic Novelty Response Framework

The following framework is offered as a proposed conceptual structure synthesised from the literature reviewed above, not as an adopted international standard.

D

Detect. Identify that evidence may fall outside validated expectations — a rejection-style signal in the spirit of open-set recognition (Geng et al., 2021).

V

Verify. Confirm the observation is technically real before any interpretive resources are spent on it.

C

Characterise. Describe what can be scientifically stated about the evidence without premature classification.

C

Compare. Evaluate known alternatives and actively expanded reference sources.

C

Challenge. Actively test competing, mundane explanations rather than defending the novelty hypothesis.

R

Replicate. Seek independent confirmation wherever independent confirmation is logistically possible.

C/A

Classify or Abstain. Reach a scientifically supported classification, or explicitly document that current knowledge is insufficient to support one.

22. A Forensic Novelty Confidence Gradient

A second proposed instrument represents novelty assessment as a gradient rather than a binary switch, with confidence required to increase, not merely accumulate, at each transition:

Observed anomaly → Technically verified anomaly → Known artefacts excluded → Known explanations evaluated → Independent confirmation obtained → Evidence consistent with novelty → Validated classification of a novel phenomenon

The critical design feature is that movement between adjacent stages should demand progressively stronger evidence, not merely additional time elapsed since the initial observation — a distinction that matters because institutional and reputational pressure tends to push in the opposite direction, treating the passage of time and repeated internal discussion as a substitute for the additional external evidence the gradient actually requires.

23. Cross-Domain Comparison Table

Forensic Domain	Nature of Novel Evidence	Reference Data Challenge	Ground Truth Challenge	Validation Limitation	Risk of False Novelty
Forensic Toxicology	Structurally altered psychoactive analogues	Certified standards lag street emergence (Pasin et al., 2017)	Metabolite pathways often unstudied for a new compound	Library-match methods cannot find unlisted compounds	Moderate — matrix interferences can mimic new peaks
Digital Forensics	Unfamiliar system/process behaviour	"Normal" baseline shifts with every update	Rarely an independently verified "true" system state	Signature-based tools miss unseen attack patterns	High — legitimate updates mimic anomalies
Cybercrime / Zero-Day	Previously uncatalogued exploit behaviour	No signature exists by definition	Attribution often circumstantial	Behavioural baselines require continuous retraining	High — benign anomalies trigger false alarms
Synthetic Media	Manipulation from an unseen generative architecture	Training corpora fixed at a point in time	Provenance ≠ authenticity (Coalition for Content Provenance and Authenticity, 2024)	Cross-manipulation accuracy drops sharply (Villari et al., 2025)	Moderate — compression artefacts can mimic manipulation traces
Trace Evidence	Rare materials, atypical transfer/degradation	Sparse experimental literature	Environmental history usually unrecoverable	Few discipline-specific black-box studies exist	Moderate — contamination easily mistaken for signal
AI-Assisted Forensics	Model output beyond training distribution	Benchmark datasets age quickly	Evaluator itself may share the model's blind spot	Closed-set evaluation overstates real-world reliability	High — confident wrong answers look like discoveries

24. Major Research Gaps

1. Discipline-specific open-set validation protocols for pattern-comparison forensic sciences (currently borrowed, if at all, from computer-vision literature).
2. Publicly available unknown-class benchmark datasets built specifically for forensic evidence types, rather than adapted from unrelated computer-vision benchmarks.
3. Standardised methods for reporting ground-truth uncertainty alongside forensic conclusions, rather than as an unstated caveat.
4. Formal abstention standards specifying when a forensic report should state "insufficient basis for classification" rather than a forced categorical conclusion.
5. Continuous or rolling validation models for methods applied to rapidly evolving evidence classes (synthetic media, novel psychoactive substances).
6. Cross-domain novelty-detection research connecting toxicology, digital forensics, and pattern evidence under a shared methodological vocabulary.
7. Empirical study of how often forensic "novel finding" claims survive the kind of structured artefact-exclusion process proposed in Section 15.
8. Replication studies specifically testing whether black-box error rates generalise to evidence populations that differ from the original study's sampling frame.
9. Statistical methods for propagating growing uncertainty as evidence departs from a validated domain, beyond the non-response modelling already demonstrated by Khan and Carriquiry (2023).
10. Independent, non-vendor-funded testing of synthetic-media detectors against manipulation techniques released after the detector's training cutoff.
11. Security and reliability analysis of content-provenance standards under adversarial removal or falsification, building on Golaszewski et al. (2026).
12. Forensic-specific application and testing of the philosophical "unconceived alternatives" literature (Stanford, 2006) to structured casework reasoning.
13. Cognitive-bias research specifically targeting the moment of novelty discovery, distinct from the better-studied moment of routine comparison.
14. Comparative international study of reference-database representativeness gaps, extending existing population-genetics work on DNA database bias to under-sampled regional and demographic groups.
15. Development of forensic-appropriate "catch-all hypothesis" statistical conventions for Bayesian casework reasoning.
16. Laboratory accreditation standards that explicitly address method-scope boundaries and novelty-response procedures, rather than treating them as implicit.

25. Future Research Agenda

Priority 1 — Open-Set Validation Frameworks

Rationale: Forensic validation science has not yet adopted the open-set vocabulary already standard in machine learning. Current limitation: Validation studies remain closed-set by design. Methodological challenge: Defining a forensic-appropriate "rejection" criterion for human examiners, not only algorithms. Expected value: Reports that distinguish "not this known class" from "no class known."

Priority 2 — Unknown-Class Benchmark Datasets

Rationale: Progress requires shared, discipline-specific test data. Current limitation: No forensic equivalent of standard open-set computer-vision benchmarks exists. Methodological challenge: Constructing such datasets without exposing sensitive casework detail. Expected value: Comparable, reproducible novelty-detection performance figures across laboratories.

Priority 3 — Ground Truth Uncertainty Reporting

Rationale: Ground truth quality varies by evidence type and is rarely disclosed. Current limitation: No standard vocabulary distinguishes experimental, simulated, and corroborated ground truth in casework reporting. Methodological challenge: Building this into existing report templates without overwhelming non-scientist readers. Expected value: More honest courtroom communication of evidentiary strength.

Priority 4 — Forensic Model Abstention Standards

Rationale: Abstention currently has no formal professional or legal standing in most forensic disciplines. Current limitation: Institutional and courtroom pressure favours conclusive answers. Methodological challenge: Distinguishing legitimate abstention from under-performance. Expected value: Reduced pressure toward forced, unsupported categorical conclusions.

Priority 5 — Continuous Validation for Evolving Evidence

Rationale: Synthetic media and novel psychoactive substances evolve faster than periodic validation cycles. Current limitation: Validation studies are typically point-in-time. Methodological challenge: Funding and resourcing rolling validation rather than one-off studies. Expected value: Validity claims that remain accurate as the evidence landscape shifts.

Priority 6 — Cross-Domain Novelty Detection Research

Rationale: Toxicology, digital forensics, and pattern evidence currently address novelty independently. Current limitation: No shared methodological forum exists. Methodological challenge: Building cross-disciplinary research infrastructure within forensic science's traditionally siloed structure. Expected value: Faster transfer of novelty-handling methods across domains.

Priority 7 — Scientific Standards for Claims of Forensic Novelty

Rationale: No accepted standard currently governs when a laboratory may responsibly claim to have found something new. Current limitation: Novelty claims are currently evaluated case-by-case, informally. Methodological challenge: Building a standard specific enough to be useful without becoming a checklist substitute for judgement. Expected value: Reduced risk of premature or overstated novelty claims entering casework or the literature.

Suggested Infographic Concept

From Anomaly to Novel Evidence: The Scientific Decision Path

Conclusion

How should science handle evidence from events that have never been seen before? The review's answer is not a procedure but a discipline of restraint. Responsible forensic science resists the pull toward premature classification; it treats an absent reference match as a statement about the reference collection before it treats it as a statement about the universe; it recognises that a validated error rate is a property of the population that produced it, not a portable constant; it tests competing, mundane explanations with the same energy it applies to the exciting one; and it permits — as a mark of rigor rather than deficiency — the conclusion that current methods cannot yet justify a conclusion at all.

The deepest risk unfamiliar evidence poses to forensic science may not be that the evidence goes unrecognised. Failure to recognise something new is, at least, a visible and correctable gap. The more dangerous failure is quieter: recognising the unfamiliar too quickly, by pressing it into a category built for something else, so that the record shows a confident answer where an honest one would have shown a question still open. A science that can say, clearly and on the record, "this does not yet fit anything we know how to test for," has not failed. It has done the one thing a science built entirely from what is already known can never do on its own — mark, precisely, the edge of what it currently knows.

References

Balasubramanian, L., Kruber, F., Botsch, M., & Deng, K. (2021). Open-set recognition based on the combination of deep learning and ensemble method for detecting unknown traffic scenarios. arXiv. https://arxiv.org/abs/2105.07635

Budding Forensic Expert. (2026). The most difficult problem in forensic validation: How do you validate a method when the true answer is unknown? https://www.buddingforensicexpert.in/2026/09/forensic-validation-unknown-truth.html

Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), Article 15. https://doi.org/10.1145/1541880.1541882

Chen, G., Peng, P., Wang, X., & Tian, Y. (2021). Adversarial reciprocal points learning for open set recognition. arXiv. https://arxiv.org/abs/2103.00953

Coalition for Content Provenance and Authenticity. (2024). C2PA FAQ. https://c2pa.org/faqs/

Content Authenticity Initiative. (n.d.). How it works. https://contentauthenticity.org/how-it-works

Cui, P., & Wang, J. (2022). Out-of-distribution (OOD) detection based on deep learning: A review. Electronics, 11(21), 3500. https://doi.org/10.3390/electronics11213500

Dror, I. E., Charlton, D., & Péron, A. E. (2006). Contextual information renders experts vulnerable to making erroneous identifications. Forensic Science International, 156(1), 74–78. https://doi.org/10.1016/j.forsciint.2005.10.017

Fazl-Ersi, E., et al. (2022). A comprehensive review of trends, applications and challenges in out-of-distribution detection. arXiv. https://arxiv.org/abs/2209.12935

Geng, C., Huang, S.-J., & Chen, S. (2021). Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10), 3614–3631. https://doi.org/10.1109/TPAMI.2020.2981604

Golaszewski, E., et al. (2026). Verifying provenance of digital media: Why the C2PA specifications fall short. arXiv. https://arxiv.org/abs/2604.24890

International Association for Identification, Firearm and Toolmark Committee. (2016). Response to the PCAST report on forensic science. https://theiai.org/docs/20161214_FATM_Response_to_PCAST.pdf

INTERPOL. (2024). Unmasking the threat of synthetic media for law enforcement beyond illusions. https://www.interpol.int/content/download/21179/file/BEYOND%20ILLUSIONS_Report_2024.pdf

Kassin, S. M., Dror, I. E., & Kukucka, J. (2013). The forensic confirmation bias: Problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition, 2(1), 42–52. https://doi.org/10.1016/j.jarmac.2013.01.001

Khan, K., & Carriquiry, A. (2023). Hierarchical Bayesian non-response models for error rates in forensic black-box studies. Philosophical Transactions of the Royal Society A, 381(2247). https://doi.org/10.1098/rsta.2022.0157

Lu, S., Wang, Y., Sheng, L., He, L., Zheng, A., & Liang, J. (2024). Out-of-distribution detection: A task-oriented survey of recent advances. arXiv. https://arxiv.org/abs/2409.11884

Markou, M., & Singh, S. (2003a). Novelty detection: A review—Part 1: Statistical approaches. Signal Processing, 83(12), 2481–2497. https://doi.org/10.1016/j.sigpro.2003.07.018

Markou, M., & Singh, S. (2003b). Novelty detection: A review—Part 2: Neural network based approaches. Signal Processing, 83(12), 2499–2521. https://doi.org/10.1016/j.sigpro.2003.07.019

Medianama. (2026, February). Experts at India AI Impact Summit say no AI model can reliably detect deepfakes yet. https://www.medianama.com/2026/02/223-ai-enabled-cybercrime-india-deepfake-threat/

Monson, K. L., Smith, E. D., & Peters, E. M. (2022). Accuracy of comparison decisions by forensic firearms examiners. Journal of Forensic Sciences, 67(6), 2265–2278. https://doi.org/10.1111/1556-4029.15152

National Institute of Justice. (2021). Challenges in identifying novel psychoactive substances and a stronger path forward. U.S. Department of Justice. https://nij.ojp.gov/topics/articles/challenges-identifying-novel-psychoactive-substances-and-stronger-path-forward

National Research Council. (2009). Strengthening forensic science in the United States: A path forward. National Academies Press.

Park, H., Jeong, E., & Teoh, A. B. J. (2022). Understanding open-set recognition by Jacobian norm and inter-class separation. arXiv. https://arxiv.org/abs/2209.11436

Pasin, D., Cawley, A., Bidny, S., & Fu, S. (2017). Current applications of high-resolution mass spectrometry for the analysis of new psychoactive substances: A critical review. Analytical and Bioanalytical Chemistry, 409(25), 5821–5836. https://doi.org/10.1007/s00216-017-0441-4

Pimentel, M. A., Clifton, D. A., Clifton, L., & Tarassenko, L. (2014). A review of novelty detection. Signal Processing, 99, 215–249. https://doi.org/10.1016/j.sigpro.2013.12.026

President's Council of Advisors on Science and Technology. (2016). Forensic science in criminal courts: Ensuring scientific validity of feature-comparison methods. Executive Office of the President.

Ruhmkorff, S. (2011). Some difficulties for the problem of unconceived alternatives. Philosophy of Science, 78(5), 875–886. https://doi.org/10.1086/662566

Soni, N. (2024). Deepfake detection and multimedia forensics: Investigating synthetic media, image forgery, and video manipulation in cybercrime cases. Department of Forensic Science, Lovely Professional University.

Stanford, P. K. (2006). Exceeding our grasp: Science, history, and the problem of unconceived alternatives. Oxford University Press.

Stanford Encyclopedia of Philosophy. (2023). Underdetermination of scientific theory. https://plato.stanford.edu/entries/scientific-underdetermination/

Taleb, N. N. (2007). The Black Swan: The impact of the highly improbable. Random House.

Ulery, B. T., Hicklin, R. A., Buscaglia, J., & Roberts, M. A. (2011). Accuracy and reliability of forensic latent fingerprint decisions. Proceedings of the National Academy of Sciences, 108(19), 7733–7738. https://doi.org/10.1073/pnas.1018707108

Villari, M., et al. (2025). Deepfake media forensics: Status and future challenges. PMC. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11943306/

Wikipedia contributors. (2026). Black swan theory. Wikipedia. https://en.wikipedia.org/wiki/Black_swan_theory