Independent defensive-security research
Can AI reliably distinguish a new security problem from a duplicate, recurrence, or already-remediated incident?
Defensive Drift is building a reproducible benchmark and evaluation pipeline for security-drift reconciliation across configuration evidence, historical incidents, scanner findings, logs, and remediation records.
Research integrity: no performance claims are published until they are measured against the frozen benchmark and linked to reproducible run artifacts.
Research question
Evidence-grounded defensive triage, not autonomous remediation.
Modern security evidence is fragmented across configuration snapshots, infrastructure-as-code, scanners, logs, deployment history, prior incidents, and operational notes. The project measures whether AI can reconcile those artifacts without hiding genuinely new or unresolved problems inside a false duplicate.
The benchmark requires each method to classify an observation as NEW, DUPLICATE, RECURRENCE, RELATED_BUT_DISTINCT, or INSUFFICIENT_EVIDENCE, while grounding its conclusion in the supplied evidence.
The historical drift corpus is intentionally heterogeneous. The current raw/confirmed-incidents/ stream contains curated ChatGPT drift records, while older and other-model drift evidence may exist elsewhere and may use different templates. Template conformity is not an evidence-validity requirement; normalization happens only in derived private research artifacts.
Frozen measurement contract
Success means learning where AI helps — and where it fails.
Duplicate precision & recall
Measures both unsafe over-merging and alert-fatigue-causing misses.
Novel-issue recall
Measures whether new or materially distinct conditions remain visible for review.
Dangerous false-duplicate rate
Primary security-safety metric for incorrect closure-like classifications.
Evidence grounding
Checks whether cited evidence actually exists and materially supports the conclusion.
Remediation accuracy
Tests whether models correctly distinguish detected, mitigated, remediated, recurred, unresolved, and unknown states.
Cost, latency & consistency
Compares practical operating cost, response time, and repeated-trial stability.
Grant-readiness execution path · rebaselined September 18
Five milestone gates. Current status takes precedence over missed historical target dates.
Research design frozen
Taxonomy, ground-truth rules, metrics, sanitization boundaries, and experiment protocol were frozen before final benchmark/model evaluation.
Freeze benchmark v0.1
The 100-case evidence base is assembled and historical context is materialized. The current scientific gate is human context-sufficiency review, followed by human relationship adjudication and benchmark freeze.
Run conventional baselines
Run normalized lexical, TF-IDF/similarity, and embedding baselines only after benchmark v0.1 is frozen so final comparisons remain reproducible.
Complete AI evaluation
Compare OpenAI and practical open-weight approaches, including repeated high-risk trials, cost, latency, evidence grounding, consistency, and failure analysis.
Grant-ready evidence package
Publish methodology, preliminary measured results, reproducibility artifacts, failure catalog, research brief, and the final grant evidence matrix. October 2 is an internal execution target, not an asserted external grant deadline.
Current status
M2: benchmark recovery and grant rebaseline are active.
M1 is complete. The M2 evidence base remains materialized and integrity-validated for 100 suitable benchmark observations, with 820 unique historical records supporting 1,952 case-to-context relationships. The last verified research artifact still records 0 of 100 human context-sufficiency decisions and no relationship ground-truth labels. Drift-governance automation has advanced separately, but it does not replace the unfinished benchmark work.
Public benefit
Built to be inspected, reproduced, and challenged.
Defensive Drift is intended to release a safe public benchmark subset, schemas, evaluator code, methodology, measured results, representative failure cases, and reproducibility guidance. Private operational records are not automatically public research data; public artifacts must pass explicit sanitization and confidentiality review.