System Characteristics: Deterministic Replay, Statistical Verdicts, Evidence Chain, CI-Native

The four system characteristics of the Cyclone deterministic scenario testing kernel: bitwise-reproducible replay per seed, Wilson+SPRT statistical verdicts, a four-piece evidence chain, and CI-native JUnit/Bazel integration.

§1 · PROBLEMWhy nobody trusts test results

The usual fate of a scenario test report: it gets produced, nobody re-checks it, and nobody owns it when things go wrong. Three root causes — irreproducibility (rerun the same scenario, get a different result; a FAIL can’t be regression-verified), verdicts by gut feel (“27 out of 30 passes” — does that pass? nobody can win the argument), and no evidence chain (a pass in the report can’t be traced back to the case version and raw data).

§2 · CHARACTERISTICSFour characteristics, four remedies

§2.1 Deterministic replay Supported

Virtual-time replay with content addressing: identical seeds produce bitwise-identical evidence logs. The clock is virtual, the scheduler deterministic, and replay outputs can be compared by SHA-256 — “works on my machine” ceases to exist.

§2.2 Statistical verdicts Supported

Wilson score lower bounds + SPRT sequential testing — statistical significance instead of hand-tuned thresholds. A verdict reads “0.99 (95% CI lower bound, n=300, case hash recomputable)”, and SPRT stops early once the evidence is conclusive, wasting zero simulation runs.

§2.3 Evidence chain Supported

The four-piece set: case hash + JSONL SHA-256 digest + failure snapshot + JUnit XML — every CI record traces back to the case version and raw data, recomputable by a third party.

§2.4 CI-native

JUnit XML out of the box Supported; rules_cyclone makes scenario tests first-class bazel test citizens Prototype; BES-ready Prototype.

§2.5 Architecture: one kernel, two notions of time

Offline mode trades the real clock for reproducibility; online mode trades back for realism — same scenario YAML, just flip the clock mode:

Capture once, replay forever: record the bus once on the vehicle, replay offline indefinitely. A layered single engine — C++ kernel + Python scenario layer; the process boundary is the YAML schema + evidence-pack format. Details in the dual-mode clock article.

§2.6 The verdict pipeline

Case YAML → dataset replay → fault injection → SUT → verdicts → evidence report

261 automated tests green · JUnit/Jenkins ready · UDS diagnostics demo Prototype · Bazel ruleset rules_cyclone Prototype.

§2.7 Integration interfaces

For Bazel users Prototype:

# MODULE.bazel
bazel_dep(name = "rules_cyclone", version = "0.1.0")

# BUILD.bazel
cyclone_test(
    name = "aeb_regression",
    scenario = "scenarios/aeb.yaml",
)
# bazel test //... — same graph, same cache, same dashboard as unit tests

Any other CI:

# single-file binary, download and run (planned)
$ cyclone run cases/aeb.yaml --out out/

# JUnit XML → consumed directly by Jenkins / GitLab CI (supported)
out/aeb_report.xml
out/aeb_manifest.json   # evidence index

§3 · DEEP DIVESLearn more

• Deterministic fault injection: five fault primitives, tick-level reproduction →
• ADAS scenario regression testing: from scenario pack to CI gate →
• Auditable test evidence chain: every verdict traceable →

Methodology background:

• Wilson lower-bound verdicts: an honest ruler for pass rates →
• SPRT and LLR: judge early when the evidence is in →
• The dual-mode clock: reproducible offline, realistic online →

§4 · HONEST BOUNDARIESHonest boundaries

• Determinism covers the platform-managed path only. Nondeterminism inside the SUT itself (threading races, unseeded randomness) must be governed first — the platform amplifies your governance, it doesn’t replace it.
• Bazel integration and BES are prototypes. rules_cyclone is demonstrable but not production-SLA; start with plain JUnit XML during a POC.
• All sample figures are demo data, not measured on a real SUT.

Scope and non-scope