You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds a kick-the-tires sanity check (welder-ae-correctness-kick-the-tires.sh,
random sample) and the full campaign (welder-ae-correctness.sh),
matching the existing Table 1/Table 2 section format. Includes
CHACTOS_WORKERS tuning guidance (SSD vs. spinning-disk hardware) and
realistic timing (~1-4 compute-days on SSD, 10+ on spinning disk).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: README.md
+45Lines changed: 45 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,7 @@ The paper claims that Welder enables compositional verification of Kubernetes co
12
12
13
13
-**Table 1: full verification.** Welder's framework, four controllers (VReplicaSet, VDeployment, VStatefulSet, VRabbitMQ), and their composition proof are fully verified by Verus (0 errors), with the code size table.
14
14
-**Table 2: competitive performance.** The verified controllers' `reconcile` and end-to-end reconciliation times are comparable to the unverified reference controllers (end-to-end differences within one standard deviation).
15
+
-**Controller Correctness (Section 5.2).** The verified controllers pass extensive functional and fault-injection testing (functional tests, controller-crash tests including correlated crashes of two interacting controllers, and Pod-crash tests) with 0 oracle violations.
15
16
16
17
Each experiment below is labeled with the claim it evaluates, and each "expected output" is followed by a note on how to read that output as evidence for the claim.
17
18
@@ -136,6 +137,23 @@ If you set up your own machine, replace `~/workdir/acto` with the path to the cl
136
137
137
138
**Expected result:** The absolute numbers depend on the platform, but you should observe that end-to-end differences are negligible.
**Evaluates: Controller Correctness (Section 5.2), sanity check.** This runs a small random sample of functional and fault-injection tests to confirm the correctness-testing pipeline works end-to-end and finds 0 oracle violations, without committing to the full multi-day campaign below.
143
+
144
+
```bash
145
+
cd~/workdir/acto
146
+
source venv-welder/bin/activate # only on your own machine
147
+
bash welder-ae-correctness-kick-the-tires.sh
148
+
```
149
+
150
+
Expected output ends with:
151
+
152
+
```
153
+
Scanned N fault-injection test results across 8 workdirs
154
+
No oracle violations found (matches paper's claim: 0 bugs found)
155
+
```
156
+
139
157
---
140
158
## Full Evaluation Instructions (~2 compute-hours + ~2 human-minutes)
141
159
@@ -171,3 +189,30 @@ Note that running all the workloads takes about 22 machine-hours. If you really
171
189
bash welder-ae-sampled.sh 1
172
190
```
173
191
</details>
192
+
193
+
### Full Controller Correctness campaign (~1-4 compute-days + ~2 human-minutes)
194
+
195
+
**Evaluates: Controller Correctness (Section 5.2), full campaign.** Reproduces the paper's fault-injection campaign (functional tests, controller-crash tests, Pod-crash tests) against the checked-in trial corpus, checking for oracle violations with `chactos`. `chactos` has no sampling flag -- each run processes every trial in its input corpus -- so this takes much longer than the sampled Table 2 workflow above, and the round counts are the closest integer approximation of the paper's reported test counts rather than an exact reproduction. In the `acto` checkout, run:
196
+
197
+
```bash
198
+
export CHACTOS_WORKERS=6 # tune per your hardware, see below
199
+
bash welder-ae-correctness.sh
200
+
cat welder-table-correctness.txt
201
+
```
202
+
203
+
Expected output:
204
+
205
+
```
206
+
Scanned N fault-injection test results across M workdirs
207
+
No oracle violations found (matches paper's claim: 0 bugs found)
208
+
```
209
+
210
+
<details><summary>How long does this take, and how do I tune it?</summary>
211
+
212
+
`chactos` fully tears down and recreates a Kubernetes cluster for every trial, so throughput is disk-bound rather than CPU-bound. `CHACTOS_WORKERS` (default 6) controls how many trials run concurrently:
213
+
214
+
- On SSD-backed machines, 6 concurrent workers complete cleanly; the full campaign takes roughly 1-4 compute-days.
215
+
- On spinning-disk machines, higher worker counts saturate the disk and the underlying Docker daemon fails to tear down containers in time (`could not kill container: ... did not receive an exit event`). If you see this, set `CHACTOS_WORKERS=1` -- this reliably completes but is much slower (on the order of 10+ compute-days for the full campaign).
216
+
217
+
If you just want to confirm the pipeline works without committing to the full runtime, use the kick-the-tires sanity check above instead.
0 commit comments