Summary
The v1.1 verifier for obsidian-linter-auto-table-of-contents couples all 41 feature-test identifiers to the reference solution's English rule display name, even though the task instruction does not require that name.
A candidate used the reasonable display name Auto TOC. The hidden feature suite executed all 41 tests and reported 36 passing / 5 failing, with test names beginning Auto TOC .... The grader expected names beginning Auto Table of Contents ..., treated all 41 expected tests as missing from the structured test report, and assigned F2P 0/41.
This makes the score depend on reproducing an unspecified UI string rather than the requested behavior.
Version
- DeepSWE commit:
e016041a6ccf8da29906afc9a3f5a8df940a1f78
- Task:
obsidian-linter-auto-table-of-contents
- Task image:
public.ecr.aws/d3j8x8q7/swe-bench-202605:kh74j15mp1vrxx737y8b47tc8h832a37-v1.1@sha256:56d3ffcde2ca373ffa0107f830b243d66191020f7021806acc8d609148518092
Evidence
The public task instruction says to export AutoToc from src/rules/auto-toc.ts, but does not prescribe an English locale/display name.
The reference solution adds:
'auto-toc': {
'name': 'Auto Table of Contents',tests/config.json then uses that display string in every exact feature-test identifier, for example:
"Auto Table of Contents No TOC markers present — text is unchanged"
The verifier reconciles the generated test-report names against those configured strings. With the candidate's name: 'Auto TOC', the report contained names such as:
Auto TOC No TOC markers present — text is unchanged
Observed test result from one run:
5 failed, 36 passed, 41 total
The subsequent grade marked all configured feature-test IDs missing and reported:
The five genuine behavioral failures are separate; the name mismatch alone changed the reported result from 36/41 to 0/41.
Suggested fix
Use stable test identifiers that do not incorporate the localized rule display name. Alternatively, normalize or map the suite prefix before reconciliation. The verifier should score the 41 executed cases by stable identity while preserving their human-readable names for reporting.
Summary
The v1.1 verifier for
obsidian-linter-auto-table-of-contentscouples all 41 feature-test identifiers to the reference solution's English rule display name, even though the task instruction does not require that name.A candidate used the reasonable display name
Auto TOC. The hidden feature suite executed all 41 tests and reported 36 passing / 5 failing, with test names beginningAuto TOC .... The grader expected names beginningAuto Table of Contents ..., treated all 41 expected tests as missing from the structured test report, and assigned F2P 0/41.This makes the score depend on reproducing an unspecified UI string rather than the requested behavior.
Version
e016041a6ccf8da29906afc9a3f5a8df940a1f78obsidian-linter-auto-table-of-contentspublic.ecr.aws/d3j8x8q7/swe-bench-202605:kh74j15mp1vrxx737y8b47tc8h832a37-v1.1@sha256:56d3ffcde2ca373ffa0107f830b243d66191020f7021806acc8d609148518092Evidence
The public task instruction says to export
AutoTocfromsrc/rules/auto-toc.ts, but does not prescribe an English locale/display name.The reference solution adds:
tests/config.jsonthen uses that display string in every exact feature-test identifier, for example:"Auto Table of Contents No TOC markers present — text is unchanged"The verifier reconciles the generated test-report names against those configured strings. With the candidate's
name: 'Auto TOC', the report contained names such as:Observed test result from one run:
The subsequent grade marked all configured feature-test IDs missing and reported:
The five genuine behavioral failures are separate; the name mismatch alone changed the reported result from 36/41 to 0/41.
Suggested fix
Use stable test identifiers that do not incorporate the localized rule display name. Alternatively, normalize or map the suite prefix before reconciliation. The verifier should score the 41 executed cases by stable identity while preserving their human-readable names for reporting.