Skip to content

fix: repair classic xref tables whose entries end with a bare LF - #474

Open
xenokratos wants to merge 2 commits into
firecrawl:mainfrom
xenokratos:fix/bare-lf-xref-entries
Open

fix: repair classic xref tables whose entries end with a bare LF#474
xenokratos wants to merge 2 commits into
firecrawl:mainfrom
xenokratos:fix/bare-lf-xref-entries

Conversation

@xenokratos

@xenokratos xenokratos commented Aug 30, 2026

Copy link
Copy Markdown

Fixes #473.

Adds a repair candidate alongside #230's startxref-pointer recovery: when the last classic xref table's entries end with a single EOL byte (19-byte stride) instead of the spec's two (exactly 20), rebuild the table at its original position with spec SP LF entries and leave every other byte untouched. The startxref pointer still lands on the table start, and object offsets point backwards at content that has not moved.

Returns None when every entry already has the spec stride, so well-formed files never produce a useless repair candidate — and any parse ambiguity inside the table (non-digit offsets, unknown entry kind, missing EOL) also bails to None rather than guessing, since this crate processes untrusted input.

The fixture is fully synthetic (generated, one page, one text line). The bare-LF variant reproduced the failure verbatim before this change; on a real 65MB / 29-page production document with this malformation, extraction goes from invalid file trailer to full text in 0.13s. 1199 existing tests pass unchanged.


Summary by cubic

Fixes #473. Classic xref entries whose EOL isn't the spec's two bytes — a bare LF (19-byte stride) or SP CR LF (21), including tables mixing both — previously failed with invalid file trailer; they now parse because the last classic xref table is rebuilt in place with spec 20-byte SP LF entries. Well-formed files are unaffected, and ambiguous tables are left alone rather than guessed at.

  • Adds a repair candidate alongside fix: recover from a corrupted startxref pointer #230's startxref-pointer recovery: rebuild the table at its original position and leave every other byte untouched; three-byte EOLs are matched first so mixed tables don't leave the next entry on a stray LF.
  • Returns None when every entry already has the spec stride, so well-formed files never produce a useless repair candidate; parse ambiguity inside the table also bails to None.
  • Fixtures are synthetic; the bare-LF and mixed-EOL variants reproduced the failure before this change, and a real 65MB / 29-page production document with this malformation now extracts fully.
  • 1199 existing tests pass unchanged.

Written for commit 69827c8. Summary will update on new commits.

Review in cubic

Some PDF writers terminate classic cross-reference entries with a single
byte (a bare LF) instead of the two-byte end-of-line the spec requires,
making each entry 19 bytes instead of exactly 20. The offsets inside such
a table are all correct - only the stride is wrong - but the file fails
every entry point with `invalid file trailer`, while pypdf, pdfium and
pdfjs all read it. Hit in the wild on a 65MB, 29-page production document.

Adds a repair candidate alongside the existing startxref-pointer recovery
(firecrawl#230): rebuild the table at its original position with spec 20-byte
entries and leave every other byte untouched. The startxref pointer still
lands on the table start, and object offsets point backwards at content
that has not moved. Returns None when every entry already has the spec
stride, so well-formed files never produce a useless candidate.

Fixture is synthetic: two byte-identical PDFs except the xref entry EOL
(SP LF vs bare LF); the bare-LF variant reproduced the failure verbatim
before this change. 1199 existing tests pass unchanged.
@xenokratos
xenokratos force-pushed the fix/bare-lf-xref-entries branch from 762e02d to 26de46b Compare August 30, 2026 23:25

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/integration_tests.rs">

<violation number="1" location="tests/integration_tests.rs:4403">
P3: The `/// Regression for #228` doc comment (about a corrupted `startxref` pointer) now sits directly above the new `test_process_pdf_recovers_bare_lf_xref_entries` test, which is about a different issue (#473 / bare-LF xref stride). The new test was inserted between that comment and the actual #228 test, `test_process_pdf_recovers_corrupted_startxref_pointer`, so the comment misattributes the regression it describes. Move the new test below the #228 block (or move the comment) so each test keeps its own doc.</violation>
</file>

Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic

Comment thread tests/integration_tests.rs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/lib.rs">

<violation number="1" location="src/lib.rs:4030">
P2: When a classic table mixes bare-LF entries with normal `SP CR LF` entries, this arm consumes only `SP CR` and the next iteration starts on `LF`, so normalization bails out and the PDF remains unrepairable. Consume three bytes for `SP CR LF` before the two-byte `SP CR`/`SP LF` cases.</violation>
</file>

Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/lib.rs Outdated
The two-byte arm matched `SP CR` inside a `SP CR LF` terminator and left the
next entry starting on the stray `LF`, so a table mixing bare-LF entries with
`SP CR LF` ones bailed out and the file stayed unrepairable. Matching the
three-byte form first fixes that, and treating any non-two-byte EOL as
malformed also covers a table that is uniformly `SP CR LF` (21-byte stride),
which failed the same way.

Fixture mixes both strides in one table. Moves the bare-LF test below the firecrawl#228
block so that issue's doc comment stays attached to its own test.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Shadow auto-approve: would auto-approve. Adds a repair candidate that rebuilds the last classic xref table in place when its EOL bytes don't match the spec, fixing 'invalid file trailer' for bare-LF and mixed-EOL xref tables; confined to malformed input and covered by new tests.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant