Skip to content

fix(auth): report exhausted Codex quota as a usage limit Codex clients stop on - #17

Open
ntindle wants to merge 4 commits into
mainfrom
fix/codex-usage-limit-fork
Open

ntindle wants to merge 4 commits into
mainfrom
fix/codex-usage-limit-fork

Conversation

@ntindle

@ntindle ntindle commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Why

Codex threads in T3 never show Limited when both ChatGPT logins are out of quota, and sit "running" for hours or days instead. While every Codex credential cools down on usage_limit_reached, the Gateway answers upstream's model_cooldown 429 with Retry-After set to the quota window. Codex 0.166 recognises a usage limit only by error.type, so it retries:

  • over WebSocket the Gateway closes silently five times;
  • over HTTPS Codex sleeps until Retry-After.

On rust-builder one turn waited 38½ hours, from 2026-10-31 02:10Z to 2026-11-01 16:49Z. Investigation: factory-notes incidents/2026-11-01-codex-quota-stops-not-limited.md.

This is not caused by a fork patch. #10's pass-through matches only Meta's refusal shape.

What

This is the change proposed upstream as router-for-me#7184, cherry-picked onto main. The selector.go conflict with #10's refusal field was resolved by keeping both. While a Codex pool's first reset is more than five minutes away:

Fork-only commits:

Verification

  • Live, on rust-builder: Codex CLI 0.166.0 ran through a local build against a mock upstream returning OpenAI's usage_limit_reached.
    • Before: Reconnecting... 2/5–5/5 (Connection reset without closing handshake), then Reconnecting... 1/5 (exceeded retry limit, last status: 429) and a Retry-After: 10640 sleep, still waiting at 75 s.
    • After: You've hit your usage limit. Try again at 7:02 PM. and turn.failed within a second, over WebSocket and HTTPS.
  • bash .github/scripts/fork-check.sh: exit 0 (blocking checks and the advisory upstream tests).
  • go test -race on the new tests passes. Each production change was reverted in turn, and its tests failed.

Applying

Merging publishes :latest. Applying it recreates the Gateway container, which cuts in-flight Codex and Claude requests, T3 threads and Rocky Brain jobs. Apply when idle. No config change is needed.

T3 needs no update for this: current T3 already shows Codex's usageLimitExceeded as Limited, and the headers give it the reset time for "Resume at reset". T3's own fix for the current behaviour is being upstreamed separately.

🤖 Generated with Claude Code

ntindle and others added 4 commits November 1, 2026 16:07
…s stop on

When every credential a request may use is cooling down after Codex's 429
usage_limit_reached, the proxy answered model_cooldown with a Retry-After as
long as the quota window. Codex recognises a usage limit only by error.type,
so it treated the refusal as a rate limit: over WebSocket the connection
closed without an event five times, then over HTTPS it slept until
Retry-After, hours or days, before retrying.

Such a pool now gets, while its first reset is more than five minutes away:
- the model_cooldown body plus type usage_limit_reached, resets_at,
  retryable false and limit_window_minutes;
- X-Should-Retry: false and Codex's rate-limit headers for the window that
  reopens first, instead of Retry-After;
- the same refusal as a WebSocket error event instead of a silent close.

The evidence is the usage_limit_reached refusal each credential's cooldown
recorded, which survives a restart. A pool with any other 429, and a short
cooldown, keeps today's retryable response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Documents the change in FORK.md, adds hook checks to fork-check.sh and
TestFork* wrappers so a sync cannot drop it silently while
router-for-me#7184 is open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h wrapping

The Codex executor stores usage_limit_reached under error, at the top
level, under body.error (WebSocket) and response.error (SSE); the evidence
check read only error. A usage limit wrapped in another error now answers
with the cooldown's own body, so error.type survives on HTTP and WebSocket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ntindle

ntindle commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Update:

Verified live on rust-builder with Codex CLI 0.166.0 and an isolated T3 dev server:

Gateway T3 Result
current main Working, "Retrying provider (1/5)", indefinitely
current T3 branch Limited after 5 min (no reset time)
this PR main Limited at once, "Resets 7:56 PM", Resume at reset
this PR T3 branch the same
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant