How to handle rate limits in GitHub API scripts?

[kjk-hard]

Select Topic Area

Question

Body

Hello,

I’m writing a Python script to gather data from multiple GitHub repos.
I noticed that after a while, requests start failing with rate limit errors.

How do you usually handle this in your scripts? Backoff? Token rotation? Or something else?

Any best practices would be appreciated.

[Eric-Lebedenko]

Hi, this is a common issue when interacting with GitHub API.

Some tips from my experience:

1. Check rate limit headers
   Each response includes X-RateLimit-Remaining and X-RateLimit-Reset. You can use these to pause requests until you have quota again.
2. Exponential backoff
   If a request fails with 429, wait a few seconds and retry. Increase the wait time exponentially with each subsequent failure.
3. Token rotation
   If you have multiple personal access tokens, you can rotate them when you hit the limit on one. Just make sure not to violate GitHub terms.
4. Caching responses
   If your script pulls the same data repeatedly, caching can save a lot of requests.

Overall, combining backoff + token rotation usually works well for long-running scripts.

Hope that helps, happy coding!

[kjk-hard]

Hmmm, thx!

[Aghorix108]

I’ve run into this a lot when scraping or scanning multiple repos.

First thing I do is always use a GitHub token, even for public repositories. Without a token, the rate limit is very small and you’ll hit it quickly.

In the script, I read the rate-limit headers that GitHub sends back. When the remaining requests reach zero, I just pause the script and wait until the reset time, instead of retrying and failing.

I also add a small delay between requests so the script doesn’t hit the limit too aggressively.

Token rotation can work if you’re running at a very large scale, but for most scripts a single token, some waiting, and basic throttling is enough.

[BMmubeez]

To avoid hitting these 429s and 403s consistently, you really need to treat your scraper like a 'polite' client rather than a flood-gate. Here are 3 steps that made a huge difference for my automation scripts:
Implement Exponential Backoff: Don't just retry immediately after a 429. If the server sends a Retry-After header, respect it. If not, wait 2^n seconds (where n is the number of attempts) before trying again to let the server cool down.
Session Persistence & Headers: Ensure you are using a consistent User-Agent that reflects a modern browser, and maintain session cookies if you are hitting gated endpoints. Sending generic library headers (like 'python-requests/2.x') is a massive red flag for anti-bot systems.
Proxy Rotation: If you are running high-volume requests, your IP reputation will naturally decay. Rotating through high-quality residential proxies, such as Magnetic Proxy, can help maintain access by distributing your request footprint across diverse, legitimate-looking residential network exit nodes.
Hope this helps keep your scripts stable!