From the course: Data Pipeline Automation with GitHub Actions Using R and Python

Unlock the full course today

Join today to access over 24,800 courses taught by industry experts.

Handling a large data request with Python

Handling a large data request with Python - GitHub Tutorial

From the course: Data Pipeline Automation with GitHub Actions Using R and Python

Start my 1-month free trial Buy for my team

Handling a large data request with Python

“

- [Instructor] In the previous video, we saw the limitation of the eia_get function to pull large dataset due to the API row limitation that got request. In this video, we'll see how to handle a large data request from the API using the eia_backfill function. The eia_backfill function splits a large data request into a sequence of small request. Send those request to the API using the eia_get function on the backend. It then appends the outputs into a single table. The function uses the same arguments as the eia_get function. Let's re-pull the series this time using the backfill function. We'll set the start argument to July 1st, 2018, and the end argument to February 24th, 2024. So we're going to again use the datetime to reformat the object into time format. So the start is going to be 2018 and it's going to be July 1st. And the first data point is at eight o'clock in the morning. And for the end, we are going to set it at 2024 and send it as before, February 24. And let's set it to…

Contents