From the course: Data Pipeline Automation with GitHub Actions Using R and Python

Pulling metadata from API with Python - GitHub Tutorial

From the course: Data Pipeline Automation with GitHub Actions Using R and Python

Start my 1-month free trial Buy for my team

Pulling metadata from API with Python

“

- [Instructor] Let's open VSCode and demonstrate how to send a get request to pull data from the API. Throughout this video and in the next ones, we will use the following Jupyter Notebook to execute the code. In addition, we'll use the EIA_API script with a set of functions to work with the API. Both files are under the course repository under chapter one folder. Before getting started, please make sure you select the virtual environment you are using for this course. If you are using the course image, you should select the LinkedIn_Learning virtual environment with the requirements for this notebook. To be consistent with the previous examples, we will continue to focus on the hourly electricity demand of the Pacific Gas and Electricity Balancing Authority subregion, or, in short, PGAE. The screenshot here provide the series metadata from the dashboard that we used in the previous examples. Let's start by importing the required libraries and functions. As mentioned earlier, the EIA_API is a local script with a set of Python functions that we will use throughout this course to query data from the EIA_API. We will import the EIA_API and set it as API. Let's go ahead and execute it. Let's continue and load the rest of the Python libraries. We are going to use the following three libraries, OS to import the API key from the environment variable, Datatime to reformat date and time objects, and Plotly to visualize the data that we're going to pull from the API. Let's go ahead and also execute this core chunk. Now we can start working with the API. Let's start by pulling the metadata of the PGAE series using the EIA metadata function. As you can see, there are two arguments for this function, and it's straightforward. The API key, which we need to provide to get access to the API, and the API path, which represents the route that we want to pull metadata from. We will use the getenv functions from the OS library to pull the environment variables with our access key, EIA_API_KEY. If you didn't set this variable, you can also pass your API key over here, but it's not recommended. Next, we are going to set this variable, api_meta_path, to set the route that we want to pull the metadata. In this case, we want to pull subregion electricity data, and this is the same route as we used before. Let's go ahead and execute this and assign it to the function. You can see the GET request returns the metadata from the API, and we can go ahead and print it. This includes the series description, the frequency of the series, the starting and ending point over here, and other information. In the next video, we'll continue to work with this notebook and use the metadata output to set the GET request.

Contents