From the course: Build a No-Code ETL Pipeline with Google BigQuery
Unlock this course with a free trial
Join today to access over 24,800 courses taught by industry experts.
Schedule transfers with Transfer Service - BigQuery Tutorial
From the course: Build a No-Code ETL Pipeline with Google BigQuery
Schedule transfers with Transfer Service
- [Instructor] We've created our data transfer and we've proven that it works. The next step is to automate it, because we don't want to remember to run it manually every day. So I am in the Data transfers page and I see my transfer over here, and I will select it. And in here I will go to Edit. Now I will scroll down to Schedule options. So instead of On-demand, I will select a repeat frequency. So how often do I want to run this transfer? Now, because I get a new version of my data every day, I want this transfer to run every day. So I will select Days. And now I need to pick the time for my transfer in UTC time zone. So what matters here in picking the time is that I need to pick a time that is after the data comes into Google Cloud storage, so that every day I can get the fresh data. Now, in our case, we are putting the data manually in Google Cloud Storage, but if it was an automated process doing this, now would be a good time to see when that automated process is running and…
Contents
-
-
-
-
(Locked)
How data load will work1m 40s
-
(Locked)
Introduction to data4m 44s
-
(Locked)
What is Google Cloud Storage?2m 42s
-
(Locked)
Put data in Google Cloud Storage3m 35s
-
(Locked)
Create table in BigQuery4m 35s
-
(Locked)
Introduction to BigQuery Data Transfer Service1m 43s
-
How we will manage data6m 4s
-
(Locked)
Use Transfer Service to ingest data6m 40s
-
(Locked)
Schedule transfers with Transfer Service3m 19s
-
(Locked)
Identify data transfer issues6m 18s
-
(Locked)
Common issues with data transfer5m 21s
-
(Locked)
-
-
-