From the course: Snowflake SnowPro Core Cert Prep

Unlock this course with a free trial

Join today to access over 25,300 courses taught by industry experts.

Best practices for data loading

Best practices for data loading - Snowflake Tutorial

From the course: Snowflake SnowPro Core Cert Prep

Start my 1-month free trial Buy for my team

Best practices for data loading

“

Now, Snowflake recommends that you aim to load data files of roughly 100-250 MB or larger in a compressed format, and stage files at a minimum of one-minute intervals for maximum efficiency using Snowpipe. Splitting larger files into these smaller files will distribute the load among the available compute resources within an active warehouse, and splitting larger files into smaller ones also allows the data loading process to scale linearly. So unless you plan on loading hundreds or thousands of files, a small, medium, or large virtual waves could well be sufficient. And you may want to do the inverse of that. So if you've got lots of smaller files, you may want to aggregate them up to minimize the process and overhead for each file. Now, both internal and external stage references can include a path to where your stage files live. When staging data sets as part of regular data load process, Snowflake recommends that you partition the data into logical file paths and note identifying…

Contents