From the course: Building Generative AI Apps to Talk to Your Data

Unlock this course with a free trial

Join today to access over 25,300 courses taught by industry experts.

Automatic processing of new documents

Automatic processing of new documents - Snowflake Tutorial

From the course: Building Generative AI Apps to Talk to Your Data

Start my 1-month free trial Buy for my team

Automatic processing of new documents

“

- Welcome back. So in the last video, we diagnosed a problem. The RAG is using stale data. So how do we fix this? If we go back to when we first talked about Cortex search, you might remember that it can automatically handle refresh and re-indexing as fast as one minute or whatever you set the target lag parameter to be. This means that the Cortex search service itself is as fresh as our data. But what happens when our data gets stale? Well, we need to update it, and we need some way to automate the process from raw PDFs to parsed and chunk data indexed by Cortex search, so we're not doing all of this manually. And that's where tasks and streams come in. (transition whooshing) Anytime data is added to stage, we can create a stream that captures changes to the stage. This stream tells us exactly what has changed. Then when the stream happens, we can use a task to run transformation on the new data, like parsing and chunking. To make this happen, the first stream we'll create will watch…

Contents