From the course: Architecting Big Data Applications: Batch Mode Application Engineering
Join today to access over 25,300 courses taught by industry experts.
Architecture process for data engineering
From the course: Architecting Big Data Applications: Batch Mode Application Engineering
Architecture process for data engineering
- [Instructor] What is the process to follow while architecting applications using batch data engineering? Let's begin with the definition of data engineering. Data engineering is the methodical process of designing and building big data pipelines that acquire, store, process, and analyze data to derive business outcomes. Data engineering provides a discipline in the architecture and design process that helps build efficient and effective pipelines. What does the architecture process look like? The first step is defining the use case. This involves clearly stating the problem to be solved, the expected solution, and the design goals. The goal would cover functional requirements, form and types of outputs, and benchmarks. The next step is to study the requirements thoroughly. This starts with understanding the outputs needed, including their form and type. Then inputs need to be analyzed for their form volume, data…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.