From the course: Apache Spark Essential Training: Big Data Engineering
Join today to access over 24,800 courses taught by industry experts.
Building a website analytics job - Apache Spark Tutorial
From the course: Apache Spark Essential Training: Big Data Engineering
Building a website analytics job
- [Instructor] Let's start building the streaming website analytics job now. The code for this is available in the notebook, code_04_04_Spark_BDE_Build_a_streaming_analytics_job. We first define a few methods. We first have the right to Redis method that updates Redis with the last action stats. Here we connect to the Redis instance running on the local Docker container. Then we increment the duration for the country in a sorted set identified by the key last action stats. We are defining this as a method here. However, in production systems, it's recommended to define it as a class that implements the data stream writer interface of Spark. Then we define another function to write to MariaDB. Here we first connect to the MariaDB database website stats. We then insert the last action and the duration values into the website stats table. Let's go to the main job now. In this job, we define a schema for the visit record received through Kafka. Then we initiate a Spark session with local…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.