Skip to main content

IBM: Apache Spark for Data Engineering and Machine Learning

This short course introduces you to the fundamentals of Data Engineering and Machine Learning with Apache Spark, including Spark Structured Streaming, ETL for Machine Learning (ML) Pipelines, and Spark ML. By the end of the course, you will have hands-on experience applying Spark skills to ETL and ML workflows.

Apache Spark for Data Engineering and Machine Learning
3 weeks
2–3 hours per week
Progress at your own speed
Optional upgrade available

There is one session available:

6,382 already enrolled! After a course session ends, it will be archivedOpens in a new tab.
Starts Apr 19
Ends Jun 30

About this course

Skip About this course

Apache® Spark™ is a fast, flexible, and developer-friendly open-source platform for large-scale SQL, batch processing, stream processing, and machine learning. Users can take advantage of its open-source ecosystem, speed, ease of use, and analytic capabilities to work with Big Data in new ways.

In this short course, you explore concepts and gain hands-on skills to use Spark for data engineering and machine learning applications. You'll learn about Spark Structured Streaming, including data sources, output modes, operations. Then, explore how Graph theory works and discover how GraphFrames supports Spark DataFrames and popular algorithms.

Organizations can acquire data from structured and unstructured sources and deliver the data to users in formats they can use. Learn how to use Spark for extract, transform and load (ETL) data. Then, you'll hone your newly acquired skills during your "ETL for Machine Learning Pipelines" lab.

Next, discover why machine learning practitioners prefer Spark. You'll learn how to create pipelines and quickly implement features for extraction, selections, and transformations on structured data sets. Discover how to perform classification and regression using Spark. You'll be able to define and identify both supervised and unsupervised learning. Learn about clustering and how to apply the k-mean s clustering algorithm using Spark MLlib​. You'll reinforce your knowledge with focused, hands-on labs and a final project where you will apply Spark to a real-world inspired problem.

Prior to taking this course, please ensure you have foundational Spark knowledge and skills, for example, by first completing the IBM course titled "Big Data, Hadoop and Spark Basics."

At a glance

  • Language: English
  • Video Transcript: English
  • Associated skills:Spark Dataframes, Stream Processing, Unsupervised Learning, Apache Hadoop, Extract Transform Load (ETL), Big Data, Apache Spark, Data Engineering, Operations, SQL (Programming Language), Batch Processing, Machine Learning, Cluster Analysis, Graph Theory

What you'll learn

Skip What you'll learn
  • Describe the features, benefits, limitations, and application of Apache Spark Structured Streaming
  • Describe Graph theory and explain how GraphFrames benefits developers
  • Explain how developers can apply extract, transform and load (ETL) processes using Spark.
  • Describe how Spark ML supports machine learning development
  • Apply Spark ML for regression and classification
  • Differentiate between supervised and unsupervised Machine learning"
  • Explain how Spark ML uses clustering
  • Demonstrate hands-on working knowledge of using Spark for ETL processes

Module 1 – Spark for Data Engineering

  • Spark Structured Streaming

  • GraphFrames on Apache Spark

  • ETL Workloads

  • Hands-on Lab: ETL for ML Pipelines

Module 2 – Spark ML for Machine Learning

  • Spark ML Fundamentals

  • Spark ML Regression and Classification

  • Spark ML Clustering

Module 3 – Final Project

o Lab: Setup & Practice Assignment

o Project Overview

o Lab: Final Assignment Project

o Project Submission & Grading

  • Final Quiz

Who can take this course?

Unfortunately, learners residing in one or more of the following countries or regions will not be able to register for this course: Iran, Cuba and the Crimea region of Ukraine. While edX has sought licenses from the U.S. Office of Foreign Assets Control (OFAC) to offer our courses to learners in these countries and regions, the licenses we have received are not broad enough to allow us to offer this course in all locations. edX truly regrets that U.S. sanctions prevent us from offering all of our courses to everyone, no matter where they live.

This course is part of Data Engineering Professional Certificate Program

Learn more 
Expert instruction
14 skill-building courses
1 year 2 months
3 - 4 hours per week

Interested in this course for your business or team?

Train your employees in the most in-demand topics, with edX For Business.