· advanced
Distributed Processing with Spark
Processing data across many machines. Partitions and lazy evaluation, why a shuffle is the expensive thing, how skew ruins an otherwise good job, and the honest answer about when you do not need Spark at all.
Written in house, from the work rather than from a syllabus. Every example is one we would use ourselves, and every quiz question explains why its answer is right.
What you’ll learn
- Explain how a job is split into partitions and why that decides parallelism
- Distinguish transformations from actions, and predict when work happens
- Identify the shuffle in a job and say what causes it
- Choose a join strategy, including broadcasting a small table
- Recognise data skew from a stage's task durations
- Judge honestly when a single machine would be faster
Before you start
- · Python and SQL
- · Cloud Storage and File Formats, or equivalent