· intermediate
Cloud Storage and File Formats
Where the data actually sits. Object storage and how it differs from a filesystem, why Parquet beats CSV by an order of magnitude, partitioning that helps rather than hurts, and the small files problem.
Written in house, from the work rather than from a syllabus. Every example is one we would use ourselves, and every quiz question explains why its answer is right.
What you’ll learn
- Explain how object storage differs from a filesystem, and what that rules out
- Choose a file format from how the data will be read
- Describe why columnar storage compresses and scans better
- Partition on a column queries actually filter by
- Recognise and avoid the small files problem
- Say what an open table format adds over bare files
Before you start
- · Data Engineering Foundations, or familiarity with ETL
- · Comfortable with SQL