Data Engineering on AWS
- Course Duration
- 3 Days
Need Group Training
Course Description
This comprehensive 3-day instructor-led training provides a deep dive into data engineering practices and solutions on Amazon Web Services (AWS). Participants will learn how to design, build, optimize, and secure data engineering solutions by using AWS services. Topics range from foundational concepts to hands-on implementation of data lakes, data warehouses, and both batch and streaming data pipelines. This course equips data professionals with the skills needed to architect and manage modern data solutions at scale.
Course Objective
- Design and implement scalable data lakes and data warehouses on AWS.
- Build, optimize, and secure batch data processing pipelines.
- Develop and manage streaming data solutions.
- Apply best practices for data governance and security.
- Automate data engineering workflows by using AWS services.
- Implement access control and security measures for data solutions.
Course Outline
Module 1: Data Engineering Roles and Key Concepts
- The role of a data engineer
- Data discovery for a data analytics system
- AWS services for data workflows
- Continuous integration and continuous delivery
- Networking considerations
Module 2: Designing and Implementing Data Lakes
- Data lake introduction
- Data lake storage
- Ingest data
- Catalog data
- Transform data
- Serve data for consumption
Module 3: Optimizing and Securing Data Lake Solutions
- Optimizing performance
- Security using Lake Formation
- Setting permissions with Lake Formation
- Security and governance
- Troubleshooting
Module 4: Data Warehouse Architecture and Design Principles
- Introduction to data warehouses
- Amazon Redshift overview
- Ingesting data into Amazon Redshift
- Processing data
- Serving data for consumption
Module 5: Performance Optimization Techniques for Data Warehouses
- Monitoring and optimization options
- Data optimization in Amazon Redshift
- Query optimization in Amazon Redshift
- Data orchestration
Module 6: Security and Access Control for Data Warehouses
- Authentication and access control in Amazon Redshift
- Data security in Amazon Redshift
Module 7: Designing Batch Data Pipelines
- Introduction to batch data pipelines
- Designing a batch data pipeline
- Ingesting batch data
Module 8: Implementing Strategies for Batch Data Pipelines
- Processing and transforming data
- Transforming data formats
- Integrating your data
- Cataloging data
- Serving data for consumption
Module 9: Optimizing, Orchestrating, and Securing Batch Data Pipelines
- Optimizing the batch data pipeline
- Orchestrating the batch data pipeline
- Securing the batch data pipeline
Module 10: Streaming Data Architecture Patterns
- Introduction to streaming data pipelines
- Ingesting data from stream sources
- Storing streaming data
- Processing streaming data
- Analyzing streaming data
Module 11: Optimizing and Securing Streaming Solutions
- Optimizing a streaming data solution
- Securing a streaming data pipeline
Module 12: Compliance and Cost Optimization
- Compliance considerations
- Cost optimization tools