DP 203 Data Engineering on Azure Training

Design a Data Storage Structure :

• Design an Azure Data Lake solution
• Recommend file types for storage
• Recommend file types for analytical queries
• Design for efficient querying
• Design for data pruning
• Design a folder structure that represents the levels of data transformation
• Design a distribution strategy

Design a Partition Strategy :

• Design a partition strategy for files
• Design a partition strategy for analytical workloads
• Design a partition strategy for efficiency/performance
• Design a partition strategy for Azure Synapse Analytics
• Identify when partitioning is needed in Azure Data Lake Storage Gen2

Design the Serving Layer :

• Design star schemas
• Design slowly changing dimensions
• Design a dimensional hierarchy
• Design a solution for temporal data
• Design for incremental loading
• Design analytical stores
• Design meta stores in Azure Synapse Analytics and Azure Data bricks

Implement Physical Data Storage Structures :

• Implement compression
• Implement partitioning
• Implement sharing
• Implement different table geometries with Azure Synapse Analytics pools
• Implement data redundancy
• Implement distributions
• Implement data archiving

Implement Logical Data Structures :

• Build a temporal data solution
• Build a slowly changing dimension
• Build a logical folder structure
• Build external tables
• Implement file and folder structures for efficient querying and data pruning

Implement the Serving Layer :

• Deliver data in a relational star schema
• Deliver data in Parquet files
• Maintain metadata
• Implement a dimensional hierarchy

Ingest and Transform Data :

• Transform data by using Apache Spark
• Transform data by using Transact-SQL
• Transform data by using Data Factory
• Transform data by using Azure Synapse Pipelines
• Transform data by using Stream Analytics
• Cleanse data
• Split data
• Shred JSON
• Encode and decode data
• Configure error handling for the transformation
• Normalize and deformalize values
• Transform data by using Scala
• Perform data exploratory analysis

Design and Develop a Batch Processing Solution :

• Develop batch processing solutions by using Data Factory, Data Lake, Spark, Azure
• Synapse Pipelines, Polybasic, and Azure Data bricks
• Create data pipelines
• Design and implement incremental data loads
• Design and develop slowly changing dimensions
• Handle security and compliance requirements
• Scale resources
• Configure the batch size
• Design and create tests for data pipelines
• Integrate Jupyter/Python notebooks into a data pipeline
• Handle duplicate data
• Handle missing data
• Handle late-arriving data
• Upset data
• Regress to a previous state
• Design and configure exception handling
• Configure batch retention
• Design a batch processing solution
• Debug Spark jobs by using the Spark UI

Design and Develop a Stream Processing Solution :

• Develop a stream processing solution by using Stream Analytics, Azure Data bricks, and
• Azure Event Hubs
• Process data by using Spark structured streaming
• Monitor for performance and functional regressions
• Design and create windowed aggregates
• Handle schema drift
• Process time series data
• Process across partitions
• Process within one partition
• Configure checkpoints/watermarking during processing
• Scale resources
• Design and create tests for data pipelines
• Optimize pipelines for analytical or transactional purposes
• Handle interruptions
• Design and configure exception handling
• Upset data
• Replay archived stream data
• Design a stream processing solution

Manage Batches and Pipelines :

• Trigger batches
• Handle failed batch loads
• Validate batch loads
• Manage data pipelines in Data Factory/Synapse Pipelines
• Schedule data pipelines in Data Factory/Synapse Pipelines
• Implement version control for pipeline artefacts
• Manage Spark jobs in a pipeline

Design Security for Data Policies and Standards :

• Design data encryption for data at rest and in transit
• Design a data auditing strategy
• Design a data masking strategy
• Design for data privacy
• Design a data retention policy
• Design to purge data based on business requirements
• Design Azure role-based access control (Azure RBAC) and POSIX-like Access Control List
• (ACL) for Data Lake Storage Gen2
• Design row-level and column-level security

Implement Data Security :

• Implement data masking
• Encrypt data at rest and in motion
• Implement row-level and column-level security
• Implement Azure RBAC
• Implement POSIX-like ACLs for Data Lake Storage Gen2
• Implement a data retention policy
• Implement a data auditing strategy
• Manage identities, keys, and secrets across different data platform technologies
• Implement secure endpoints (private and public)
• Implement resource tokens in Azure Data bricks
• Load a Data Frame with sensitive information
• Write encrypted data to tables or Parquet files
• Manage sensitive information

Monitor Data Storage and Data Processing :

• Implement logging used by Azure Monitor
• Configure monitoring services
• Measure performance of data movement
• Monitor and update statistics about data across a system
• Monitor data pipeline performance
• Measure query performance
• Monitor cluster performance
• Understand custom logging options
• Schedule and monitor pipeline tests
• Interpret Azure Monitor metrics and logs
• Interpret a Spark directed acyclic graph (DAG)

Optimize and Troubleshoot Data Storage and Data Processing :

• Compact small files
• Rewrite user-defined functions (UDFs)
• Handle skew in data
• Handle data spill
• Tune shuffle partitions
• Find shuffling in a pipeline
• Optimize resource management
• Tune queries by using indexers
• Tune queries by using cache
• Optimize pipelines for analytical or transactional purposes
• Optimize pipeline for descriptive versus analytical workloads
• Troubleshoot a failed spark job
• Troubleshoot a failed pipeline run