Azure Data Engineering Preparation With Real World Projects

2 Real World Projects Using Azure Data engineering Tool Like Azure Data Factory, Functions App, Azure Data Bricks, AZSQL
What you'll learn
2 End to End Azure Data Engineering Projects
Learn Azure Data Factory
Learn Azure Databricks
How to transform data in the Azure Databricks using the pyspark
How to mount a storage account in Databricks Azure?
10 Real time use cases of Azure Data engineering services
15 Common Azure Data Engineering Interview questions and answers
Requirements
Basics of Cloud computing
Internet Connections
Mobile Phone or Laptop Or Desktop
Description
Hello, Welcome to this New Azure Data Engineering Course. In this course We will demonstrate two real world scenario based projects using Azure Data Engineering Tool. This will give a real time experience to some of the common scenarios that you may face in your day to day life as azure Data engineer. Is Azure good for data engineer?Microsoft Azure is one of the best tool for data engineers as they can build different applications and deploy it any azure regions to utilizing their existing capabilities. No matter where their data is stored, azure data engineering services help you to access them, transform them and export them into any storage system that you want use. In this projects we will demonstrate some of the feature that Azure data factory supports. Project One: Read Data From A Online Storge System and process them using Azure Data Factory. This project covers one real-time industry use cases and project work to give you hands-on experiences on using Azure data engineering tools and services like Azure Data factory, Azure functions and Azure SQL .Project Two: Read multiple files using ADF. Validate your file structure using Azure SQL Data base table. Process valid files using Azure Data Factory. Process multiple files using same Azure Data Factory Pipelines Data Flow (Dynamic Data flows) . Make your data flow dynamic, so that single data flow can process multiple files with different structures and columns names and data types. Just wanted add one point: This course cover some of the advanced azure data factory features and functions. It will be really helpful, if you already know some of the basics of azure data factory. But still we will cover all the topics, that are necessary for completing this project in details (End to End).
Overview
Section 1: Projects 1
Lecture 1 Introduction To Project 1
Lecture 2 Introductions To Part 1 Of This Project
Lecture 3 Introductions To Part 1 Of This Project
Lecture 4 Save Raw Data In GitHub
Lecture 5 Create Azure Data Lake Storage Gen 2 Account (ADLS)
Lecture 6 How To Create Azure Data Factory Account
Lecture 7 How To Create Containers in ADLS?
Lecture 8 How To Create Linked Services ADF?
Lecture 9 How To Create Data Set In ADF?
Lecture 10 How To Create A Pipeline In ADF and Configure Copy Activity
Lecture 11 Create New Data Set and Copy Second files
Lecture 12 How To Reuse Data Set With The Help Of Parameter
Lecture 13 Copy 16 Files Using Single Copy Activity
Section 2: Project 1 :Second Part
Lecture 14 Azure Functions: Intro
Lecture 15 How to Test & Validate Blob Trigger Functions In Azure Functions App
Lecture 16 How To Add Logical Testing Code In Azure Functions, For Validations
Lecture 17 How To Add Output Binding in Azure Functions
Lecture 18 End To End Testing HTTP to Azure Storage Using ADF And Validate Functions App
Lecture 19 Azure Function App: Fix File Name Issues
Section 3: Project 1 : Final Part
Lecture 20 Final Part Of This Project
Lecture 21 How To Create Azure SQL DB ?
Lecture 22 How to Connect To Azure SQL Using SSMS & From Azure Portal
Lecture 23 How To Create Linked Service To Access Azure SQL
Lecture 24 How To Create Data Set To Access Azure SQL DB?
Lecture 25 How To Copy Data Into Azure SQL
Lecture 26 How To Copy Full Data Into Azure SQL
Lecture 27 How To Fix Common Issues
Section 4: Project 2: Mastering ADF Dynamic Pipelines
Lecture 28 Introductions To Project Requirements
Lecture 29 Understand Data and Data Transformations Requirements
Lecture 30 Design Target Table For First Data Set
Lecture 31 Create Data Set: Azure Data Lake and Azure SQL Data set
Lecture 32 Create Data Flow And Add Multiple Source ( ADLS File & Azure SQL Table)
Lecture 33 Make Our Data Flow Using Parameters
Lecture 34 How To Derive New Columns From Existing Columns And Parameters.
Lecture 35 How To Use Exist To Validate Source And Target Data
Lecture 36 Calculate New Surrogate Key And Max Surrogate Key
Lecture 37 Join Max Surrogate Key With New (Or Updated) Data Set
Lecture 38 Derive Additional Columns: Active Status ,and Current Dates
Lecture 39 Select Relevant Column Using Select Activiti-Role Based Mapping
Lecture 40 Process Updated Data Using New Branch Activity
Lecture 41 Select Proper Columns Using Role Based Mappin(Different Expression)
Lecture 42 Define Insert Set Data And Update Set Data
Lecture 43 Merge Insert And Update Data Sets
Lecture 44 Add Sink And Execute Our Pipeline
Lecture 45 Unit Testing: Validate Pipeline Execution Step 1
Lecture 46 Unit Testing : Validate Pipeline Executions Step 2
Section 5: Project 2: Part B
Lecture 47 Introduction To New Data Set
Lecture 48 Make Our Data Set Dynamic Using Parameters
Lecture 49 Make Our Pipeline Dynamic
Lecture 50 Execute Our Pipeline With New Data Set
Section 6: Project 2:Part C
Lecture 51 Introductions To Final Requirements
Lecture 52 Defining Table To Store Structure Of The Table
Lecture 53 How Define A Dynamic Stored Procedure To Read File Structure
Lecture 54 How To Validate File Structure of Two files
Lecture 55 How To Store Structure Details In SQL Table
Lecture 56 Validate Structure Using Azure SQL table and Stored Procedures
Lecture 57 Execute New Pipeline After validations
Lecture 58 Test All the Scenarios (Same Structure , different Structures ) End To END Unit
Section 7: Project 3
Lecture 59 Create Azure Data Lake Gen 2 And Azure Databricks
Lecture 60 Register an application with Azure AD and create a service principal
Lecture 61 Assign Roles To The Application To Provide The Service Principal Permissions
Lecture 62 Add application secret to the Azure Key Vault
Lecture 63 Create a Secret Scope in Azure Databricks
Lecture 64 Create Containers ( bronze/ Raw, silver / Processed , and gold/Final)
Lecture 65 Create Your First Cluster in Databricks
Lecture 66 Create A Notebook
Lecture 67 Mount Azure Data Lake without Key Vault
Lecture 68 Read CSV file from Data Lake
Lecture 69 Mount Data lake using Azure Key Vault
Section 8: Bonus
Lecture 70 Bonus
If you are looking for a Real World Data engineering uses cases, then this course is for you,Any student who is planning to learn azure data bricks,Any student who is planning to learn azure data factory,Students looking for a career in Azure Data Engineering,For all the database developer who wants to learn azure data engineering,For business analyst and data analyst who wants to learn azure data engineering


Homepage

https://www.udemy.com/course/azure-data-engineering-projects/