Azure Data Factory: An advance Hybrid ETL platform

Intro to Azure data factory (ADF):

Microsoft Azure Data Factory is a cloud platform on azure, which is a zero Maintenance, cost-efficient, secured data integration tool with a variety of data source connection and ETL and monitoring capabilities.

In the world of emerging data, where data is growing constantly in different forms like structured, unstructured and semi-structured from various business source but due to unformed data we cannot make much sense and insight out of it.

Data generated required a service that can orchestrate and operationalize process to refine this data to get actionable insight for organizations to take steps in developing Business. Azure data factory gives you this opportunity to take benefits from any form of data and make sense out of it.

Why ADF:

As the question arises in everyone’s mind that there are many ETL services but why to go for Azure data factory?

Hybrid ETL platform: ADF is an ETL platform which can support various data sources and Enrich ETL activities. Data collection, cleansing and modelling are possible with no-structured, Semi-structured and structure where data can reside anywhere in the cloud environment

Pre-build connector for everything: ADF has all-most linked service connection to any kind of data sources like EC2, Azure blog, Databases and many more.

Fully Managed and Monitored: ADF comes with Integration Runtime (IR), which help to run developed ETL and Dataflow pipelines. Also monitoring dashboard is there to all the running ETL pipelines.

Simplified UI for Everyone:  Microsoft is also famous for providing advance techno-tools with simplified UI and great documentation. Which help to learn and understand the technology better.

Secure and cost-efficient: ADF management and computation resource are built on Azure security infrastructure and use all the Azure security measures. The Azure cloud platform is one of the most secure of all cloud platform, also Microsoft provides personalized security recommendation bases on you use cases.

How does it Work:

Azure data factory come-up with various data sources connection and activities which can be performed on your data.

Data Collection (Source): data source can be anywhere in the cloud or on-premises, so it is required to have a proper data collection to perform ETL. ADF provides a vast variety of data source connections.

Enriched ETL: After getting data from the source, different ETL activities need to be performed on the datasets as per the business logic like aggregation, join, filter. ADF provides Enrich ETL activities which can be used in cleansing, modelling and data preparation. Ex: Azure function, Copy Activity, Hive Activity, Azure Databrick activity, HDInsight Activity

Publish and versioning:  with changing business logic and requirement, the developer needs to make changes enhancement in ADF pipeline. ADF provides version controlling with GIT, where at any point we can revert the changes and fetch older versions.

Monitor pipelines:  ADF pipeline monitoring makes it easy to debug and monitoring with logging the pipeline activity. Also, pipeline after and triggers are possible which can give failure alert and initiate auto-execution of pre-debugging steps.

More about ADF: -

ADF platform fits for everyone, as it provides enrich functionality for maintaining, monitoring and upgrading monotonous ETL and data related task with simply UI base platform. It provides flexible pricing tier and charges-based usage. Upgrading resources for any services is easy, also provides lift and shift of on-premises SSIS ETLs in Azure SSIS integration services.

Choose ADF data integration capabilities to fit your scale, infrastructure, compatibility, performance and budget needs—from managed SQL Server Integration Services for seamless migration of SQL Server projects to the cloud to large-scale, serverless data pipelines for integrating data of all shapes and sizes.

Reference:

·         https://docs.microsoft.com/en-us/azure/data-factory/introduction

·         https://docs.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor

·         https://cloudacademy.com/blog/what-is-azure-data-factory/