Job Description: Data Engineer - Azure Data Factory and SQL Database Expertise
We are seeking a skilled Data Engineer with expertise in Microsoft Azure Data Factory (ADF) and SQL Database to join our dynamic team. The ideal candidate will have a strong background in Data Warehousing, ETL/ELT processes, and building robust data pipelines on Azure cloud platforms. This role is critical to ensuring efficient data modeling, integration, and transformation processes to support business operations and decision-making.
Key Responsibilities
* Design, build, and maintain scalable data pipelines and ETL/ELT processes using Azure Data Factory (ADF)and other Azure cloud services.
* Write and optimize T-SQL procedures for querying, data transformation, and validation in a Data Warehouse environment.
* Develop and maintain metadata-driven data load frameworks for automated processing.
* Implement dynamic data masking and ensure data security within Azure SQL Databases.
* Perform data modeling to support Data Warehousing and Data Mart architecture.
* Collaborate with stakeholders to gather requirements and provide technical solutions that meet business needs.
* Ensure data quality checks and implement quality management processes.
* Contribute to CI/CD pipeline implementation and management using Azure DevOps.
* Maintain repository management using Git version control.
* Engage in peer programming and contribute to agile project delivery.
* Work with Azure Functions and Logic Apps to extend data integration capabilities.
Essential Criteria
* 5+ years of experience as a Microsoft Azure Cloud ADF Data Engineer, including T-SQL in a Data Warehouse environment.
* Solid understanding of Data Warehousing and Data Mart Architectures on Azure Cloud.
* Proficiency in SQL for data querying, transformation, and validation.
* Experience in building metadata-driven frameworks for automation on Azure.
* Strong knowledge of data security practices and dynamic data masking in Azure SQL Database.
* Hands-on experience with Azure Data Lake, Azure SQL Database, Azure Functions, and Logic Apps.
* Minimum 3 years of experience in design and build using an agile delivery framework.
* Familiarity with peer programming practices.
Desirable Criteria
* Knowledge of SPSS programming language.
* Experience in the Transport Safety industry (highly beneficial).
* Proficiency with CI/CD pipelines in Azure DevOps and version control with Git.
* Agile Scrum Master experience (optional but advantageous).
* Strong communication skills for engaging business and technical stakeholders.
Qualifications and Skills
* Expertise in data engineering technologies including data extraction, integration, and transformation.
* In-depth knowledge of data modeling and working with various file types, databases, and APIs.
* Hands-on experience with data quality management processes.