Job Description:
The role of Data Engineer is key to the development and maintenance of a data-driven organization.
The successful candidate will report to the Head of Data and Analytics and be responsible for designing, developing, and maintaining a robust data ecosystem using Databricks, Data Factory, and other Azure cloud data engineering tools.
* Key responsibilities include creating data models, building data pipelines, implementing test automation solutions, and mapping business requirements to technical systems.
* Collaboration with stakeholders to define data engineering requirements and align solutions with business objectives.
* Development and optimization of data pipelines for efficiency, accuracy, and scalability.
* Implementation of best practices for data governance, security, and compliance within the Azure ecosystem.
* Monitoring system performance, troubleshooting issues, and implementing enhancements to optimize the data environment.
* Staying updated on emerging technologies and methodologies in data engineering and incorporating relevant advancements into the organization's infrastructure.
Requirements:
Extensive experience in data engineering with expertise in Azure Databricks, Microsoft Fabric, Python, and the Azure cloud data ecosystem is required.
Strong background in ETL/ELT processes, data modeling, and performance optimization is essential. Familiarity with CI/CD pipelines and DevOps practices in a cloud environment is also necessary.
Experience with machine learning pipelines is an advantage.