Lead Data Engineer required to support the development of an Azure Databricks platform for a leading financial services business.
Responsibilities:
* design and development of pipelines using Databricks and Lakehouse
* optimisation and maintenance of data workflows, ensuring quality and integrity
* performance tuning and monitoring
* Notebook development using Python and/or Pyspark
* implementation of best practices for data engineering, including governance and security
* mentoring and coaching of junior Data Engineers
Requirements:
* extensive experience building data pipelines in a Databricks Lakehouse environment
* well-versed with Spark and other big data technologies
* excellent coding skills using SQL and Python
* strong background developing solutions in an Azure Cloud environment, including strong ADF development skills
Click on the 'Apply' button to submit your CV.