Job Description
">
As a Data Engineer, you will play a pivotal role in designing and maintaining cutting-edge data pipelines, working across Azure and Databricks.
* Create and manage scalable SparkSQL or PySpark pipelines within an Azure Data Lakehouse setup.
* Leverage tools like Azure Data Factory, Data Lake Gen2, Databricks, and Synapse to consolidate large datasets into clean, reliable, and usable forms.
* Collaborate with stakeholders across the organization to comprehend requirements and contribute to shaping the data architecture that supports it.
* Maintain a close watch on pipeline performance, resolve any issues, and explore opportunities to enhance efficiency.
* Ensure data security across both cloud and on-prem platforms.
* Keep documentation up to date to facilitate understanding of the system's workings among team members.
* Support improvements to our DevOps approach, including CI/CD processes for data work.