Principal Data Engineer Job Description
The Role: This position requires a dynamic, hands-on individual who can build real-time data pipelines and implement data mesh architectures to enhance client data interactions. The ideal candidate will have deep technical expertise in modern data engineering methods and a client-facing consulting approach.
">
* Key Responsibilities:
o Design and Architecture: Design scalable solutions using Databricks Lakehouse Architecture, applying best practices around Delta Lake, data modeling, and distributed processing.
o Developing Data Solutions: Implement and enhance data-driven solutions integrating with clients' systems using state-of-the-art tools such as Databricks, Snowflake, Google Cloud, and AWS.
o Data Pipeline Development: Develop and optimize high-performance, batch and real-time data pipelines employing advanced streaming technologies like Kafka and Flink.
o Database and Storage Optimisation: Optimise and manage a broad array of database technologies from traditional relational databases (e.g., PostgreSQL, MySQL) to modern NoSQL solutions (e.g., MongoDB, Cassandra).
Required Skills and Qualifications:
* Experience with Databricks Platform, Core Data Engineering Tools & Technologies, and Data Storage Expertise.
* Proficient in SQL and Spark, familiar with platforms such as Databricks and Snowflake.
* Knowledgeable in managing various data storage formats including Parquet, Delta, ORC, Avro, and JSON to optimise data storage and retrieval.
* Experienced in developing and overseeing large-scale data pipelines and data-intensive applications within production environments.
Benefits:
* Attractive salary package
* Opportunity to work on cutting-edge projects
* Collaborative and dynamic work environment
Professional Experience and Qualifications:
* At least 8+ years of data engineering or equivalent experience in a commercial, enterprise, or start-up environment.
* Degree or equivalent experience in computer science or a related field.