Job Summary:
* As a Senior Data Engineer, you will be responsible for designing and building end-to-end Big Data Pipelines using Apache Spark, Py-Spark, Hive, and HBase.
Requirements:
* 7+ years of experience in data engineering with expertise in Big Data technologies such as Apache Spark, Py-Spark, Hive, and HBase.
* Experience with AWS environment and Agile methodology.
* Knowledge of ETL pipeline design and real-time data processing using HBase and Hadoop ecosystem.
* Excellent problem-solving skills and ability to work independently.
* Familiarity with Atlassian tools, including Jira, Confluence, and Bitbucket.
* Proven track record of delivering projects successfully through effective planning, prioritizing, team building, task scheduling, and tracking scheduled tasks.
Responsibilities:
* Design and build complex data structures.
* Analyze jobs and provide root cause analysis (RCA) if issues are raised post-sprint release.
* Participate in implementation and design walkthroughs with various teams and work on production deployment on change-related activities.
* Develop Python scripts for temporary transformation tables using Py-Spark and Spark SQL before populating the records into SCD tables.
* Build an end-to-end metadata-driven data quality framework that can be utilized across different projects, consisting of metadata, attribute, and process sheets.
Working Environment:
* The ideal candidate will have a strong background in data engineering with experience in Big Data technologies.
* The role requires excellent communication and collaboration skills, with the ability to work effectively in a team environment.
* The successful candidate will have a proven track record of delivering projects successfully through effective planning, prioritizing, team building, task scheduling, and tracking scheduled tasks.
Benefits:
* This is an opportunity to work on complex data structures and contribute to the success of the organization.
* The role offers a challenging and dynamic working environment with opportunities for growth and professional development.
* The ideal candidate will be motivated by the prospect of working on high-profile projects and contributing to the success of the organization.