Data Engineer – Spark & Kafka
We are looking for a Data Engineer with experience in Spark and Kafka to operate with one of our large Federal Government Clients. As a Data Engineer, you will have the opportunity to work with cutting‐edge technologies and collaborate with cross‐functional teams to ensure successful data integration.
Responsibilities
* Maintain and develop Spark pipelines to process large‐scale data sets.
* Ingest tables into our data lake from Kafka using batch processing, micro‐batching, or real‐time processing techniques.
* Run Spark jobs for ingestion, curation, and extraction of data.
* Alter schemas and configs to meet business requirements and ensure data quality.
* Attend meetings regarding data ingestion and collaborate with cross‐functional teams to ensure successful data integration.
* Troubleshoot and resolve bugs in the data processing pipelines.
* Monitor performance and troubleshoot issues using logging and monitoring tools.
* Document best practices and develop standards for data processing pipelines.
* Stay up‐to‐date with emerging technologies and industry trends in big data processing and analytics.
Requirements
* Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field.
* 3+ years of experience in data engineering, software development, or a related field.
* Strong programming skills in Scala, Python, or Java.
* Expertise in Spark, Kafka, and big data processing concepts.
* Experience with data storage solutions such as Hadoop HDFS or Amazon S3, and data serialization formats.
* Knowledge of database concepts, data modelling, schema design, and normalisation.
* Familiarity with monitoring and logging tools, and workflow management tools such as Apache Airflow.
* Experience with version control systems such as Git.
* Security clearance required – Baseline (NV1 preferred).
Attributes and Aptitude
* Strong communication, problem‐solving, and collaboration abilities.
* Ability to work independently and in a team‐oriented, collaborative environment.
Benefits
We offer competitive compensation packages and benefits, along with opportunities for growth and advancement within the company. As a member of our team, you will have the chance to work on challenging projects, develop your skills, and make a real impact on our business.
#J-18808-Ljbffr