Job Overview
We are seeking a Data Processing Expert to join our team. This role involves working with cutting-edge technologies and collaborating with cross-functional teams to ensure successful data integration.
Key Responsibilities
* Maintain and develop scalable data pipelines to process large-scale datasets.
* Ingest tables into our data lake from real-time streams using batch processing, micro-batching, or real-time processing techniques.
* Run jobs for ingestion, curation, and extraction of data.
* Alter schemas and configs to meet business requirements and ensure data quality.
* Attend meetings regarding data ingestion and collaborate with cross-functional teams to ensure successful data integration.
* Troubleshoot and resolve bugs in the data processing pipelines.
* Monitor performance and troubleshoot issues using logging and monitoring tools.
Requirements
* Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field.
* 3+ years of experience in data engineering, software development, or a related field.
* Strong programming skills in Scala, Python, or Java.
* Expertise in big data processing concepts and experience with data storage solutions.
* Knowledge of database concepts, data modelling, schema design, and normalisation.
* Familiarity with monitoring and logging tools, and workflow management tools.
Benefits
We offer competitive compensation packages and benefits, along with opportunities for growth and advancement within the company.
About Us
We are a specialist technical data consultancy that provides consultancy services to both private and public organizations throughout Australia.