Position
: AWS Data Engineer
Location
: Sydney, NSW
Duration
: 6 months
Job Details:
Must Have Skills:
* Strong hands-on experience with
AWS Data Services
: S3, Glue, Lambda, Redshift, Athena, EMR
* Expertise in building
ETL/ELT pipelines
using AWS Glue / PySpark
* Proficiency in
Python
and
Advanced SQL
for data transformation
* Strong understanding of
data modeling
, partitioning strategies, and performance optimization
* Experience with
distributed data processing
(Spark / EMR)
* Knowledge of
IAM
, KMS, VPC, and AWS security best practices
* Hands-on experience with
CI/CD
, version control (Git), and Infrastructure as Code (Terraform / CloudFormation)
* Ability to work with large-scale datasets and optimize compute & storage costs
Nice to Have Skills:
* AWS Certifications (AWS Certified Data Engineer – Associate, Data Analytics Specialty, Solutions Architect)
* Experience with
real-time streaming platforms
(Kinesis or Kafka)
* Familiarity with orchestration tools (Airflow, Step Functions)
* Knowledge of
AWS Lake Formation
for security, governance, and cataloging
* Experience supporting ML pipelines using SageMaker
* Knowledge of DevOps and monitoring tools (CloudWatch, Datadog, Grafana)
* Exposure to containerized ecosystems (Docker, EKS)
Detailed Job Description:
* The AWS Data Engineer is responsible for designing, developing, and maintaining scalable, secure, and high‑performance data pipelines and cloud-native data solutions on the AWS platform.
* The role involves working with large and diverse datasets and enabling analytics, reporting, and advanced machine learning workloads.
* You will work closely with cross-functional teams including Data Architects, Data Scientists, Analysts, and Application Engineers to implement data ingestion, transformation, storage, and retrieval processes.
* The engineer will be accountable for ensuring high data quality, lineage, metadata management, and governance throughout the lifecycle.
* Responsibilities include building and optimizing ETL/ELT workflows using services like AWS Glue, Lambda, EMR, Redshift, and Step Functions. You will also design and maintain data lakes on Amazon S3, implement schema evolution, support partitioning strategies, and ensure pipeline reliability and cost efficiency.
* The role requires deep AWS knowledge, excellent coding practices in Python and SQL, and adherence to AWS security standards, including IAM, VPC configuration, encryption, and compliance.
* The AWS Data Engineer will also play a key role in implementing CI/CD pipelines for data workflows, automating deployments using Cloud Formation or Terraform, performing operational monitoring, and troubleshooting production workloads.
* You will be expected to collaborate in Agile teams, participate in sprint planning, conduct code reviews, and contribute to architectural decisions.
Minimum Years of Experience:7