About Us
IAG Defence & National Security (DNS) is part of IAG, a specialist advisory firm supporting complex infrastructure, Defence and national security programs across Australia.
The Role
The Australian Signals Directorate (ASD) is seeking an EL1 Equivalent Level Data Engineer (Data Platform Engineer). The Platform Engineer enables efficient delivery by aligning development practices to business priorities, minimising operational complexity and lifecycle management overhead across applications and supporting infrastructure. The Data Platform Engineer owns end-to-end data platform outcomes, delivering scalable, resilient, and observable ingestion, streaming, storage, and analytics capabilities that underpin mission-critical intelligence and cyber security operations.
About You
1. Contribute to the optimisation and maintenance of data ingestion pipelines including both stream and batch technologies.
2. Develop and manage data storage table and message schemas for various event formats.
3. Support Hadoop and similar technologies to ensure data is stored and accessed efficiently.
4. Ensure the effective implementation of role-based access controls across data lake components through technical enforcement and audit.
5. Identify, report and remediate platform issues through proactive monitoring of health and performance metrics.
6. Identify process improvements and contribute to data ingestion standards and definition of best practice.
7. Communicate and report data ingestion and analysis related risk and issues to the project manager.
8. Enhance and configure highly scalable, resilient, and observable data analysis platforms, including serialization, transformation, enrichment and store.
9. Manage multiple, concurrent tasks with competing demand.
10. Motivate and mentor teams, aligning complementary skills and allocating resources in a manner that delivers results.
11. Installation, optimisation, configuration, and operational management of search and analytical clusters.
Criteria Considered Essential For This Role
12. Hold or obtain a TSPV AGSVA Clearance.
13. Demonstrated expertise in the installation, configuration, optimisation, and operational management of large-scale search and analytical clusters, ensuring performance, resilience, security and availability across their full operational lifecycle.
14. Accountable experience delivering resilient, scalable data streaming capabilities using Kafka, NiFi, and Spark to enable real-time analytics and mission-critical data flows.
15. Demonstrated competence in software development using Python, Scala, and Java, applying sound engineering practices to build, integrate, and maintain production-grade systems.
16. Demonstrated proficiency with infrastructure automation frameworks, including Ansible, Puppet and Chef, alongside the design and operation of CI/CD pipelines to enable reliable, repeatable and scalable delivery.
17. Demonstrated strong hands-on capability in Linux system administration, including performance optimisation, fault diagnosis and troubleshooting across production environments.
18. Demonstrated experience designing, implementing and supporting data pipelines across their full lifecycle, ensuring reliability, performance and operational stability.
Benefits
19. Offices in Perth, Brisbane, Melbourne, Sydney and Canberra.
20. Opportunity to work on critical infrastructure and Defence projects across Australia.
#J-18808-Ljbffr