Senior Data Engineer: Firewall Automation Program
* Contract: 6-12 months (Pty Ltd only)
* $1,150-$1,200/day + GST
* Sydney based (Hybrid)
* Urgent start - interviewing immediately
About the role
Build MS SQL ETL pipelines that consolidate 5 million network flows down to 5,000-10,000 automated firewall rules for a tier‑1 bank's CoreMod migration. You'll enrich NetFlow data with CMDB/DNS/VMware metadata, produce JSON/CSV deployment feeds for APIC/CSM, and optimise performance at scale. This role suits a data engineer who thrives on genuinely interesting technical problems in collaborative mob programming environments.
About the engagement
Major infrastructure transformation migrating 400 applications across ~3,000 servers. Three-person specialist team (you, Python dev, network engineer) automating firewall implementation across Legacy and CoreMod infrastructure. Fast-tracked delivery with high autonomy and creative problem‑solving.
What you'll own
Data Pipeline: Consolidate 5M network flows stored in MS SQL into contextualised, validated firewall rules using high-performance T‑SQL ETL jobs with deduplication, enrichment and sliding‑window aggregations.
Deployment Feeds
Produce controller‑ready JSON/CSV exports conforming to CSM import formats and APIC object model payloads that enable the Python dev to automate rule implementation via Cisco APIs.
Orchestration
Create Tidal job definitions, retries, alerts and exception queues. Implement lineage tracking from aggregated tuples back to raw flows for audit purposes.
This is collaborative data engineering – mob programming setup where all three specialists work in lockstep. You'll optimise partitioning, indexing and query plans for very large NetFlow tables while navigating access problems and data challenges with a proactive mindset.
What we need
Must have: Expert MS SQL Server (T‑SQL, indexing, partitioning, window functions, query tuning), high-performance ETL pipelines for large‑scale datasets (5M+ rows), controller‑ready export formats (JSON/CSV) for network automation platforms, Tidal or similar orchestration tools, data engineering mindset (NOT DBA), collaborative team player, Pty Ltd contractor.
Nice to have
Dask/Spark experience, Power BI, understanding of network flows/NetFlow data, migration project experience.
The opportunity
Real infrastructure impact enabling automated firewall deployment at enterprise scale. Strong likelihood of 12-month engagement.
#J-18808-Ljbffr