* Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform.
* Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale.
* Optimize and troubleshoot AWS Glue jobs for performance and reliability.
* Utilize Python and PySpark to efficiently handle large volumes of data during the ingestion process.
* Design and implement scalable data processing solutions using PySpark to transform raw data into a structured and usable format.
* Apply data cleansing, enrichment, and validation techniques to ensure data quality and accuracy.
* Create and maintain ETL processes using Python and PySpark to move and transform data between different systems.
* Optimize ETL workflows for performance and efficiency.
* Collaborate with data architects to design and implement data models that support business requirements.
* Ensure data structures are optimized for analytics and reporting.
* Work with distributed computing frameworks, such as Apache Spark, to process and analyze large-scale datasets.
* Manage and optimize databases, both SQL and NoSQL, to support data storage and retrieval needs.
* Implement indexing, partitioning, and other database optimization techniques.
* Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver effective solutions.
* Work closely with software engineers to integrate data solutions into larger applications.
* Implement monitoring solutions to track data pipeline performance and proactively identify and address issues
* Ensure compliance with data privacy regulations and company policies.
* Stay abreast of industry trends and advancements in data engineering, Python, and PySpark.
Requirements
* Proficiency in Python and PySpark.
* Strong knowledge of data engineering concepts and best practices.
* Hands-on experience with AWS Glue and other AWS services.
* Experience with big data technologies and distributed computing.
* Familiarity with database management systems (SQL and NoSQL).
* Understanding of ETL processes and data modeling.
* Excellent problem-solving and analytical skills.
* Strong communication and collaboration skills.
* Bachelor's degree in Computer Science, Information Technology, or a related field.