A dynamic technology company is looking for an experienced Senior Data Engineer (Spark & Python Specialist) with deep expertise in building, optimizing, and maintaining high-performance data processing engines and modern data lakehouse architectures. The ideal candidate will contribute to technical excellence, refactor legacy ETL logic, and support scalable, cloud-agnostic data solutions.
Responsibilities:
-
Apply Spark best practices for memory management, shuffle tuning, and partitioning to optimize data pipelines.
-
Develop and maintain modular Python/PySpark data pipelines with Delta Lake/Parquet in a cloud-agnostic environment.
-
Refactor complex SQL-based ETL into maintainable Python libraries.
-
Build and optimize Medallion Architecture layers (Bronze/Silver/Gold) in the data lakehouse.
-
Support code-first orchestration using tools like Airflow or Dagster, reducing dependency on GUI-based orchestration.
-
Participate in code reviews, mentor junior engineers, and contribute to automated testing frameworks (Pytest).
-
Collaborate with data scientists, analysts, and business stakeholders to translate requirements into actionable data solutions.
-
Lead initiatives to modernize data workloads, introduce new technologies, and drive best practices.
-
Ensure compliance with data security, governance, and quality standards.
Requirements:
-
Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.
-
6+ years of experience with Spark/PySpark, with the ability to optimize complex DAGs and diagnose performance issues.
-
Advanced Python skills, including building reusable libraries and implementing automated testing.
-
Strong SQL (T-SQL) skills for interpreting and migrating legacy logic.
-
Hands-on experience with data lakehouse technologies, including Delta Lake and Parquet.
-
Experience with Azure Synapse Analytics, Dedicated SQL Pools, and Data Factory for complex pipelines.
-
Familiarity with containerization (Docker) and open-source standards for cloud-agnostic workloads.
-
Proven collaborative experience within technical teams and mentoring junior engineers.
Benefits
- Salary: negotiable.
Contact RPO Recruitment for your next career opportunity.
- Our client is offering a highly competitive salary for this role based on experience.
- Apply for this role today, contact Jarad Pentz at RPO Recruitment or on LinkedIn.
- You can also visit the RPO Recruitment website: rporecruitment.us or email us your
CV: tech@rpo-recruit.za.com.
We will contact you telephonically in 3 days should you be suitable for this vacancy. If you are not suitable, we will put your CV on file and contact you regarding any future vacancies that arise.