About the role
Description
We're on the lookout for exceptional individuals to join our team as Senior Lead Data Engineers – Databricks. As part of our team, you'll play a key role in designing and delivering modern, scalable data platforms for clients across multiple industries, leveraging Azure and Databricks technologies.
You will lead the design, development, and optimization of end-to-end data engineering solutions built on modern Lakehouse architectures. Working closely with clients and cross-functional teams, you'll translate business requirements into scalable, high-performance data platforms while mentoring engineers and driving technical excellence.
Requirements
- Responsibilities
- Lead the design, development, and maintenance of scalable data engineering solutions using Azure and Databricks. - Design and implement modern Lakehouse architectures following industry best practices. - Build and optimize large-scale batch and streaming data pipelines. - Develop ETL/ELT pipelines using Databricks, PySpark, SQL, and Azure Data Factory. - Design, implement, and optimize Delta Lake solutions for reliable and performant data processing. - Collaborate with clients to gather requirements and translate business needs into technical solutions. - Optimize Spark workloads for performance, scalability, and cost efficiency. - Design and maintain enterprise-grade Data Warehouses, Data Lakes, and Lakehouses. - Implement data governance, security, and metadata management using Azure services such as Purview, Key Vault, and Microsoft Entra ID (Azure Active Directory). - Establish CI/CD pipelines and DevOps practices for data engineering workloads. - Mentor and coach junior data engineers while promoting engineering best practices. - Participate in architecture discussions, solution design, and technical leadership initiatives. - Contribute to internal knowledge sharing and continuous improvement.
- - Requirements - Required
- 10+ years of professional experience in Data Engineering. - 4+ years of hands-on experience with Databricks. - Strong experience developing data pipelines using PySpark and Apache Spark. - Strong experience designing and implementing Lakehouse architectures. - Experience working with Delta Lake and Medallion Architecture. - Strong knowledge of SQL and Python. - Experience with Azure Data Factory. - Experience working with Azure Data Lake Storage Gen2 (ADLS Gen2). - Experience with Azure Synapse Analytics. - Experience with Data Warehousing concepts and dimensional modeling. - Strong understanding of ETL and ELT design patterns. - Experience optimizing Spark jobs, partitioning strategies, joins, caching, and workload performance. - Experience with Git and Azure DevOps CI/CD pipelines. - Experience implementing secure data platforms using Azure Key Vault, Microsoft Entra ID, and Microsoft Purview. - Experience building scalable enterprise data platforms in Azure. - Experience working directly with clients and gathering technical requirements. - Experience mentoring engineers and leading technical initiatives.
- - Strongly Preferred
- Experience with Snowflake. - Experience building real-time streaming solutions using Structured Streaming, Kafka, or Event Hubs. - Experience with Databricks Workflows. - Experience with Delta Live Tables. - Experience with Unity Catalog. - Experience with Databricks SQL. - Experience with MLflow. - Experience with dbt. - Experience with Airflow or other orchestration platforms. - Experience working in Agile environments.
- - Nice to Have
- MSc in Computer Science or a related field. - Experience with DataOps practices. - Experience with MLOps. - Experience with Azure Machine Learning. - Experience with Azure Cognitive Services. - Experience with Docker and Kubernetes. - Experience working with Microsoft Fabric. - Experience with Power BI.
- - Technical Skills - Core Technologies
- Azure Databricks - Apache Spark - PySpark - SQL - Python - Delta Lake - Azure Data Factory - Azure Synapse Analytics - ADLS Gen2 - Snowflake
- Data Engineering
- Data Warehousing - Data Lakes - Lakehouse Architecture - Medallion Architecture - Dimensional Modeling - ETL / ELT - Batch Processing - Streaming Data Pipelines - Data Governance - Performance Optimization
- Azure Ecosystem
- Azure DevOps - Microsoft Purview - Azure Key Vault - Microsoft Entra ID - Azure Functions - Event Hubs
- Nice-to-Have Technologies
- Kafka - Airflow - dbt - MLflow - Docker - Kubernetes - Power BI
Benefits
- Salary paid in USD - Six-month career advancing opportunities - Supportive and friendly work environment - Premium medical insurance [employee +family] - English language development courses - Interest-free loans paid over 2.5 years - Technical development courses - Planned overtime program (POP) - Employment referral program - Premium location in Maadi - Social insurance
Text as published by Integrant. We don't edit postings; apply and ask questions on the employer's site.