About the role
Position : Senior Data Engineer. Location : Malaysia and Singapore. Language : Mandarin Language must. Role : Full Time Perm role + one month Bonus.
Requirements
We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.
Key Responsibilities
- Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python. - Implement Delta Lake and Medallion architecture for scalable enterprise data platforms. - Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources. - Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines. - Optimize Spark jobs for performance, reliability, and cost efficiency. - Implement data validation, quality checks, and monitoring with automated alerts and retries. - Design secure and governed datasets using Unity Catalog and cloud security best practices. - Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics. - Provide technical leadership and guidance to junior team members.
Required Skills
- 5–8+ years of experience in Data Engineering. - Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python. - Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda). - Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage. - Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing. - Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines. - Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).
Good to Have
- Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC). - Knowledge of Power BI or data visualization concepts. - Familiarity with Terraform, Scala, or PostgreSQL. - Experience with SharePoint development or .NET (optional)
Text as published by Unison Group. We don't edit postings; apply and ask questions on the employer's site.