About the role
Senior Data Engineer; Must Have: Strong hands-on AWS, Snowflake, SQL and dbt experience. Nice to have – Python, PySpark, Airflow, Informatica
Requirements and Responsibilities
What You’ll Do
- Design, develop, and maintain scalable data pipelines and data transformation frameworks using AWS, Snowflake, and dbt. - Develop robust ETL/ELT pipelines to ingest, transform, and integrate data from multiple source systems into Snowflake. - Build and maintain data models in Snowflake supporting analytics, reporting, operational applications, and downstream data consumers. - Develop reusable dbt models, macros, tests, snapshots, and incremental pipelines following established data engineering standards. - Write complex and optimized SQL for data transformation, aggregation, validation, and analytical workloads. - Design and implement scalable data ingestion solutions using AWS services such as S3, Glue, Lambda, and related data services. - Develop data pipelines for both structured and semi-structured data and work with formats such as JSON, CSV, Parquet, and Avro. - Implement incremental loading, CDC, partitioning, clustering, and other optimization techniques to efficiently process large datasets. - Design and maintain appropriate data warehouse schemas and dimensional models, including fact tables, dimension tables, star schemas, and data marts. - Implement data quality and validation frameworks using dbt tests, SQL-based validation, reconciliation, and data profiling. - Monitor data pipelines and proactively identify failures, performance issues, data quality problems, and processing bottlenecks. - Troubleshoot pipeline failures and data discrepancies and perform detailed root-cause analysis. - Optimize Snowflake workloads through appropriate warehouse sizing, query optimization, clustering, partition pruning, and workload management. - Implement appropriate logging, monitoring, error handling, alerting, and recovery mechanisms for production data pipelines. - Develop automated data pipelines and workflows using orchestration tools such as AWS Glue, Airflow, or similar technologies. - Implement version control, CI/CD, and deployment processes for dbt models, SQL code, and data pipelines. - Work closely with Data Architects, Business Analysts, ETL Developers, SQL Developers, QA Engineers, and application teams to understand requirements and deliver scalable data solutions. - Participate in data architecture and technical design discussions and contribute to data platform modernization initiatives. - Support migration of legacy ETL processes to cloud-based AWS/Snowflake ELT architectures. - Develop and maintain technical documentation covering data pipelines, data models, source-to-target mappings, dependencies, and operational procedures. - Conduct code reviews and ensure adherence to development, security, performance, and data engineering standards. - Support production releases, incident management, troubleshooting, and post-production validation. - Work in an Agile/Scrum environment and actively participate in sprint planning, daily stand-ups, reviews, retrospectives, and technical discussions. - Mentor junior data engineers and contribute to knowledge sharing and continuous improvement.
What You’ll Bring
- Bachelor’s degree or higher in Computer Science, Information Technology, Engineering, or a related field. - 5+ years of relevant experience in data engineering, data integration, data warehousing, or cloud data platforms. - Strong hands-on experience with Snowflake as an enterprise cloud data warehouse. - Strong hands-on experience with AWS data services, particularly: - Amazon S3 - AWS Glue - AWS Lambda - IAM and security concepts - Other AWS data/analytics services as applicable
- Strong hands-on experience with dbt (data build tool), including: - dbt models - Incremental models - Macros - Sources - Seeds - Snapshots - Tests - Documentation - Model dependencies
- Advanced proficiency in SQL, including complex joins, CTEs, subqueries, window functions, aggregations, and performance optimization. - Strong understanding of ETL/ELT architecture and data pipeline development. - Strong understanding of data warehousing and dimensional modeling concepts. - Experience designing and implementing fact and dimension models, star schemas, snowflake schemas, and data marts. - Experience working with large-volume datasets and optimizing data processing workloads. - Experience with structured and semi-structured data such as JSON, CSV, Parquet, and Avro. - Strong understanding of data quality, validation, reconciliation, and data governance practices. - Experience implementing incremental loads and CDC processes. - Experience with Git and CI/CD practices for data engineering projects. - Experience with pipeline orchestration and scheduling tools such as Airflow, AWS Glue Workflows, or similar technologies. - Strong troubleshooting and root-cause-analysis skills. - Experience working in Agile/Scrum environments. - Banking or financial-services domain experience is preferred. - Experience with Python for data engineering, automation, or scripting is a plus. - Experience with PySpark / Apache Spark is a plus. - Experience with Informatica or other enterprise ETL tools is a plus, particularly for legacy ETL modernization. - Experience with data migration from traditional data warehouses to AWS/Snowflake is a plus. - Knowledge of Snowflake security, RBAC, data sharing, streams/tasks, and stored procedures is a plus. - Strong analytical and problem-solving skills. - Ability to work independently and collaborate effectively with cross-functional teams. - Strong communication skills in English, both written and verbal.
Recommended External Screening Criteria
For Senior Data Engineer – AWS/Snowflake/dbt, I would keep the following as the core mandatory screening criteria:
Must Have
- 5+ years of Data Engineering / Data Warehousing experience - Strong hands-on Snowflake - Strong hands-on SQL - Hands-on dbt development - AWS data services experience, particularly S3 and Glue - ETL/ELT pipeline development - Data Warehousing & Dimensional Modeling - Experience with large-volume data - Data quality and validation - Git / CI/CD experience
Preferred
- Python - PySpark / Spark - Airflow - Snowflake Streams & Tasks - Snowflake performance tuning - AWS Lambda - Informatica - CDC - Legacy ETL modernization - Banking / Financial Services experience
Text as published by Ness Digital Engineering. We don't edit postings; apply and ask questions on the employer's site.