About the role
We are seeking a Director of Software Engineering with deep expertise in Node.js development and large-scale web scraping. This role will lead the engineering team, designing and optimizing high-performance, distributed web scraping systems. The ideal candidate has extensive experience in handling anti-bot measures, data pipeline optimization, and scalable cloud-based architectures.
Key Responsibilities- Software Engineering & Web Scraping Leadership: • Architect, develop, and maintain scalable and distributed web scraping systems using Node.js. • Design and implement data extraction pipelines to process large volumes of structured and unstructured data. • Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation. • Optimize scraping processes for performance, reliability, and efficiency while managing proxy services(residential, datacenter, rotating).Oversee data storage and processing strategies, ensuring high availability and consistency. • Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications. • Implement best practices for microservices, API integrations, and real-time data streaming.
Key Responsibilities- Scalability, Security & DevOps: • Lead the transition to cloud-native, containerized, and serverless architectures for web scraping. • Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.).Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping. • Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks. • Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability.
Key Repsonsibilities- Engineering Team Management & Strategy: • Lead, mentor, and grow a high-performance engineering team. • Define and execute the technology roadmap, aligning with business objectives. • Foster a culture of continuous learning, collaboration, and innovation. • Implement agile development methodologies (Scrum, Kanban) to optimize project execution. • Ensure code quality, security, and best practices across all engineering efforts.
Qualifications & Experience- Technical Expertise: • 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction. • Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation. • Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques. • Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.). • Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).
Qualifications & Experience- Leadership & Soft Skills: • Proven track record of scaling engineering teams and leading complex projects. • Strong problem-solving and debugging skills, especially for scraping challenges and performance bottlenecks. • Excellent communication and stakeholder management skills. • Passion for mentorship, team development, and continuous learning.
Preferred Qualifications: • Experience with machine learning for data extraction and NLP. • Knowledge of browser fingerprinting and bot detection mechanisms. • Familiarity with enterprise-scale web crawling frameworks (Scrapy, Colly, Apify, etc.). • Prior leadership experience in data-driven businesses or web scraping startups.
Text as published by PortPro. We don't edit postings; apply and ask questions on the employer's site.