Note: The job is a remote job and is open to candidates in USA. Two Six Technologies is a company that builds, deploys, and implements innovative products to solve complex challenges. They are seeking a highly skilled Data Collection Engineer to design, scale, and maintain distributed web scraping and data extraction infrastructure, focusing on building resilient data pipelines and ensuring data quality.
Responsibilities
- Design and deploy high-performance, distributed web scrapers using Python and Scrapy to extract massive datasets efficiently
- Utilize Browser Scripting tools to navigate, interact with, and extract data from modern, dynamic, and JavaScript-heavy websites
- Deploy, scale, and manage scraping workloads on Kubernetes, ensuring optimal resource allocation and fault tolerance
- Define strict JSON Schemas and leverage Pydantic to enforce data types, validate incoming payloads, and catch data drift early
- Build and optimize search and storage pipelines using Elasticsearch, transforming raw web dumps into highly structured, searchable data
- Architect robust pipeline workflows to manage the end-to-end data lifecycle—from discovery and extraction to validation and storage
- Manage complex proxy rotation, session handling, and browser fingerprinting to maintain high success rates against advanced anti-scraping systems
Skills
- 7+ years of professional software engineering experience, with a heavy focus on web scraping, data engineering, or distributed systems
- Excellent reverse-engineering skills, with the ability to dissect network traffic, unearth hidden APIs, and bypass complex web barriers
- A strong commitment to data integrity, system monitoring, and building self-healing scraping systems
- Expert-level proficiency in Python
- Deep experience with Scrapy and distributed scraping architectures (e.g., handling distributed queues, broad vs. deep crawling)
- Proven experience with browser automation tools (Playwright, Selenium, or Puppeteer)
- Mastery of JSON, JSON Schema, and data validation using Pydantic
- Hands-on experience indexing, querying, and optimizing Elasticsearch clusters
- Strong proficiency in managing and scaling applications within Kubernetes environments
- Experience building structured pipeline workflows to handle complex, multi-stage data extraction tasks
- Bachelor's degree in Computer Science, Engineering
- Eligible to obtain a clearance
- Experience leveraging LLMs or Computer Vision for adaptive scraping, parsing unstructured HTML, or bypassing CAPTCHAs (AI in data collection)
- Strong hands-on experience with AWS ecosystems (e.g., EKS, EC2, S3, RDS)
- Proficiency in SQL for querying, schema design, and storing structured relational data
- Experience with Redis (specifically for caching, deduplication, or as a Scrapy distributed queue back-end)
- Familiarity with Apache Kafka for real-time data streaming and decoupled pipeline architectures
- Strong foundation in Docker for local development and containerizing scraping microservices
- Experience with CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins) for automated testing and deployment of crawlers
Company Overview