Data Engineer – Build Scalable Data Collection Pipelines

About the role

Dataweav is looking for a Data Engineer to own and scale automated data collection systems. You will design extraction logic, harden pipelines against site changes, normalize data into clean schemas, and maintain rigorous QA. The team works AI‑first, using Claude Code and other assistants daily.

Key responsibilities

• Design and build data collection workflows using Python, Playwright, and HTTP APIs.
• Own end‑to‑end pipelines, from request initiation to QA sign‑off.
• Reverse‑engineer site behavior by reading network traffic, replicating API calls, handling authentication, pagination, and anti‑bot measures.
• Diagnose and fix failures quickly, including rate limits, structural changes, and corrupted data.
• Define and maintain standardized data schemas with validation.
• Develop and improve QA checks to catch regressions before release.
• Mentor interns through code reviews and technical guidance.
• Use AI‑assisted development (Claude Code) as a core part of daily workflow.

Required profile

• 1‑2 years of hands‑on experience in data extraction, web scraping, frontend or backend development.
• Strong proficiency in Python and solid understanding of HTML, CSS, JavaScript, and browser internals.
• Ability to read network traffic in browser DevTools and understand HTTP headers, cookies, and auth flows.
• Experience building reliable pipelines at scale, not just one‑off scripts.
• Self‑directed, ownership mindset, and clear written communication in English.

Required skills

• Python
• Playwright
• BeautifulSoup or lxml
• Git
• SQLite
• Pydantic
• pytest
• HTML, CSS, JavaScript
• HTTP APIs and network traffic analysis
• Proxy and captcha handling (anti‑bot evasion)
• Claude Code (AI coding assistant)