Research Crawling Engineer
True Talent Solutions Partners
Salary
$160,000 to $250,000
Experience
2+ years
True Talent Solutions Partners is hiring a Research Crawling Engineer, fully remote. This role is tied to United States; check the apply page for its specific location or time-zone requirements.
The role typically suits candidates with around 2+ years of experience in engineering.
Key skills
About this role at True Talent Solutions Partners
About the Role
Join a data infrastructure team building large-scale web crawling systems and datasets for AI research and model development. This role focuses on reliable data acquisition across diverse, changing web environments.
What You'll Do
- Build and maintain web crawlers across social, travel, and multilingual websites.
- Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
- Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy sites.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at TB-to-PB scale.
- Build and maintain datasets for research and model training in collaboration with research teams.
- Monitor crawl performance, coverage, and data quality, and adjust systems as websites change.
- Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.
What We're Looking For
- At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, or 2 or more years with a PhD.
- Hands-on experience building and operating web crawlers at scale, including data pipelines handling TB-scale workloads.
- Strong proficiency in Go, Rust, Python, Java, or C++, with experience in distributed systems or parallel processing.
- Deep knowledge of HTTP, networking, and browser behavior, including experience working with anti-bot systems, rate limits, and JavaScript-heavy sites.
- Experience with headless browsers, proxy systems, IP rotation, and request orchestration is valuable.
- Familiarity with data quality evaluation, large-scale data processing, and cloud or bare-metal infrastructure.
- Experience with NLP pipelines, machine learning dataset curation, or AI research data is a plus.
- A degree in computer science, engineering, or a related technical field is helpful; a PhD is a plus.
Compensation & Benefits
Salary range: $160,000 to $250,000 annually. Visa sponsorship is not available.
Location
Remote.
Description provided by the employer’s job posting.
Apply directly through the employer
This listing links straight to True Talent Solutions Partners’s own application page.
More remote jobs in United States
View all →Rust Senior Software Engineer – Bare Metal as a Service & Datacenter Networking
Mirantis
Voluntary Benefits Representative - Remote
AO Garcia Agency
Senior Director, AI Strategy and Innovation
KIPP