Description
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardless of language, proactive risk identification, 360-degree insights, high-speed automation, and seamless integration into existing systems. Babel Street empowers government and commercial organizations to transform high-stakes identity and risk operations into a strategic advantage. The actionable insights we deliver safeguard lives and protect critical assets around the world. Babel Street is headquartered in Reston, Virginia, with regional offices in Boston, MA and Cleveland, OH, and international offices in Australia, Canada, Israel, Japan, and the U.K. For more information, visit www.babelstreet.com.
ROLE SUMMARY:
The primary purpose of this Software Engineer III position is to architect, develop, and maintain advanced automated data extraction systems (spiders) that harvest critical business intelligence from complex web environments. This role ensures the continuous, reliable flow of high-quality data into our internal databases by engineering solutions to overcome sophisticated technical barriers and anti-bot security measures. Candidates near our Somerville, MA or Reston, VA offices are preferred but remote work it potential for the ideal candidate.
ROLE FOCUS;
As a Software Engineer III specializing in data extraction, you will be responsible for the end-to-end lifecycle of web-based data collection. This includes designing scalable crawling architectures, reverse-engineering web applications to identify data points, and implementing evasion techniques to bypass IP rate-limiting and bot detection. You will also manage the storage and integrity of this data using advanced SQL and relational database management.
We are looking for a Senior Python Engineer with a "hacker" mindset to join our team as a Software Engineer III. This role is dedicated to large-scale web scraping and data harvesting. If you have deep experience with Scrapy or Playwright, know how to defeat Cloudflare or DataDome, and can write high-performance SQL to manage millions of records, we want to hear from you. This is a specialized role for an engineer who enjoys reverse-engineering the web to unlock data.
The primary purpose of this Software Engineer III position is to architect, develop, and maintain advanced automated data extraction systems (spiders) that harvest critical business intelligence from complex web environments. This role ensures the continuous, reliable flow of high-quality data into our internal databases by engineering solutions to overcome sophisticated technical barriers and anti-bot security measures.
Responsibilities:
- Spider Development: Design and deploy robust, distributed spiders and crawlers to extract data from a variety of web architectures (SPAs, SSR, etc.).
- Bot Evasion Engineering: Research and implement strategies to bypass anti-scraping technologies, including proxy rotation, browser fingerprinting, and CAPTCHA solving.
- Database Management: Create and optimize SQL schemas for large-scale data storage and perform complex data transformations and validation.
- System Maintenance: Proactively monitor the health of extraction agents and refactor code quickly in response to target website updates or layout changes.
- Performance Optimization: Utilize asynchronous Python programming to maximize the throughput and efficiency of data collection pipelines.
Requirements:
- Advanced Python: Mastery of Python 3.x with deep experience in extraction frameworks (Scrapy, Playwright, Selenium, or Puppeteer).
- Technical Resilience: Proven ability to bypass high-level bot detection (e.g., Cloudflare, Akamai, or PerimeterX).
- Database Mastery: Expert-level SQL skills and experience managing relational databases like PostgreSQL or MySQL.
- Network Proficiency: Expert understanding of HTTP/S, TCP/IP, TLS fingerprinting, and browser-header manipulation.
- Problem Solving: A specialized ability to reverse-engineer JavaScript-heavy websites and hidden API endpoints.
- Able to write, debug, and deploy complex Python code in a distributed environment.
- Must be able to analyze and interpret complex web structures and network traffic using browser developer tools.
- Ability to design and maintain relational database tables containing millions of rows.
- Able to pivot and respond quickly to technical "break-fixes" to ensure data continuity for the business.
- Collaboration with data analysts to define and validate data requirements and output formats.
Education & Experience:
- Bachelor’s degree in Computer Science, Information Systems, or a related field (or equivalent professional experience).
- Minimum of 5+ years of experience in Software Engineering, with at least 2–3 years focused specifically on large-scale web scraping or data extraction.
Benefits at Babel Street (just to name a few...)
- Health Benefits: Babel Street covers 85-100% monthly premium costs for Medical, Dental, Vision, Life & Disability insurances – for you and your family!
- Retirement Plans: Babel Street offers both a Traditional and Roth 401(K) with a very competitive match.
- Unlimited Flexible Leave: We trust our employees to manage their own time and balance their personal and work lives.
- Holidays: Babel Street provides employees with 12 paid Federal Holidays
- Tuition Reimbursement: We are committed to investing in our employees. One way we do that is with our Tuition Reimbursement Program for continuing education.
Babel Street is an equal opportunity/affirmative action employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law. Further, Babel Street will not discriminate against applicants for inquiring about, discussing or disclosing their pay or, in certain circumstances, the pay of their co‐worker, Pay Transparency Nondiscrimination. In addition, Babel Street's policy is to provide reasonable accommodation to qualified employees who have protected disabilities to the extent required by applicable laws, regulations and ordinances where a particular employee works. Upon request, we will provide you with more information about such accommodations.
Similar jobs
Est. 124,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 105,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 110,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 127,500 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 85,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 64,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 155,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 124,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 122,500 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 152,900 USD
Meet the Team: Torc's BI team partners across the business to bring visibility and reporting that help leadership and teams make faster, better-informed decisions. This is an exciting opportunity to help define how data…
About Tekmetric Tekmetric is the all-in-one, cloud-based software helping auto repair shops run smarter, grow faster, and serve customers better. Built by a shop owner and shaped by the needs of the industry, Tekmetric b…
Est. 124,000 USD
Strength in Trust OneTrust’s mission is to enable innovation through the responsible use of data and AI. We believe that ensuring data is trusted shouldn’t slow teams down—it should accelerate what’s possible. This led u…
Est. 137,500 USD
This position supports a federal client and requires US citizenship for Public Trust eligibility Attain Talent is searching for talented data engineers for our GovCon client. This is a full time position with benefits, a…
Est. 140,000 USD
This Is the Place to Be:Connecting Futures Now! DHI Group, Inc. is the parent company of career marketplaces, Dice and ClearanceJobs. We connect candidates with career advice, resources and ultimately a dream job. At DH…
Est. 120,000 USD
This Is the Place to Be:Connecting Futures Now! DHI Group, Inc. is the parent company of career marketplaces, Dice and ClearanceJobs. We connect candidates with career advice, resources and ultimately a dream job. At DH…
Est. 124,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 152,900 USD
About the Role As a Software Engineer II on the Build Tools team at Torc Robotics, you will serve as the automation backbone for our entire engineering organization. In this highly impactful DevOps/Full-Stack role, you w…
Est. 180,000 USD
ZoomInfo is where careers accelerate. We move fast, think boldly, and empower you to do the best work of your life. You’ll be surrounded by teammates who care deeply, challenge each other, and celebrate wins. With tools…
Est. 144,000 USD
About SHEIN SHEIN is a global online fashion and lifestyle retailer, offering SHEIN branded apparel and products from a global network of vendors, all at affordable prices. Headquartered in Singapore, SHEIN remains commi…
Est. 146,000 USD
Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by mor…
Est. 215,000 USD
At Forge, we know our team is our greatest asset. As technology innovators in the private market, our vision is to deliver a richer future for everyone. We live that vision through our values of being bold, accountable,…
Est. 121,750 USD
Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by mor…
Est. 147,500 USD
Capital Technology Group provides expert consulting services software development, digital transformation, human-centered design, data analytics and visualization, and cybersecurity. Our multidisciplinary teams use agile…
Est. 126,000 USD
Founded in 2004, NetBrain is the leader in no-code network automation. Its ground-breaking Next-Gen platform provides IT operations teams with the ability to scale their hybrid multi-cloud connected networks by automatin…
Est. 170,000 USD
This Is the Place to Be:Connecting Futures Now! DHI Group, Inc. is the parent company of career marketplaces, Dice and ClearanceJobs. We connect candidates with career advice, resources and ultimately a dream job. At DH…
Est. 120,000 USD
Babel Street is the trusted technology partner for the world’s most advanced identity intelligence and risk operations. We deliver advanced AI and data analytics solutions providing unmatched, analysis-ready data regardl…
Est. 176,900 USD
About the Role: We are seeking a Senior Software Engineer to join our Platform Engineering team. You will build and evolve the internal platforms and services that empower engineers across the company to automate and acc…
Est. 92,500 USD
(ID: 2026-3389) Axle is a bioscience and information technology company that offers advancements in translational research, biomedical informatics, and data science applications to research centers and healthcare organiz…
Est. 144,000 USD
ZoomInfo is where careers accelerate. We move fast, think boldly, and empower you to do the best work of your life. You’ll be surrounded by teammates who care deeply, challenge each other, and celebrate wins. With tools…
Est. 222,500 USD
Space is a warfighting domain. True Anomaly seeks those with the talent and ambition to build the technology that secures it. OUR MISSION True Anomaly delivers decisive capabilities for space superiority. We build autono…