Data Engineer Intern - Web Crawling

Sayari • Internship • United States, N/A, United States • 1d ago

About Sayari: Sayari is the counterparty and supply chain risk intelligence provider trusted by government agencies, multinational corporations, and financial institutions. Its intuitive network analysis platform surfaces hidden risk through integrated corporate ownership, supply chain, trade transaction and risk intelligence data from over 250 jurisdictions. Sayari is headquartered in Washington, D.C., and its solutions are used by thousands of frontline analysts in over 35 countries.Our company culture is defined by a dedication to our mission of using open data to enhance visibility into global commercial and financial networks, a passion for finding novel approaches to complex problems, and an understanding that diverse perspectives create optimal outcomes. We embrace cross-team collaboration, encourage training and learning opportunities, and reward initiative and innovation. If you like working with supportive, high-performing, and curious teams, Sayari is the place for you.Internship Description:Sayari is looking for a Data Engineer Intern specializing in web crawling to join its Data Engineering team! Sayari has developed a robust web crawling project that collects hundreds of millions of documents every year from a diverse set of sources around the world. These documents serve as source records for Sayari’s flagship graph product, which is a global network of corporate and trade entities and relationships. As a member of Sayari's data team your primary objective will be to work on maintaining and improving Sayari’s web crawling framework, with an emphasis on scalability and reliability. You will work with our Product and Software Engineering teams to ensure our crawling deployment meets product requirements and integrates efficiently with our ETL pipelines.This is a remote paid internship with work expectations being between 20-30 hours a week. Job Responsibilities: Investigate and implement web crawlers for new sourcesMaintain and improve existing crawling infrastructureImprove metrics and reporting for web crawlingHelp improve and maintain ETL processesContribute to development and design of Sayari’s data product Required Skills & Experience: Experience with PythonExperience managing web crawling at scale, any framework, Scrapy is a plusExperience working with KubernetesExperience working collaboratively with gitExperience working with selectors such as: XPath, CSS, JMESPathExperience with WebDev tools (Chrome/Firefox) Desired Skills & Experience: Experience with Apache projects such as Spark, Avro, Nifi, and AirflowExperience with datastores Postgres and/or RocksDBExperience working on a cloud platform like GCP, AWS, or AzureWorking knowledge of API frameworks, primarily RESTUnderstanding of or interest in knowledge graphsExperience with *nix environmentsExperience with reverse engineeringProficient in bypassing anti-crawling techniquesExperience with Javascript

Related Jobs

Post-Undergraduate/Graduate (Summer) Advanced Computing Operations - AI Engineering Internship

National Renewable Energy Laboratory • Internship • CO - Golden, United States • $43.70k - $69.90k / year • 21h ago

AI/Machine Learning Engineer Software Engineer AI Engineer Engineer ML Engineer early career New Grads intern Internship Internship Program

21h ago

Apply

AI Applications Engineer Intern

Hitachi • Internship • (HAL) Santa Clara, United States • 1d ago

AI/Machine Learning Engineer Software Engineer AI Engineer Engineer ML Engineer early career New Grads intern Internship Internship Program

1d ago

Apply

Summer Intern – Data Analytics & Strategy (DAS)

EagleBank • Internship • Bethesda, MD, United States • 1d ago

AI/Machine Learning Engineer early career New Grads intern Internship Internship Program

1d ago

Apply

Intern, Sales Operations

Presidio • Internship • Hauppauge, New York, United States • 1d ago

AI/Machine Learning Engineer early career New Grads intern Internship Internship Program

1d ago

Apply

Portfolio Analytics and Operational Intelligence Intern

Recursion • Internship • Salt Lake City, Utah • 1d ago

AI/Machine Learning Engineer early career New Grads intern Internship Internship Program

1d ago

Apply

Solutions/Technology Intern

RIVA Solutions, Inc. • Internship • Reston, Virginia, United States • 2d ago

AI/Machine Learning Engineer early career New Grads intern Internship Internship Program

2d ago

Apply

Intern, Data Engineer

AI/Machine Learning Engineer early career New Grads intern Internship Internship Program

2d ago

Apply

Machine Learning Engineer Intern (LLM Risk Surface) - 2025 Summer (PhD )

TikTok • Internship • San Jose, California, United States • 2d ago

AI/Machine Learning Engineer Software Engineer AI Engineer Engineer ML Engineer early career New Grads intern Internship Internship Program

2d ago

Apply

Computational Biology Intern

eGenesis, Inc. • Internship • Cambridge, MA • 2d ago

AI/Machine Learning Engineer early career New Grads intern Internship Internship Program

2d ago

Apply

Revenue Management Intern

StockX • Internship • Detroit, MI • 2d ago

AI/Machine Learning Engineer early career New Grads intern Internship Internship Program

2d ago

Apply