AIMLEAP is Hiring Web Scraping - Scrapy Expert - WFH

Web Scraping - Scrapy Expert
Experience: 2+ Years
Location: Work From Home/India
Salary: 3 to 6 INR LPA
No of positions: 2 No
Industry: Automation/Data Processing
Notice Period: Immediate
 

Job Description :

We are looking to add a talented and motivated Web Scraping Expert to our team. Using the high-performance Python framework Scrapy, you will be in charge of creating and managing reliable web scraping pipelines. You will use your knowledge of Python and web scraping methods to reliably and scalable retrieve useful data from a variety of online sources.

 

Responsibilities :
 

1. Design, develop, and maintain web scraping projects using Scrapy, including:
2. Crawling websites to extract specific data points, implementing efficient crawling strategies to handle pagination, dynamic content, and complex website structures, Employing data extraction techniques with CSS selectors and XPath, Processing and cleaning scraped data using Python libraries (e.g., Pandas, NumPy), Storing extracted data in appropriate formats (e.g., CSV, JSON, databases).
3. Collaborate with data engineers and analysts to identify data needs and define scraping requirements.
4. Write well-documented, maintainable, and efficient Scrapy code
5. Integrate scraping pipelines with other Python frameworks and tools as needed
6. Stay up-to-date with the latest web scraping trends and best practices
7. Troubleshoot and address challenges related to rate limiting, authentication, and website changes
8. Implement caching strategies for efficient data retrieval and reduced website load
9. Consider potential ethical and legal implications of web scraping

Requirements :
 

1. Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent work experience).
2. Proven experience in web scraping, data extraction, and automation using tools like BeautifulSoup, Scrapy, Selenium, etc.
3. Proficiency in programming languages such as Python, JavaScript, or Ruby, with a focus on writing clean, maintainable code.
4. Strong understanding of web technologies (HTML, CSS, XPath, DOM manipulation, AJAX, etc.) and web protocols (HTTP, HTTPS).
5. Familiarity with database systems and data storage solutions, including SQL and NoSQL databases.
6. Excellent problem-solving skills and attention to detail, with the ability to analyze complex data structures and troubleshoot issues effectively.
7. Good communication and teamwork skills, with the ability to collaborate with stakeholders from diverse backgrounds.