Tech Lead - Web Scraping and Crawling

KEY RESPONSIBILITIES:
Manage individual projects priorities, deadlines, and deliverables
Gather and process raw data at scale (including writing scripts, web scraping, calling/create APIs, etc.) from the web / internet
Develop frameworks for automating and maintaining constant flow of data from multiple sources
Identify, analysis, design, and implement internal process improvements
Design and implement tooling upgrades to increase stability and data quality
Help team to fix issues that occur in test and production environments
Automate software development processes, including build, deploy, and test
Mange and guide the team members

REQUIRED QUALIFICATIONS:
4+ years of web crawling/ scraping experience is a must
Strong knowledge of scraping frameworks such as Scrapy, Beautiful Soup, HTQL, Jsoup, Web-Harvest and others
Excellent verbal, written, and interpersonal communication skills in English
Good to have Experience of complex crawling (like captcha, Mobile OTP based crawling, bypassing proxy)
Sound Knowledge in Bot Management Techniques
Experience in various data extraction methods (like data extraction from PDF Files, web pages, etc)
Good understanding of HTML DOM, CSS, Javascript, and RESTful web services
Good to have understanding of AWS
Experience with Linux
Experience with Java / Python