KnowFirst is a data intelligence company specialising in large-scale web data acquisition and analysis. We build robust, production-grade data collection systems that power real-world intelligence products across modern web platforms.
Key Responsibilities
* Design, develop, and maintain Python-based web scrapers
* Maintain and improve scraping systems to handle routine target-site changes
* Containerise scraping services using Docker
* Deploy and operate scraping workloads on AWS
* Monitor scraper performance, reliability, and data quality
* Troubleshoot common failures and performance issues
Required Skills & Experience
* Commercial experience using Python in production environments
* Experience building and maintaining web scrapers
* Practical experience with Playwright and/or Selenium
* Experience containerising applications with Docker
* Working knowledge of AWS services (e.g. EC2, ECS, Lambda, S3, SQS)
* Understanding of:
* Proxies, rate limiting, and request optimisation fundamentals
* Ability to write clear documentation and communicate effectively within a team
* Experience collaborating using tools such as Slack, Google Workspace, Jira, and Confluence
Desirable (Nice to Have)
* Exposure to sites protected by Cloudflare or similar WAFs
* Familiarity with CAPTCHA and bot-detection challenges
* Experience with higher-volume or distributed scraping systems
* Familiarity with monitoring and logging using AWS CloudWatch
What KnowFirst / Deep Future Offers
* Ownership of meaningful parts of core data acquisition systems
* Opportunity to work on real-world data intelligence problems
* Support and mentorship from experienced engineers
* Flexible working arrangements
* Competitive remuneration aligned with experience
#J-18808-Ljbffr