Less than 1 week –
10-30 hrs/week –
I need someone to create a Scrapy program to crawl all of the urls on a given website up to X levels deep (internal links only) and return a list of unique urls.
They will also need to provide a basic web API for creating new crawl jobs. Just something simple like: ..../crawl?url=somedomain.com&depth=4
They will need to provide an API interface for returning and purging the results: .../results?url=somedomain.com
They will also need ...