Est. Budget: $20.00
We currently use Xenu Link Slueth to scrape seed domains in search of expired domains.
The software becomes unstable over 900k urls crawled.
We are looking to have a custom crawler built that can crawl, output, run api calls to url metric tools such as Majestic and Moz and then output the domains with specific metrics into downloadable files.
Here is an example http://126.96.36.199/#/dashboard
The crawler needs to be able to use seed domains that ...