Est. Budget: $300.00
I need a Scrapy Crawler that imports about 100 Start-URLs / Domains and crawler each of them. The list of domains are attached.
The crawler should handle filter files. One file for allowed URL-Pattern(regex) and another for forbidden URL-Patterns(regex).
If the filter matchs the forbidden rules thant this url shouldnt be saved in the reports.
The crawler should generate two reports in CSV: Examples are attached.
One report should have this columns and get this results:
URL / Adress, Status-Code ...