Make us a crawler!
Closed - This job posting has been filled and work has been completed.
We need a program to crawl target websites (entered by a user) and produce (and database) a list of all URLs on the target websites' homepage, and any pages linked from that page. Optimally, this bot would also only capture the links that link to pdf files (or some other user designated file extensions). That's it!
Please respond to this post with answers to the following questions, to ensure applicants have read the post and have sufficient English skills. All others may be ignored.
1. What is your favorite color?
2. What is your strongest programming language?
3. What is your weakest language (that you are familiar with)?
We're on an asp.net platform but that shouldn't be a problem (as this should be able to be adapted to run independently).
The budget is just a suggestion - feel free to suggest otherwise, if you can reasonably justify it.