I want you to develop a web crawler for me, the crawler should visit the destination site daily and get the new content + store to the database + upload to my site. I can input a list of proxy and the crawler can select random from the list and using the proxy to crawl the destination site.
Please give me an advise:
- Which technology will you use?
- How long will it take?
- The estimate cost?