Looking for a web crawler program that will crawl a directory and export specific data on each page into an excel sheet.
The crawler should be written in VB or can be code within a Excel Spreadsheet (VBA) I am flexible as long as the data is retrieved
The site is very easily navigated with static layouts of infomation.
I will send you the site in PM and further information. I will need to the category, sub category and 2nd sub-category (if available) web page URL and some misc data on each page. This should crawl with some random delays built in so as not to get the IP blocked. In other words I do not need all the info crawled and extracted at blazing speeds. This can be a longer term extraction project (over 30 days) if needed.
I had a crawler once that ran at blazing fast speeds and within 3 hours I was banned so I know that these delays are very important.
There can be no hidden coding and all coding will be open for me to view. This is a private project just for me and will not be for resell.
All coding should have good comments so I can observe what is going on.
I would also appreciate that this project not be added to your online portfolio
If you are interested let me know any question you may have and I will send you a .pdf with further details. This should be a very easy project for somebody with web crawling experience. If this is your first VB project then this probably is not the job for you.
I am seeking a fixed price agreement. My budget is just a placeholder. Let me know your thoughts after you review everything and are comfortable with what is needed and then we can talk about the price.