I need a data extractor built for a password protected site (OpenTable). While there is very little data to be extracted (6 or 7 fields), the scraper needs to run every 5-10 minutes, cycle through 150-250 pages, and be *completely* anonymous.
If you've worked with the OpenTable before (which is preferred), you know they ban bots and web crawlers very quickly. Also, the OpenTable portals - Restaurant Center and Remote Manager - can load slowly, so the scraper needs to handle exceptions if the site is dragging.
As the data is extracted, I need it loaded into a SQL 2012 database on my host. So the scraper will essentially be doing the entire ETL process directly to the database tables.