landing page crawler and index
Closed - This job posting has been filled and work has been completed.
I need to build a web app that scrapes google, indexes the pages linked from that page, and puts them into a database that parses a bunch of different details about the indexed pages.
The app needs to do the following.
1, query google for a keyword.
2. scrape the adwords ads for that keyword. (the test, display url, destination url. and any ad extensions like a phone number or address.
3. open the destination url that each adwords ad opens (i.e. clicks on each ad for that keyword to open the page it points to.)
4. crawl the landing page for each ad, and places all the scraped content into a database.
5. parse the database to analyze all sorts of information found in those pages.
6. visually display the results in a user friendly way.
If you can scrape google, crawl landing pages, build a database, and use that data to answer a bunch of questions or fields (i.e. is the landing page unique or the websites homepage.) etc... Than please apply for this job.