Use Scrapy to get content

Use Scrapy to get content


Job Description

I have a spider written in Scrapy to try and launch a search on the Urbansitter site. Right now, I cannot start a search to get a list of sitters because I think something is wrong with the authentication flow I have with Urbansitter.
Urbansitter uses Facebook to log people in to the site and while it looks like I can log in through Facebook ok, I end up getting a 403 error that prevents me from getting to the page where I can start a search.

I will supply you with the code for the Scrapy spider I have written, the log file that shows what Scrapy is doing, and the raw HTTP request and responses from a Firefox browser that illustrates what a normal browser communicates back and forth with to Urbansitter.

Just to give you an idea of what the request flow looks like right now, the chain looks like this:

start_requests -> visit_home_page -> process_fb_login -> post_fb_login -> post_dashboard_post -> visited_dashboard_parent_page -> posted_dpadp -> process_srp_quick

You will need your own Facebook account to log in to Urbansitter and to test with.