Scrapy expert needed

Scrapy expert needed


Job Description

Scrapy expert needed to develop script to scrape products from webshops. The script needs to be able to fetch product image, product title, product description, product URL and price.

The information that is scraped shall be stored into a MySQL database and used on a SocialEngine website. The database structure has already been developed and we already have the functionality in place to manually create products. What I want is to automate the process filling up the products by scraping the products from every webshop that has also a presence on our website.

More specifically what I am looking for is:

- Most webshops have a larger image for the product if you click the thumbnail. Please scrape the big image

- For each product, scrape the following:
o The category that the product is within
o The (big) product image
o The product title
o The product price
o The product description
o Where variations are given in a combo-box scrape the values and store as a comma separated list (typically color)
o Where variations of a product are given with a unique price for each variation type, create a new product for each variation and use the product name as the product title

I will provide you with webshops that the script shall be tailored to scrape from, but the script also needs to be written so it is possible to easily add new web shops that shall be scraped.

Update: I have added a pdf with three shops that I would like to have scraped. It shows the categories the script need to traverse, and also the information that is to be scraped for each product. As you can see the sites are in Norwegian, but I hope the screenshots will give you enough explanation to get an understand what I want.

Please only respond to this job if you have done similar work before.

Open Attachment