Real-Time Scraping App (Long Term Contract)
Closed - This job posting has been filled.
We're looking for a talented Ruby/Rails programmer to build a rails application from scratch. The app will scrape and aggregate results in real-time from 8 different sources(referral websites) and store & display those results with referral linking back to the sources. We have a BRD document with more details we can share. Here's a brief overview, this is a long term engagement and we're interviewing to begin immediately.
Proposed Technology Stack:
- Typhoeus (invoke parallel web requests)
- Nokogiri (used for scraping data from typhoeus responses)
- Rails (for webapp)
- Heroku (web host)
Phase 1: Website and Crawlers (3 weeks)
- Write 8 crawlers using rake tasks to scrape, collect and store company data
in db from up to 8 referral sources.
- Create a Rails app to query the db for this data
- When users search from our website for Companys or product-types, Rails should first preform a real time scrape against all 8 referrals again and should match the results against our pre-populated db, and only return the results that have a match grouped by Company having many referrals.
Phase 2: Additional Crawlers (4 weeks ongoing)
- Write additional crawlers to search our db and find Companys which have missing information and then populate that Company with the info needed (store schedules, ratings and reviews and so on, we have a list).
Code should be clean, well-thought out making use of new types of classes and modules and be extensible.