Build a large Image Scraper

Completed Posted Nov 3, 2015 Paid on delivery
Completed Paid on delivery

Stagecoach Software Pty Ltd is developing a website to contain statistical profiles of a particular industry in Australia. Our stack is Ruby on Rails 4.2, Ruby 2.2, Passenger 4 and MySQL 5.5. We run a 3 server configuration on Engine Yard with dedicated instances for our db, web app and utilities. Our database receives about 30,000 new rows per day through scraping operations (22 different scrapers using nokogiri). Presently our app instance only provides admin operations; the consumer front end is under development.

Currently our team comprises 3 backend ruby developers and 2 front end developers.

We have about 16 M products in our database. Each product has a distinct id number. Our front end website must be able to display an image of each product.

There is a website that contains images of a larger range of products but which includes the ones in our database. The ones that match our database are identifiable on that site based on the descriptions in our database. We need copies of these images downloaded to our AWS S3 bucket. Each image must be given a filename that connects it to the relevant product id in our database. The image table in our database must be updated to reflect the details necessary to allow the images to be uploaded to our website. In other words a scraper is needed that will search the image website based on the product descriptions in our database, download matching images and update our database to record the details of the images downloaded and facilitate retrieval.

Development and testing must be done locally. Then code must be uploaded to our github repo and approved by the lead developer before deployment.

This is one of a pipeline of projects in this enterprise. Further ruby projects will be available if the above tasks are carried out well. Applications from a freelancer experienced with Engine Yard are welcomed.

MySQL Ruby Web Scraping

Project ID: #8825124

About the project

9 proposals Remote project Active Dec 8, 2015

Awarded to:

danielloureiro

A proposal has not yet been provided

$555 AUD in 10 days
(3 Reviews)
3.8

9 freelancers are bidding on average $560 for this job

mantislin

Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi

$568 AUD in 6 days
(136 Reviews)
6.7
Verz1Lka

Hello! I'm web scraping expert. I use python scrapy framework. My scripts can run on windows or linux, but linux is preferably. I can schedule scripts on server if it is required. I have a lot of finish projects (go More

$500 AUD in 3 days
(45 Reviews)
5.7
mirniyazuddin92

Dear Sir/Ma'am, Price and duration will vary depending on actual project. I am a Web research and Data Entry expert. I checked and understood your requirements. I can handle this job very well to your apprec More

$500 AUD in 10 days
(22 Reviews)
4.7
ayushchd

Hey, Could you share the source of the data / images you want scraped?

$700 AUD in 3 days
(28 Reviews)
4.6
alextakitani

Hello! I'm a senior fullstack developer, 18+ yrs of experience, 6 yrs working with Rails. I work daily with Rails, and have made many scraping jobs. I have some questions: Will be some kind of throttling be m More

$555 AUD in 10 days
(5 Reviews)
3.4
DB1Devs

Hello, My name is Dmytro Bibik, I am a PM of an outsourcing software and web development company “DB Devs” from Kharkiv, Ukraine. Your task seems to be utmost coincident with our focus of expertise. We can build More

$555 AUD in 10 days
(0 Reviews)
0.0