Build a large Image Scraper
$250-750 AUD
Paid on delivery
Stagecoach Software Pty Ltd is developing a website to contain statistical profiles of a particular industry in Australia. Our stack is Ruby on Rails 4.2, Ruby 2.2, Passenger 4 and MySQL 5.5. We run a 3 server configuration on Engine Yard with dedicated instances for our db, web app and utilities. Our database receives about 30,000 new rows per day through scraping operations (22 different scrapers using nokogiri). Presently our app instance only provides admin operations; the consumer front end is under development.
Currently our team comprises 3 backend ruby developers and 2 front end developers.
We have about 16 M products in our database. Each product has a distinct id number. Our front end website must be able to display an image of each product.
There is a website that contains images of a larger range of products but which includes the ones in our database. The ones that match our database are identifiable on that site based on the descriptions in our database. We need copies of these images downloaded to our AWS S3 bucket. Each image must be given a filename that connects it to the relevant product id in our database. The image table in our database must be updated to reflect the details necessary to allow the images to be uploaded to our website. In other words a scraper is needed that will search the image website based on the product descriptions in our database, download matching images and update our database to record the details of the images downloaded and facilitate retrieval.
Development and testing must be done locally. Then code must be uploaded to our github repo and approved by the lead developer before deployment.
This is one of a pipeline of projects in this enterprise. Further ruby projects will be available if the above tasks are carried out well. Applications from a freelancer experienced with Engine Yard are welcomed.
Project ID: #8825124
About the project
Awarded to:
9 freelancers are bidding on average $560 for this job
Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi
Dear Sir/Ma'am, Price and duration will vary depending on actual project. I am a Web research and Data Entry expert. I checked and understood your requirements. I can handle this job very well to your apprec More
Hello! I'm a senior fullstack developer, 18+ yrs of experience, 6 yrs working with Rails. I work daily with Rails, and have made many scraping jobs. I have some questions: Will be some kind of throttling be m More