Research off the shelf solutions for a back end batch system (scraping application)
- Status: Closed
- Prize: $170
- Entries Received: 2
- Winner: TheGuyver2040
Contest Brief
The task is to research and suggest possible off the shelf solutions for a back end batch system (scraping application) that fulfill the following requirements:
Business Objectives:
Get competitors product information for branded shoes on a daily basis. Information like Product name, Product colorname, other product attributes example materials, list of competitors, Competitor prices (Black and discounted prices)).
Proposed Work Flow:
The system fetches a product list from our system with EAN codes (for identification). The system then uses the EAN in google shopping to fetch product information and prices per competitor.
Required Features:
Scraping engine to handle over 15.000 requests per day. The normal request per day will be 50-500. The information should be structured on model level and SKU / EAN level (ie we send over/fetch one size/EAN to scrape).
Inputs:
There will be a product feed into the system from our product system. Alternativly we want to be able to fetch the data through a feed / API.
Dependencies:
The system should have an easy way to export / import information preferably using some API / feed.
Integrations:
There should be an API available. To fetch the scraped products. And possibly what products of interest to scrape.
Server / Hosting:
AWS on a Kubernetes server, if hosted on our servers. Otherwise software as a service is fine.
User Count:
Performance is hard to estimate but roughly 50-500 daily requests but up to 15.000 when going through our total range of products.
Deliverable:
A summary of the system you have found with a short explanation of its functionalities and how it can be used to fulfil our requirements
Recommended Skills
Employer Feedback
“It was great! ”
Footway, Sweden.
Top entries from this contest
-
TheGuyver2040 United States
-
talon2 Bolivia
Public Clarification Board
How to get started with contests
-
Post Your Contest Quick and easy
-
Get Tons of Entries From around the world
-
Award the best entry Download the files - Easy!