Python code to automate Web-scraping for images and articles with Google search
$750-1500 SGD
Closed
Posted over 3 years ago
$750-1500 SGD
Paid on delivery
Please review the project description carefully before bidding. In your proposal, please mention how would you approach this task and what limitations/challenges do you anticipate with this project?
We are seeking the proposals from freelancers to develop a python code and demonstrate its
functionality to carry out the following web-scraping tasks.
1. Web-scraping for images: The python module needs to be able to scrape the web for
specific images and extract all the associated metadata. The images of interest would be
mainly related to rural and agricultural background. For example, if an image of a ‘tractor in
the farm’ is searched, then the images and the metadata for various tractor images should
be fetched and stored.
2. Web-scraping for articles: The expert blogs and articles need to be searched and
downloaded. The articles written in the domains of farming and rural life are of interest.
3. Google search engine: The articles and images are to be searched mostly using Google
engine. Other good search engines along with google will be an advantage.
4. The code must be deployable on the AWS: The deployment must be with the docker
containers and the scraped data must be stored in the S3 buckets. The complete devops
pipeline must be readily usable.
5. Scalable solution: The deployment needs to be horizontally scalable.
6. Demo with personal account: The demo of the code execution along with the CI/CD pipeline
must be demonstrated using the developers/company’s personal account, we will be
testing it separately on our platform.
7. We are not looking for any UI development, basic text entry even with command line is
acceptable.