Find Jobs
Hire Freelancers

Build a Python Based Web Scraping Application

$750-1500 USD

Closed
Posted about 9 years ago

$750-1500 USD

Paid on delivery
I'm looking for an expert in Python and Web Scraping to take on a very specific project. The ultimate project will actually NOT be Django, but rather using a microframework called Flask (If you have Django experience, Flask is about a one day learning experience). For Python, we prefer people with Scrapy experience, and would like to stick with Scrapy and Flask. Essentially, you'll be building a spider that will hit a specific website, and your job is to crawl approximately 20K pages. The first time, we do a full scrape, if the page exists in the Database then you'll check for changes and updates. Accuracy is 100% vital. All data should be collected using JSON, the spider will of course run server-side, but there needs to be a corresponding web interface for manually performing CRUD operations. You should also note, that we expect the collected data to be stored in a MySQL database (which should be appropriately indexed and normalized). Finally, it is an absolute must that you have GIT experience. Not just doing code checkins 1x per day, but routinely doing pushes to our repository. If you don't understand GIT and how team development works with a DVCS, please don't waste the time bidding. Also, if you don't get anything herein, again don't waste our time. If you send me 20 useless website URLS I'll passover you entirely. I'm a little jaded by people that keep submitting the same "blanket response" to every project. You have enough information to accurately bid within reason, but ask questions if you don't.
Project ID: 7108070

About the project

24 proposals
Remote project
Active 9 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
24 freelancers are bidding on average $1,263 USD for this job
User Avatar
Dear Sir, I'm a seasoned web scraper. Have written hundreds of web scrapers which scrape millions of pages each day. I'm ready to fulfill your requirement. Please give me the opportunity to do the work. Warm Regards, Debdulal Roy Proshanta
$1,250 USD in 20 days
5.0 (89 reviews)
7.6
7.6
User Avatar
Hi, I can do this for you. Please send a massage in the PMB for details.......Best Regards flashsaiful
$1,250 USD in 20 days
4.8 (134 reviews)
6.7
6.7
User Avatar
A proposal has not yet been provided
$1,250 USD in 20 days
5.0 (56 reviews)
6.4
6.4
User Avatar
A proposal has not yet been provided
$750 USD in 20 days
5.0 (108 reviews)
5.9
5.9
User Avatar
hello i have good experience in writing crawlers in python . And also with scrapy and crawlera . i can work with django,flask and git . thanks
$1,400 USD in 10 days
4.8 (84 reviews)
6.2
6.2
User Avatar
Hi, I have experience working with flask, scrapy and mysql and GIT. I'm also proficient in regular expressions and general web development which come in handy for web scraping. I've done web scraping projects in the past and have always been able to retrieve the required results. I've been working as a freelance software developer for over a year specializing in python and web development. I have an engineering degree and excellent problem solving abilities. I pride myself if delivering high quality maintainable code and in working with my clients to ensure they are 100% satisfied with the products I develop. I would be happy to discuss the project in more detail and learn more about your requirements and what exactly you want scraped. If your interested in my services feel free to contact me at any time and I'll get back to you as soon as possible. Thanks, Joshua Cruikshank
$1,250 USD in 20 days
5.0 (13 reviews)
5.3
5.3
User Avatar
Hello, sir. My name is Alex, i'm professional python developer. About my bid first - i'm just making portfolio on freelancer, so my rates are low now, later it will cost twice =) I like your project and i'm good candidate because: 1. I have 5 years of web development experience, last 2 years i was working as system architect. 2. I was working with flask, django and scrapy. 3. I have experience about building muthithreading appilcation making mass page downloads (i've made 1000000 google result downloads in 3 days). 4. I'm damn good. I have some questions we can talk about if you like my bid. Looking forward to work with you!
$833 USD in 15 days
5.0 (3 reviews)
4.9
4.9
User Avatar
Hello, I have experience with Flask/SQLAlchemy, will use Scrapy to do the crawling. My solution will keep the historical downloaded data, and will provide a web interface to view it. Updates will be done using HTTP HEAD requests (checking Date and If-Modified-Since headers, to lower total traffic). Not sure why you need JSON here (do you want to parse the content in some way?), I'd rather store whole pages as strings in the database.
$1,000 USD in 10 days
5.0 (1 review)
2.7
2.7
User Avatar
Hello Sir, I can develop your web scraper, I have already developed this type of work using BeautifulSoup. It's nota problem for me to use Scrapy. For example, I get the entire database of citizens in France using my web scraper (with address, phone number, name, etc), 60 millions entries, The accuracy is also my goal. I work with passion, that is why I work the day, the night even the weekend (and I'm not a freelancer flooder, with blanket responses) Best Regards,
$1,250 USD in 25 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I have substantial experience developing services and applications in Python, Django and Flask which would be ideal for this project. I've around 5 years commercial experience with Python, and over the past year have used Flask to develop complete API's from scratch, which persisted using MySQL or mongodb. I also use git on a daily basis, and believe regular commiting is of the utmost importance. Please let me know if you have any questions about my bid. Thanks, Tim
$1,111 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
This is a expert python developer with experience in Django and Scrapy. Hope to establish cooperation relationship. Regards.
$1,250 USD in 15 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I used Python and Scrapy to extract trading card data from four different HTML-based sites. The data was extracted in JSON format, and the script was manually run from the server side, with no available CRUD web interface aside from the default Django admin site used alongside the project. If I may ask, what site will we be extracting data from? Thank you!
$1,400 USD in 15 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of JAPAN
Yomitan-son, Japan
4.9
52
Payment method verified
Member since Dec 9, 2012

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.