Automate monthly CSV file extraction w/ CAPTCHA, write entries to PostgreSQL database + log

Completed Posted 7 years ago Paid on delivery
Completed Paid on delivery

Develop a code that:

1. Downloads all available links at both tabs (IMPORTACAO/EXPORTACAO)

[login to view URL]

attention to CAPTCHA necessary at every link download

2. Unzip and save the CSV files into a PostgreSQL database. Entries are separated by "@" separator.

Problems might occur when some fields have "@" in the text, so this error also be handled.

3. Every month, check for new download links. Download new files and add to database.

4. All steps should be logged in a log table.

Expected result is:

1. Populated PostrgreSQL table with all results available on IMPORTACAO tab

2. Populated PostrgreSQL table with all results available on EXPORTACAO tab

3. Populated PostrgreSQL table with all logs

4. Actual code(PHP, Ruby or Python) delivered through github that will be used to run every month to extract new data

This data is published by Customs and will be used to update www.tradeadvisor.com.br - a website to help importers/exporters check prices of similar products that have been imported/exported recently.
Our goal is to make the data more easily accessible and searchable through keywords using Elasticsearch.

PHP PostgreSQL Python Ruby on Rails Web Scraping

Project ID: #12047135

About the project

31 proposals Remote project Active 7 years ago

Awarded to:

nmarkovickv

Hi friend, I am one of the top freelancers in this category. Feel free to check my reviews for reference of my work. This looks like a tricky job, exactly the job for me :) I will be happy to help you with this More

$560 USD in 5 days
(96 Reviews)
7.1

31 freelancers are bidding on average $625 for this job

kchg

Hello, I've reviewed your project. So totally you need to update table data via CSV import as schedule. Am I right? This is TOP 5th freelancer in this site. As you see my profile(https://www.freelancer.com/u/kchg.h More

$705 USD in 10 days
(426 Reviews)
9.6
kabirchy

Hi there - My name is Khorshed. I've read your brief and can see that you’d like to automate process of data extraction and store in DB. I can help you get this done. I will develop this script in PHP and will run as a More

$675 USD in 12 days
(1190 Reviews)
9.2
nuked24

A proposal has not yet been provided

$564 USD in 25 days
(175 Reviews)
8.7
opencollar

Hi, This application will be developed using PHP. The PHP programming has all the solutions for your current problems as mentioned below. What do we need to do to captcha? do we need to stop at captcha to ask input fro More

$705 USD in 10 days
(139 Reviews)
8.7
FASTGuru

Hi Sir! I'm the expert developer. you can check my profile here https://www.freelancer.com/u/FASTGuru.html .with 5 years of experience I assure you the highest quality of the work. Once you have awarded me your project More

$529 USD in 5 days
(207 Reviews)
8.0
seaanddream

Hi, my name is Sevinc. Thank you for the invitation... I read your "Automate monthly CSV file extraction w/ CAPTCHA, write entries to PostgreSQL database + log" project descriptions carefully before bidding. I checked More

$560 USD in 10 days
(285 Reviews)
8.0
patzzVJSOFT

Hi we can automate and download the files using PHP CURL or PantomJS and CasperJS headless browser depending on the security applied to the website for bots. once we down loaded we can load the data to database running More

$564 USD in 20 days
(220 Reviews)
8.2
latatestTech

With Freelancer Preferred badge bound to give 100% Quality (Let's Chat) Hello, A great team is here for your service!! I read the project description properly, and I agreed to fulfill 100% of all your requiremen More

$823 USD in 20 days
(160 Reviews)
7.9
pointlogic

Hello..I have an experience of more than 8 years in web development and maintenance. I have in-depth knowledge of php, mysql, jquery, paypal integrations, API's, css, html, html5 and SEO. Our team is experienced, cr More

$529 USD in 10 days
(321 Reviews)
8.1
toseef3

Dear Sir, Hope you are doing well, My name is Toseef I have extensive experience in PHP, CI, Laravel, Javascript, jQuery, SCSS, AngularJS and HTML and I would love to have the opportunity to discuss your project w More

$588 USD in 10 days
(64 Reviews)
7.9
Eagles90

Dear Client, I am interested in your project and ready to go for it. Can we discuss more details of the project if you are available? I look forward to hearing from you. Kind Regards, Safeer Ali

$558 USD in 20 days
(60 Reviews)
7.3
aamaia

Sorry, updating the cost for solving captchas. 200 downloads/month (two for each chapter). Yearly cost is 2usd. My solution for the captcha is a 3rd party service which is more robust to algorightm changes on their sid More

$588 USD in 3 days
(36 Reviews)
6.9
Forket

Hello. I remember I had similar project related to polish government site. Captcha can be easy bypassed :) In other words - it you need script - no problem. In case of any questions feel free to chat. Thanks and have More

$941 USD in 30 days
(37 Reviews)
6.4
shafaqat11

Hello Sir, How are you? I have read the project and understand your requirement. I am a highly trained on Data Entry ,Web search, Web scraping Expert with great knowledge of Excel. I also can do manual/ automate More

$352 USD in 8 days
(61 Reviews)
5.8
webstersys

A proposal has not yet been provided

$588 USD in 20 days
(38 Reviews)
5.7
vkrahu

Hello sir, Thank you for reviewing my proposal. I have reviewed the job description. The only section which can not be done is point 1. We can not read captcha using PHP or other language. Other task can be do More

$647 USD in 10 days
(74 Reviews)
6.1
Ricardolg

I'll write in Portuguese, since you're from Brazil. Olá tudo bem? Posso escrever um script em Python para executar esta função e você instala no servidor que está o postgresql como um cronjob mensal (posso instal More

$350 USD in 15 days
(12 Reviews)
5.5
dirak696

Hello, Im very interested in you project, have extensive experience in graphic design 2d and 3d,adobe master suite,infographics, logo design, packaging design, web design,photo edit, ilustrator among other services wou More

$705 USD in 25 days
(22 Reviews)
5.4
pasuf

I have excellent reviews, a near 100% completion rating and I have earned my preferred freelancer badge. I have read over your project description and I'm interested in completing your task. I am a professional with ro More

$699 USD in 10 days
(29 Reviews)
5.4
tasleem83

A proposal has not yet been provided

$705 USD in 5 days
(32 Reviews)
4.3