Automate monthly CSV file extraction w/ CAPTCHA, write entries to PostgreSQL database + log
$250-750 USD
Paid on delivery
Develop a code that:
1. Downloads all available links at both tabs (IMPORTACAO/EXPORTACAO)
[login to view URL]
attention to CAPTCHA necessary at every link download
2. Unzip and save the CSV files into a PostgreSQL database. Entries are separated by "@" separator.
Problems might occur when some fields have "@" in the text, so this error also be handled.
3. Every month, check for new download links. Download new files and add to database.
4. All steps should be logged in a log table.
Expected result is:
1. Populated PostrgreSQL table with all results available on IMPORTACAO tab
2. Populated PostrgreSQL table with all results available on EXPORTACAO tab
3. Populated PostrgreSQL table with all logs
4. Actual code(PHP, Ruby or Python) delivered through github that will be used to run every month to extract new data
This data is published by Customs and will be used to update www.tradeadvisor.com.br - a website to help importers/exporters check prices of similar products that have been imported/exported recently.
Our goal is to make the data more easily accessible and searchable through keywords using Elasticsearch.
Project ID: #12047135
About the project
Awarded to:
Hi friend, I am one of the top freelancers in this category. Feel free to check my reviews for reference of my work. This looks like a tricky job, exactly the job for me :) I will be happy to help you with this More
31 freelancers are bidding on average $625 for this job
Hi, This application will be developed using PHP. The PHP programming has all the solutions for your current problems as mentioned below. What do we need to do to captcha? do we need to stop at captcha to ask input fro More
Hi, my name is Sevinc. Thank you for the invitation... I read your "Automate monthly CSV file extraction w/ CAPTCHA, write entries to PostgreSQL database + log" project descriptions carefully before bidding. I checked More
Hi we can automate and download the files using PHP CURL or PantomJS and CasperJS headless browser depending on the security applied to the website for bots. once we down loaded we can load the data to database running More
With Freelancer Preferred badge bound to give 100% Quality (Let's Chat) Hello, A great team is here for your service!! I read the project description properly, and I agreed to fulfill 100% of all your requiremen More
Hello..I have an experience of more than 8 years in web development and maintenance. I have in-depth knowledge of php, mysql, jquery, paypal integrations, API's, css, html, html5 and SEO. Our team is experienced, cr More
Dear Client, I am interested in your project and ready to go for it. Can we discuss more details of the project if you are available? I look forward to hearing from you. Kind Regards, Safeer Ali
Hello Sir, How are you? I have read the project and understand your requirement. I am a highly trained on Data Entry ,Web search, Web scraping Expert with great knowledge of Excel. I also can do manual/ automate More