Completed

Need a web spider scraping program written for the following username DFUSER33803

I need a web scraper written for the following url:

[login to view URL]

Click on "KNIGHT" tab at the top of the page, in the middle. Data to be scraped will be under the "KNIGHT" tab.

All pages will need to be retrieved not just page one.

The data on this site changes and the number of pages will vary, however, we need to scrape data from all available pages.

The output should be a pipe (|) delimited file with the following column mappings:

origin_city --> data located in the "Origin" column before the comma

origin_state --> data is the 2 letter abbreviation in the "Origin" column after the comma

the 5 digit zip code after the abbreviation is not needed

ship_date --> data located below the origin_city in the "Origin" column, changed to the YYYY-MM-DD format,

the time located after the date is not needed

destination_city --> data located in the "Destination" column before the comma

destination_state --> data is the 2 letter abbreviation in the "Destination" column after the comma

the 5 digit zip code after the abbreviation is not needed

receive_date --> data located below the destination_city in the "Destination" column, changed to the YYYY-MM-DD format,

the time located after the date is not needed

trailer_type --> data located in the "Equipment" column

load_size --> add the text "Full" to this column

weight --> data located in the "Weight" column

length --> Leave blank

width --> leave blank

height --> leave blank

trip_miles --> data located in the "Miles" column

pay_rate --> leave blank

contact_phone --> 866-500-5623

contact_name --> leave blank

tarp_required --> leave blank

comment --> leave blank

load_number --> leave blank

commodity --> leave blank

The first line of the output should contain all of the column headers.

Any field that contain no data should be left blank.

Please do not use words like "null" or "blank" in blank columns.

Below is a sample output of the first 5 columns using sample data:

origin_city|origin_state|ship_date|destination_city|destination_state|

chicago|IL|2017-03-15|new york|NY|

kansas city|MO|2017-03-15|houston|TX|

The deliverable will be a Perl .pl file that must run on

Ubuntu Linux and must use Modern::Perl. The Perl .pl file

should be called '[login to view URL]' and the output file should be

called '[login to view URL]'

It will be scheduled in cron to run unattended every 15 minutes.

Please specify what language/OS/modules you plan to use.

Also, please include the word "raccoon" in your bid so I know that

you read this description.

Skills: Perl, Web Scraping, Linux

See more: need web master, need web designer landing page, write web spider, clickbank need web site, web spider development, need web site, web spider source, web spider crawling website robot vbnet, software experience need web designer, need web developer, web spider collect data, email scraping program, build program written, web scraping program, need program written, need program written website, need web program, i need a developer to help develop a web based software program i will provide full project details once you place a placeholder, why do we need web scraping

About the Employer:
( 68 reviews ) Chillicothe, United States

Project ID: #21957805

Awarded to:

gangabass

I know about "raccoon" and I can provide you a Perl scraper that will use WWW::Mechanize, Text::CSV_XS and other modules to get information from above website.

$88 USD in 1 day
(480 Reviews)
7.4

4 freelancers are bidding on average $109 for this job

mantislin

Hi there, This is Lin, i did more than 380+ scraping project, please view my profile page here https://www.freelancer.com/u/mantislin.html then you will know. Can we discuss more details about your project? then i ca More

$109 USD in 3 days
(290 Reviews)
7.6
jinridong19713

raccoon Hello, sir Thank you for your job posting. I have read your post. I am an expert who have many experiences in Web scraping with python selenium, scrapy and bs4, PHP and nightmarejs.. In particular, I had great More

$100 USD in 7 days
(3 Reviews)
2.5
AntonKozel

Hello, dear, Thanks for posting your job. Please check my review. I have many years of work experience in python, scrapy, Selenium Webdriver, BeautifulSoup, Scrapy, [login to view URL], Database(Mysql, Mssql...), Data Mining/Dat More

$140 USD in 1 day
(2 Reviews)
2.2