Compare 1.4 million sites against 1 million authoritative sites

Completed Posted 7 years ago Paid on delivery
Completed Paid on delivery

I have a list of about 1.4 million URLs (Sites A) that need to be checked against another 1 millions sites (Sites B). Sites B are the authoritative sites that are from Alexa’s top 1 million sites.

I would like to have a program or script made that each of Sites A (1.4 million URLs) checked if it is listed in Sites B in a reasonable time (say, in less than 1 hour, but faster the better). The program can be made with: C, Ruby, Perl, or Python etc.

When the URL of one site in Sites A is listed in Sites B, I will need a flag be put up, and it must be saved as a text file along with the URL, one site per line. When the site in Sites A is not listed in Sites B, it will have a flag put down and it also be saved as a text file along with the URL.

Algorithm C Programming Perl Python Software Architecture

Project ID: #10107416

About the project

4 proposals Remote project Active 7 years ago

Awarded to:

kostiapl

Hi. I would like to work on your project. Im ready to create code without milestone and show you demo result to confirm my skills. Also before urls will be processed they need to be normalized.

$30 USD in 2 days
(7 Reviews)
2.9

4 freelancers are bidding on average $40 for this job

ahmsak

Hello Sir... I have a very good experience in C, Java, Ruby and Python,. I can do this within the mentioned constraints well (time, output and size). Please contact me for more details when possible. I look forward More

$50 USD in 1 day
(53 Reviews)
5.5
fejs

Hi Sir/Madam, I'm expert in Python programming and I can help You with this script. Can You send me examples of those 2 files (I suppose they are too big to be send whole). Best regards, Fejs.

$30 USD in 2 days
(75 Reviews)
5.7
tomkusvw

Hi! My name is Tomasz Kustra, and I am from Poland. I am interested in this project. I can do such thing for you in PERL. I am experienced programmer, you can see ma resume on VWorker (now part of FREELANCER. More

$50 USD in 3 days
(44 Reviews)
5.2