Comparision of two large files

I got several large files from a few hundred megs to 1.5 gigs. I already have a program that splits the already sorted files into files named a-z/0-9 etc then using diff compares a to a and so on gives me the modifed,deleted,added. It's written in perl and i am looking for a faster solution. Right now the order is split file into small files then using diff it compares... Basicly its thee programs written in perl that it calls. I am looking for a very fast way to do this including using multiple machines/ftp sessions/parrilel tasking to get the files and process them. Here's exactly what i need 1) Get files daily as soon as its updated(would need to check at a common time and get it as soon as its released).Will get files from two different sites with two diff 2) Compare morning to evening and evening to morning differences between 3 set's of [login to view URL] file1,file2,and file3 evening would be compared to file 1,2,3 of morning and the morning would be compared to evening . 3) Get added,changed,modifed,and deleted results and put them in seperate files. 4) I assume all three files would be gotten at once from the ftp site at the maximum bandwidth. 5) Highly prefer a solution that can be split between machines. 6) Highly prefer some sort of parriel tasking . 7) add ability to check results against fixed files and see if it matches certain words say in a dictionary or whatever. Right now i am using normal old diff to compare if theres a better way i certainly up to that. My machine specs for my main server is 1.5GIGS of ram and a p4 2gighertz. I don't mind if the system is entirely taken over for the task. I expect it to be using 100% cpu and 100% of memory if needed. I am up to using another os or dedicated os if need be. Speed of execution is the #1 priority with this project. All bids must have details ..Bids without details will be ignored. If you want to see my current solution please let me know and i will give login info to check it ou ## Deliverables 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Installation package that will install the software (in ready-to-run condition) on the platform(s) specified in this bid request. 3) Complete ownership and distribution copyrights to all work purchased. ## Platform linux/unix