I need an algorithm that will compare two emails (already parsed into in memory data structures) to see if they are essentially equivalent. It must ignore headers that would obviously be different, like who it is To: and also be able to deal with word salad within the body.
So, for example:
To:john at somewhere dot com
Buy lots of viagara. Fish leaves dung forage issue stamp view front wish.
And
To:george at somewhereelse dot com
Buy lots of viagara. Dump knife tweezers frontal Jeff spank lizard.
Should come out as a match (at at least some confidence level) since the basic message is the same in both cases, "Buy lots of viagara". Note that the word salad might be in the middle of things or even at the beginning. Essentially what is desired is a distance of phrase matches… I think. The idea is if they both contain enough phrases in common, then they are the same. The definition of "Enough" should of course be tweakable. Of course some headers might be potentially useful to compare, such as portions of the Subject: line.
The other result that I need from this project is that if two emails are found to be equivalent, then a hash function must be created that would allow for a third email to be compared to the first two that matched without the text of the original email. For example, in the above case, the hash function would include the range of characters in the email containing the words "Buy lots of viagara." Along with something indicating that the common feature is in the body of the email, And a hash based upon hashing those words, but not the words themselves. It should hash into a VERY large hash space so that false positive matches would be on the order of only 1 in 100,000,000 or so.
If you have other good ideas for comparing emails or creating such a hash function, I’m open to any suggestions.
Note from RAC: For the protection of all parties concerned, RAC does not allow the posting of the solution to a project before funds are escrowed. Any persons found doing so despite this warning may be permanently banned from the site. *******************************
## Deliverables
1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done.
2) Deliverables must be in ready-to-run condition, as follows? (depending on the nature? of the deliverables):
a)? For web sites or? other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment.
b) For all others including desktop software or software the buyer intends to distribute: A software? installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request.
3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement).
## Platform
C# code in Windows .NET environment. All managed code.