Popular Link Aggregator Site Crawler

Need a FAST web crawler that will run server-side and crawl a link-aggregating website. Website returns multiple in-line iframe result boxes using rotating javascript. Crawler must identify all relevant boxes (reults) and prepare a report containing all results which includes source page url and related links. Links are often not hotlinkable, rather, plain text. Need the ability to store reports and purge off server as needed. Secondary switch should have the ability to use crawler and compare reported link urls, using [login to view URL]'s api, and only keep those urls that are live for the reports. There should be a totaling system for total, live and dead collected links. Please note, we will not always be using this api so an On/Off mechanism should be in place. Program needs to have a table to access both public and private (dedicated & authenticated) proxies. Crawler will use proxies when it encounters detected blocking. Program should be ability to include future use of cron or equivalent to scrape unattended on terms tabled by user. Tabled terms should be able to be broken down by client.