I'm looking for someone to write a web crawler that can allow us to make a list of every URL on our website domain, which is www.basketballmegastore.com.
We are about to rename our business and move over to a new domain, so I need all of the URL's from our current site so that I can set up forwarding for every page to the new domain.
I've tried using some off the shelf crawlers that are available on the internet, but they only ever pull back maybe 700 - 1000 page URLs which is too few, and the lists were definitely incomplete.
I would need the data supplied in an Excel sheet, with a URL on each line.
Any questions, please just ask.