HEADMasterSEO is a bulk server response checker written in PureBasic. It works on both Windows/Mac OS X.
The tool takes a list of URLs, gets the response headers and presents the results in a big grid for filtering, sorting and exporting to CSV. It follows all redirects and you can view every location in a redirection path.
It can use both HEAD and GET requests and can handle millions of URLs with up to 200 threads.
Supported filters: http/https urls, 200 status code, 3xx status code (301 redirects, 302 redirects, redirect chains, redirect loops), 4XX client errros, Page Not Found (404 and 410 status codes), 5xx server errors, slow pages, timeouts.
Use custom request headers and user agents. View and export server response headers to CSV.
I've used it to test website migrations, http to https migrations, canonical redirects and XML sitemaps.
It's my first commercial project in PB (I've used Delphi before) and I'm happy with the compiler/language.
Some technical background and challenges:
- it uses WinHTTP on Windows and a custom libcurl on Mac OS X
- virtual grids work differently on Win/Mac OS X
- thread sync -> updating the grid, adding incoming URL results and responding to filtering/sorting events with millions of URLs was a challenging programming exercise
- PB string reading from text files is slowish
- using UTF8 strings saved a lot of memory on huge URL checks
- after checking the top 1 million domains responses you'll find so many header errors and badly configured servers
Thank you to every1 who helped with my Mac OSX questions and the PB team for the incredible product.