One of the great things about WordPress is how 404 error pages are handled. If a page isn't found then you can show a proper dynamic error page giving the user things to do - this removes a lot of the traditional problems with 404 pages, ie the dead end syndrome.
404 is the server error code that is generated for content that doesn't exist.
However these new 404 pages are a lot slower than normal 404's, they require database contact, and PHP processing so if you have a lot of them they can slow down your server. Most times the benefits outweigh the negatives however there is one time when this isn't the case.
Images and other media that are not loaded, do not need a full 404 page. If you're linking to an image that doesn't exist then you could create a lot of extra work for your web server.
How to make it work
To stop the 404 page from executing fully I look at the url and work out if it's a url for a blocked file type (jpg, css, js, etc). If the file is in the list of bad file types then it gets stopped and a message is displayed.
function bm_404Response() {
header('HTTP/1.1 404 Not Found');
if (!empty($_SERVER['REQUEST_URI'])) {
$fileExtension = strtolower(pathinfo($_SERVER['REQUEST_URI'], PATHINFO_EXTENSION));
} else {
$fileExtension = '';
}
$badFileTypes = array(
'css', 'txt', 'jpg', 'gif', 'rar', 'zip', 'png', 'bmp', 'tar', 'doc', 'xml', 'js',
);
$badFileTypes = apply_filters('bm_404BadFileTypes', $badFileTypes);
if (in_array($fileExtension, $badFileTypes)) {
echo 'error - file does not exist';
die();
}
}
Where to use it
This function should be added to your themes functions.php file. To execute it I placed it at the top of my 404.php template page, however in hindsight I could probably also use an action instead and may just tweak my theme to use that instead. There's nothing like writing about something to make the problems clear! 
Do you have any ideas for other things I could do to improve this code? Do you think it's a good idea?
You could just hook it to
initlike this:add_action('init', 'bm_404Response');This only works for static file types, because of the way your server serves these things: for images etc. it first tries to find the file on your server and serves it immediately without loading WordPress. If it can't find the file, it defaults to loading WordPress, then when you have that function you wrote and do the add_action as above, it'll immediately output and stop right there.
By the way, W3 Total Cache has an option where you can set it to quit even sooner, it adds some lines to your .htaccess file that essentially do the same without even booting WordPress at all.
April 25, 2011 • @yoast
Hey Joost - thats for the feedback. I didn't realise w3 total cache added rewrite riles for that. That's clever stuff!
I did try adding the code to the init hook, but WordPress hasn't decided if the content is a 404 or not by that time so I have moved it to the template_redirect hook which seems to work quite well, and cleans up my templates a little.
April 25, 2011
The funny thing is, when it's a static file, if you're not running any weird plugins, WordPress doesn't have to decide whether it's a 404. If WordPress is being called for a static file, it is a 404 otherwise the web server would have served the file directly.
April 25, 2011 • @yoast
By the way, you should really disable paged comments, you're already having a comment-page being indexed and I don't think you need them often, to be honest.
Feel free to just delete this comment
April 25, 2011 • @yoast
Hmm - that's interesting. I thought WordPress took care of paged comments with it's rel=canonical, and nofollow settings. I wonder if I can build support for that into my theme...
The reason for the paged comments is that I have a number of posts with massive amounts of comments (500+), and even more with 100+ comments. I've bumped up the number of posts per comment page but that won't help with this issue since the comments still link to the comments pages.
April 25, 2011
WordPress does set the canonical correctly, but it's still not ideal, ideally the comments after 100 or so would be served using JavaScript... Funny thing is, the issue is not with the comments, even at 500 that'll hardly affect pageload, it's with the gravatars...
April 25, 2011 • @yoast
Given I brought this article to Joost's (Yoast's) attention, I figured I'd chime in and say it's been interesting to read. I implemented the stuff in this article into my 404 page via functions, but I haven't tried via init as of yet. And W3TC does have the 404 option for static files via the "Browser Cache" page (not sure why it's there). And I guess you could customize the 404 code with that as well by setting the "Error 404" entry in the .htaccess.
6 of 1, half a dozen, you know. Either way, getting the load for 404's off WP when it's bad files being requested isn't a bad idea. And Yoast's 404 article (http://yoast.com/404-error-pages-wordpress/) is what started all this anyway. Oh, and Google's article about intelligent 404's (http://www.google.com/...swer.py?answer=93641&hl=en)
April 26, 2011 • @stazeii