Code like song
ruby linkcheck
2009-03-28
I thought a script to check our website for broken links might come in handy, so I decided to whip something up in ruby. The script takes a starting page, visits it, follows all its links, and keeps going as long as it’s still in the original domain. There is also an option to exclude certain portions of your site from being checked. I call the script like this:
$ linkcheck -e '^http://jungwirths\.com/gallery/.*' jungwirths.comIt was fun trying out something a little bigger. I’ve written a few other scripts in ruby lately; perhaps I’ll post them later.
All my work so far has been with ruby 1.8. I know 1.9 is out now, so I’d like to get it installed eventually. The Unicode support in 1.8 is pathetic. I found this out by trying to write a od-like program for UTF-8 files that mix English with polytonic Greek. I’m not sure what Unicode is like in 1.9. I haven’t been able to find anything very revealing from googling. It looks like they did something, though it was controversial. There was a lot of argument on the ruby mailing list a couple years ago. But for the life of me, I can’t find anything that explains what the final decisions were.
In my opinion Java gets Unicode just right: all strings are UTF-16, and encoding conversions happen on I/O. For instance, to read a UTF-8 file, you use a FileReader initialized to convert from that encoding. There are also I/O classes to handle binary data. The programmer’s job is simple, because Strings always act the same, and they just do the right thing. It is a very effective use of modularization and separation of concerns. But I don’t think ruby went this route, because for various reasons Unicode is unpopular in Japan. I hope the ruby approach isn’t too painful. I enjoy ruby a lot, but I can’t imagine using it in production with such remedial capabilities. Unicode for me is a real deal-breaker.
Here is the code:
#!/usr/bin/env ruby
# == Synopsis
#
# linkcheck: recursively look for broken links on a website
#
# == Usage
#
# linkcheck [options] start_url
#
# -h, --help
# show help
#
# -e, --exclude [regex]
# If any link matches the given regex, linkcheck will test it but not follow further links found there.
# You can use this to mark off sections of your site that shouldn't be crawled.
# Multiple -e flags may be used, and crawling will cease if the link matches any of them.
require 'rdoc/Usage'
require 'getoptlong'
require 'net/http'
require 'uri'
require 'hpricot'
# TODO: allow a list of "irrelevant" url parameters, so we don't keep checking the same page.
def within_site?(url)
$site_domain == url.host
end
def parse_links(html)
doc = Hpricot(html)
html_links = (doc/"a[@href]").select {|e| e['href'] !~ /^javascript:/ }.collect {|e| e['href']}
# TODO: do something more intelligent with forms:
# - skip POST forms.
# - supply data to GET forms, or skip them too?
html_links.concat((doc/"form[@action]").select {|e| e['action'] !~ /^javascript:/ }.collect {|e| e['action']})
other_links = (doc/"img[@src]").collect {|e| e['src']}
# TODO: get css stylesheets
# TODO: get external javascript
return [html_links, other_links]
end
def retrieve_page(url, limit=10)
raise ArgumentError, 'HTTP redirect too deep' if limit == 0
res = Net::HTTP.get_response(url)
case res
when Net::HTTPSuccess then res.body
when Net::HTTPMethodNotAllowed then '<html></html>' # good enough; this happens hitting POST forms
when Net::HTTPRedirection then retrieve_page(url + res['location'], limit - 1)
else
raise "Broken link: #{res.code}: #{res.message}"
end
end
def confirmed_good(url)
$good_links.has_key?(url.to_s)
end
def confirmed_bad(url)
$bad_links.has_key?(url.to_s)
end
def record_link(src, url, linklist)
u = url.to_s
# We use a hash of hashes instead of a hash of arrays
# because arrays could permit duplicates of src.
linklist[u] = {} unless linklist.has_key?(u)
linklist[u][src ? src : '[initial]'] = 1
end
def record_good_link(src, url)
record_link(src, url, $good_links)
end
def record_bad_link(src, url)
record_link(src, url, $bad_links)
end
def check_page(src, url, is_html)
url.fragment = nil
return if confirmed_good(url) # don't bother recording the location of all good links
if confirmed_bad(url)
record_bad_link(src, url)
return
end
# puts "checking #{url}: #{is_html}"
begin
data = retrieve_page(url)
# p data
rescue
puts "error: #{$!} on #{url}"
record_bad_link(src, url)
return
end
record_good_link(src, url)
if (is_html and within_site?(url) and not excluded(url))
links = parse_links(data)
# p links
links[0].each {|u| check_page(url, url + u, true) }
links[1].each {|u| check_page(url, url + u, false) }
end
end
def excluded(url)
u = url.to_s
$excludes.each do |ex|
return true if u =~ ex
end
return false
end
def print_links(hash)
hash.each {|(k,v)|
puts k
v.each {|(k2,v2)| puts "\t#{k2}"}
}
end
$site_domain = nil
$good_links = {}
$bad_links = {}
# TODO: allow some token to represent the hostname. (will $site_domain work? #{$site_domain}?):
# $excludes = [%r,http://jungwirths\.com/gallery/.*,]
$excludes = []
opts = GetoptLong.new(
[ '--help', '-h', GetoptLong::NO_ARGUMENT ],
[ '--exclude', '-e', GetoptLong::REQUIRED_ARGUMENT ]
)
begin
opts.each do |opt, arg|
case opt
when '--help'
RDoc::usage 0
when '--exclude'
$excludes.push Regexp.compile(arg)
end
end
rescue Exception
puts $!
RDoc::usage 1
end
RDoc::usage 1 unless ARGV.length == 1
# p $excludes
starting_url = URI.parse ARGV[0]
starting_url = URI.parse("http://" + ARGV[0]) unless starting_url.scheme
starting_url.path = "/" unless starting_url.path.length > 0
$site_domain = starting_url.host
check_page(nil, starting_url, true)
# now print the results
puts "Good links:"
print_links($good_links)
puts "Bad links:"
print_links($bad_links)
Code
