About SmiffyDCMetaSpider

SmiffyDCMetaSpider is a Web robot that crawls pages, looking for metadata. The results are saved in a database for review by the content owners, allowing for the editing and creation of appropriate Dublin Core metadata.

This robot will rarely be seen in the wild, as it will generally be looking at specific collections of pages, with the owners’ knowlege.

Occasional experimental crawls may be run on the Web as a whole, respecting any robots.txt files found.

This spider should operate from a limited number of IP addresses:

  • The IP address associated with andreas.kbc.net.au
  • The IP address associated with jr.kbc.net.au
  • The IP address associated with prospero.kbc.net.au (This may come through a proxy, but the IP address should be determinable through the HTTP_X_FORWARDED_FOR header row.)

Visits from robots claiming to be SmiffyDCMetaSpider from any other IP addresses should be regarded as suspect, especially if they do not obey the robot rules.

A file in the format used by robotstxt.org is available here.

Once the development phase of this project has been completed, it is anticipated that the source and database schema will be released under a Creative Commons license, commercial support being available from the writer.

I may be contacted about this robot through the e-mail address at the bottom of this page.