Resource Database Link Checker Specification

Functionality

The main program will run as a cron script (daily). It will read a configuration file to find a list of tables/columns to check (this is so it won't end up being tied to a particular version of the resource database). For a particular table/column in the resource database, the checker should access each URL and obtain the HTTP status (probably using the LWP library). It will use a module to actually perform the checking (this module might provide some functionality for the CAS service we've been discussing later on). Broken links (ones whose status is not "OK" or "Authentication required") should be reported via email in the first instance to the appropriate administrator, with as much information as possible. (When some sort of Web interface for the RDB is developed, the information should also be available there.) If the configuration file lists a place to put link-last-checked information in the database, this will be updated. The checker will put the HTTP status code returned in a link-last-checked-status field, whether it is 200 (OK) or not. Since availability may be affected by other factors than the Web server from which the resource is delivered (e.g. local network outage), it will be necessary to have a code for time out.

Since the location availability field is used to determine whether a link is displayed to users, it will probably be confusing to change this as a result of the link becoming unavailable. Users who follow the link will therefore see just what would happen if they normally accessed a link with the problem, and may well not be affected if there has been a temporary change in the link status when the link checker ran. The user would also want to be able to get at the resource as soon as the administrator fixed it, rather than having to wait until the next time the checker ran.

The appropriate administrator. I want to add information to the resource database to determine which groups are to be allowed to edit the resource description. The configuration file would say which groups the link checker is to be run for, and for each group it would provide an email address. (When there is a Web interface, we'll need information about who can edit what to be in the resource database for that as well. We don't then want to end up in the situation where vast numbers of links are reported as broken to an administrator who cannot fix them.)
The information reported should explain which link is broken, and which location of which resource this relates to. I'd probably want to use configuration files to get something like:
```
Resource/about_url link http://www.brokenlink.com/help_for_resource is 
"not found" for resource 138439 (Brokenlink Service)
Location/url link http://www.morebrokenlinks.com is "permanently moved" 
for location 342349 of resource 138439 (Brokenlink Service)
```
More generally, this takes the form:
```
Table/column URL (as link) is "HTTP error message" for table unique_id 
[link to resource table] (resource name)
```
This would need to be made more user friendly when there is a web interface, because the librarian would not know anything about tables and columns in the resource data model.

Configuration and data structure

The configuration file will contain the following information:
- Resource database connection details (this isn't found from a local PIE configuration file to make it possible to run the resource database completely separately)
- Library script location details
- List of tables - each contains unique_id column name, optional link to resource table linktable/column name, list of columns to check
- List of columns - each contains optional link-last-checked column names
- List of group names - each contains administrator email address
The initial list of tables columns will be:
- resource/about_url
- resource/training_url
- location/url
- location/help_url
- location/license_url
- supplier/url
- icon/url
The following changes will need to be made to the resource database.
- For each resource, a group name will need to be given "edit" rights to the resource (in the group_rights table). I would expect this to be something like "lse-admin".
- At the moment, there is only one link-last-checked column in each table, but resource and location tables have several URL columns. It would probably make sense to change the structure to have a link-last-checked column for each URL column. This is not an urgent change but something that should be incorporated into the next version of the resource database (nothing uses this column yet, so it won't make much difference to anything else). At the same time, we will add a link-status column for each link, to contain the HTTP error code (whether it is 200 for OK or not).

Resource Database Link Checker Specification

Functionality

Configuration and data structure