Contents 

1. Background  

1. Background 
This document defines the purposes of evaluating the project - both formatively (to assist in design of HeadLine) and summatively (to review the project as a whole) and the tasks to be undertaken to achieve these. This document will discuss the various methods of evaluation and the criteria for selecting the most appropriate evaluation techniques; it will then provide an outline of the timescale for these activities and the output (formal and non-formal) from evaluation activities, together with the means of feeding these to appropriate stakeholders. However as evaluation is an ongoing activity, priorities may change as a result of early findings, so the evaluation process may be reviewed and altered throughout the course of the project. 

Related documents 
This report relates to the Communications Plan (HL-1998-04) as some feedback detailed here will feed into dissemination activities as outlined in that document. Some of the dissemination tools described in the Communications Plan will be used to solicit feedback from users. 

The Technical Workplan (HL-1998-03) is relevant as evaluation tasks outlined below will feed into the detailed workplan (either as a whole or within the three phases.) 

Finally the Project Plan (HL-1998-02) is important to this document as it explains the aims of the whole project, which provide the starting point for this evaluation strategy. The evaluation activities are described in much greater detail in this document than in the Project Plan, so in this area, the Evaluation Plan can be regarded as the definitive document. 

2. Purposes of Evaluation  
The Evaluation Plan exists aims to assist the project in fulfilling its aims, and to measure the outcomes of the implementation of the HeadLine model, both formatively, in contributing to design decisions, and summatively in looking at the impact and effects of the project as a whole. 

The aims of the project are: 

• To define the next stage in the development of the end-user library environment, going beyond resource discovery to resource access
• To establish a working model for consistent access for library materials regardless of physical form, which is transferable across broad subject disciplines
• To implement the model in a real-life environment, using tools and technologies which will be widely available to the academic library community

These aims reflect the respective strategic aims of the partner sites, so the results of evaluation will be useful in measuring the project against eLib aims in addition to evaluating the goals of the individual sites. All participating sites have strategic aims to develop electronic delivery of library material and enhance learning with integrated library systems. 

In practice, evaluation will fall into two areas: 

Formative Evaluation 
During Phase One the prototype components of HeadLine will be extensively tested on an operational level and feedback extracted from users, in order to assist design and ensure that the design suits user needs. This will be continued throughout the first two phases of the project. The aims of formative evaluation are: 

• To assist project management, by supporting the operational activities of the project and the development of the demonstrator
• To support project management and steering
• To provide input into the design and development of the database and interface and to provide feedback on the use and usability of the service, in order to continually refine the product in the light of available resources and changing user expectations and needs.
• To provide input into the content of the database - to look at resources available and select most appropriate ones, in terms of access, cost, 'networkability', usefulness, subject coverage and range; to consider whether the content adds value to the provision of services within the partner institutions. Finally to provide the best match of resources and technological organisation to meet the needs of the user community, and to suit all stakeholders.
• To develop new insights into user behaviour and establishing lasting contacts with end-users tracking changes in user behaviour, changing attitudes to HL and also user opinion of the system's development.
• To monitor traffic between sites (ie one way system or reciprocal) and determine whether use is concentrated towards resources at one particular site or spread throughout.
• To implement sensible access rights (in technical, political and commercial terms).

• To assess whether the HL project is meeting its objectives (above) and to provide information on the factors influencing the extent to which they are achieved, amended or changed during the life span of the project. To assess whether a hybrid library model is feasible in general terms.
• To test whether the HeadLine system performs to specification. To look at content and interface and to test views of stakeholders towards the product.
• To provide evidence of the benefits of and potential demand for the service. If possible to get some baseline data to help with a future or potential business case (ie were people happy with it, how much did it cost etc.) To provide information on issues and viability of maintaining it, in tandem with the exit strategy.
• To assist in the formulation of an exit strategy so the completion of the project can be managed whilst maintaining any services that may be useful to the partner sites, with as little disruption to users as possible.
• To assess portability to other subject areas, other institutions, other types of institution (public libraries etc), and to provide guidelines for other institutions wishing to implement this system and some issues for implementation. Also to identify problems, obstacles, things that need to change for it to work in the long term (resource implications etc).
• To document the process of innovation that can help shorten the learning curve when taken up by other institutions or environments.
• To provide evidence of the impact and effectiveness of the HeadLine model.
• To test the project against wider eLib objectives - that it assists the movement towards the electronic library, demonstrates new systems, shapes new user/supplier communities, speeds up uptake of network solutions, encourages co-operation and debate. Also to collate results of various eLib projects, and to help the learning of those involved (internally and externally).
• To examine the impacts for partner sites (eg BLPES as in Section 5 - HeadLine as a Resource) and for any environment which adopts this kind of system. To look at issues for validating new services in libraries.

3. Methodology 

Criteria for Selection of Methods/Activities 
Whilst some evaluation techniques are very useful and can provide interesting results, cost and time restrictions mean that evaluation cannot be pursued at the expense of performing the project itself. Therefore some criteria have been established for selecting evaluation methods, in order to perform effective evaluation. 

Cost  
Some methods may be useful exercises but are just too expensive to be performed in this project. The project proposal and budget agreed by the eLib project office does not provide for lengthy and in-depth exercises so activities have been chosen for their relevance and range. A cost benefit exercise was not included in the Project Plan. Therefore whilst evaluation activities will aim to provide information on how to implement a similar project elsewhere, they will not incorporate a detailed cost benefit analysis that is outside the scope of this project (although this may be a valuable exercise for a later date or project). 

Similarly while evaluation activities will take into account the general effects and popularity of the service there will not be a detailed feasibility study into charging for services. 

Range  
In order to use resources effectively activities have been chosen which complement each other rather than those that will provide a large quantity of the same type of information. The evaluation package will combine several activities in order to produce a useful mix of qualitative and quantitative data, recording views, experiences, preferences, needs and ideas from various stakeholders and members of the partner institutions (eg librarians, academics, students (undergraduates, postgraduates). 

Relevance  
Activities must be used which are directly relevant to the project and which have the potential to provide usable outcomes. A case study would be interesting (to provide examples of user experience and perhaps to identify issues, which may be common to other libraries and subject areas). However this would be a lengthy exercise and would produce qualitative data rather than facts about usage and as several other methods (questionnaires, interviews) aim to achieve similar outputs this activity was not chosen. Also this evaluation is not sufficiently central to the aims of the project to justify the time it would incur. 

Feasibility  
Another constraint on evaluation activities is available resource. It may not prove possible to perform the same evaluation activities in equal depth at all sites. Therefore some selective evaluation may be used, for instance performing some evaluation activities at one site or two of the three, in order to get a good range of feedback from different stakeholder groups, without overloading the team with too much data to analyse. 

Issues for effective evaluation 

• In order to achieve consistent results all activities must be structured and planned, and follow a specific format. Then they will be portable to other sites and environments and results will be comparable making their content more valuable.
• It is important to be aware of the differences between the partner sites in terms of administrative environment, subject content, procedures, student community, resources.
• Some sites may have different content within HeadLine due to license agreements or copyright. These will have to be considered when evaluating user feedback.
• Also the various other systems available in some sites may affect usage, so the existing conditions in each site must also be noted in the evaluation process.
• Evaluation must be performed ethically, and project members must maintain the anonymity of responses, and allow participants access to their own views on request.
• Finally evaluators must consider all the questions that need to be addressed when embarking on and planning evaluation activities, in order to cover all important issues and avoid repeat interviews for a question or topic that was missed out.

Users 
This group constitutes the end users of the product and will be made up of students, librarians and academics at the partner sites who will provide data to the project team. User input is useful when choosing between design alternatives, generating usability objectives, guiding revisions, trying out early versions and suggesting improvements. They will express views about the type of product they want, test the product and provide feedback on it. All have needs that must be met by the project, and will be consulted in the following main areas: 

• Usability of the system
• Value of content available through it

• Project team has an active role in evaluation, by performing and co-ordinating evaluation activity, implementing feedback into system design and project planning; also disseminating results outwards.
• Academics are potential users of the system and also help with design by suggesting which resources are needed, providing of course material and reading lists for the project and encouraging their students to use the system.
• Library staff are facilitators for the system. They will provide input into what users want, how it should be accessed, what could be on the system, as well as participating in testing and feeding back their experiences of it. There would be implications for their jobs if a HeadLine type development went forward, so their involvement is vital throughout the project.
• IT staff in each site, will be approached for their opinions, advice and feedback about technical aspects of HeadLine.

Commercial partners 
They will provide input about electronic resources, and their opinions on useful products to include in HeadLine. They will also provide assistance with copyright issues and sharing material between sites. 

Wider audience 
Users may also come from outside the partner sites, for instance, interested parties who have joined a user panel via the Website. There will be guest access to the system via the Website with structures for feedback during year 3 (after it has been tested in partner sites). ELib and JISC also participate in the evaluation process by receiving deliverables and providing the standards against which the project must be monitored. 

5. Evaluation Questions in Priority Areas  
The main evaluation questions, which arise from the aims of evaluation, fall into four main areas. These are explored below with some indication of the way they shall be addressed (although this will be explained further in the next section). 

• Content
• Interface
• Evaluating HeadLine as a resource
• Portability

• Existing Resources - what materials do users access from the partner sites? What do users like/use frequently etc?
• Potential Resources - what resources are available to the project from partner sites? What do users and stakeholders consider to be worth including on HeadLine?
• Profile of user's role as it impacts on selection of resources (status, department, registered courses, location etc).
• How do stakeholders use HeadLine? Type of use - ie different at various levels, academic, general, staff etc.
• How could the content be improved, to best suit needs of stakeholders?
• Relative use of content at different sites, how does use vary between sites and how much inter-site activity occurs?

• Standards, and security - the project aims to identify existing external standards relevant to Headline and will use formal ones (z39.50 or Athens) if they are suitable; but where necessary define project specific working standards and procedures.
• Conformance to adopted standards, ie does the site conform to standards used by other eLib projects?
• Due to the mobile nature of the user population HeadLine must be accessible from all sites (not just on campus)
• Ergonomic issues (efficiency, convenience, ease of use and content). Does the design of HeadLine meet the needs of its users?
• Conformance to accessibility standards, for use by the visually or aurally impaired.

• Whether the project met its own and eLib's aims
• Strategic effect on partner sites - outputs of usage and feedback will impact upon policy making, collection development and resource management. Also impact on copyright issues, resource sharing, and relationship between institutions.
• Costs - it would be helpful to examine the costs of the project, in terms of money, time, inconvenience, training, resources etc, (also opportunity costs and dead weight - what would have happened anyway). Although it is unlikely that a precise costing will be reached there will be some indication of the types of cost incurred, which may provide useful pointers for other projects or institutions. This activity will also seek to define the best product mix to include in a continuing service after the completion of the project.
• Analysis of alternative routes - would a different way have been better?
• Benefits - in terms of time, satisfaction, ease of use, variety for various stakeholder groups (eg freeing librarians from other activities). General implications for librarians' jobs, in terms of, professional activity, time and personnel resourcing.
• Usage patterns - how does usage change as the system develops? What are the determining parameters of usage and how do they differ between sites? What is the critical mass or threshold, for success (in terms of resources on the system and it being well used?) Who are the non-users or inhibitors to uptake?
• What are the implications of HeadLine's model of integrated service for teaching and learning in the Higher Education environment

Testing conditions at the University of Herts, and consultation with IT staff there, to see whether the system can be used across a large and wide-ranging campus - would be a good test of portability, as Herts has a large student population spread amongst sites. Testing at Herts will also help to assess the portability of content, to see if the system is non-specialist enough to be user-friendly to students at Herts (which is not business or economics focused). 

Guest logins on the web will also assist testing of portability of the system, as it will determine whether it can be accessed externally and via the Internet. 

• Review of feedback and whole project to identify problem areas and ways in which the conditions at partner sites are unique or that the project could be expanded.
• Technical conditions which would allow the project to work.
• Factors which may affect transferability - institutional contexts, conditions, policies, level of staff support, student population, usage of library systems, format of stock (mostly hard copy - mostly online), subject focus.
• Organisational/institutional/resource issues for implementation.
• How far is the subject matter in HeadLine unique in terms of nature and amount of resources that are available for incorporation into the system? Would a similar system in a different subject area encounter the same difficulties or situations?

Methods 
The following show some of the methods, which will be used to obtain answers to the questions outlined above. These methods are either user-centred (Stakeholder Consultation), enacted without user participation (Resource and Systems Evaluation) but still focused on the HeadLine product. Lastly, Project Evaluation describes assessment of the project as a whole. The expected outcomes of these methods are also included here. 

Stakeholder Consultation 

Focus groups, or steering groups 
These are useful for projection, ie estimating how users will behave or how things will happen in the future. They will be used to explore issues such as user expectations, views about resources, content and accessibility in an open forum setting. 

It is likely that these shall be held in separate sites, as differences between user populations and practical issues would make integrated groups impractical. Participants will be invited and selected via HeadLine publicity (see Communications plan). Groups will be formed that provide a suitable range of interests and user profiles. The format of the groups will be discussion focused around several structured questions or issues, so users can discuss and compare experience with their peers. Topics that may be raised are reaction to the system or what they think of the content and interface. 

These may occur several times during the project, to assess how user views are changing and to allow the project to be tailored to user needs, and so will be paced throughout the project. There may be some focus groups before the launch of the product, to discuss user needs, but they will certainly occur once the project is live. 

The outcome of these would be qualitative feedback in the form of internal reports to be analysed by project teams to determine user views and preferences. These may feed into other evaluation activities, in particular monitoring the whole project. 

Logging records  
This evaluation technique is a form of monitoring use of the product without making direct contact with the user. This enables real use of the system to be tracked without any of the distorting factors which occur when a user feels under scrutiny. However the project must endeavour to maintain anonymity of users, and use the log records to analyse how types of stakeholder use the system without 'spying on' users. 

The main reason for using logs in this way is not to examine how the system is used but to feed back information about use into the system in order to make it 'self teaching'. 

Another useful exercise will be to monitor logs in the light of views expressed in interviews, questionnaires etc, as previous studies have shown that users often give answers in an interview situation which do not correlate with their logged activity on the system. It appears that they often exaggerate their skills and requirements when being asked face to face. 

Information will be recorded about session information, ie identity of user, the time they logged in and logged out, actions - showing the searches made and resources used, and finally errors - which can be used to find bugs in the system which need correcting. 

Summaries will be created from these logs which can provide information about many aspects of a user session, for instance monitoring use by location, use by type of user, use of a certain resource etc. 

This activity will provide quantitative data which can be easily analysed to answer questions about levels of use, sources accessed, as well as user profiles. User logs will be monitored regularly throughout the project to note changes in activity throughout the project. 

The output from this activity will take the form of an internal report which will feed into wider studies, and may also feed dissemination by providing material for conference papers, journal articles or debate on mailing lists. 

Questionnaires  
Questionnaires will be used to collect quantitative data, for instance facts about how many users access a particular service, also views about quality of content ease of access etc. 

Respondents will be targeted across the whole user community, in one or more of the sites to gain facts about information needs and usage, at various stages of the project. An initial questionnaire in Phase One would help to establish the levels of use of the resources in the partner sites as well as gauging interest in the HeadLine system and establishing the information needs of students and academics. This could be followed up by a questionnaire in Phase Two, measuring feedback of the initial site (or sites) to help refine the product and a final questionnaire (in phase three) would collect feedback on the finished product. 

To gain feedback from those people who actually use HeadLine the user's identity will be recorded (as part of logging process) and a questionnaire sent at a later date (online or via hard copy) asking for his or her views on the HeadLine product and its content. 

It is envisaged that the questionnaire will address issues such as access to the project, use of library resources, opinions on new developments. The format will include tick boxes (with more scope than merely yes or no answers) and possibly some room for additional comments in order to obtain some qualitative responses as well as quantitative data. 

Questionnaires may be used in conjunction with interviews (see below), during which answers from a questionnaire will be discussed with the respondent. The output from this activity will be an internal report summarising responses which will be used by the project team to make decisions on the content and format of the product and also to evaluate user responses to the project and system. 

Structured Interviews  
This method serves a similar purpose to that of focus groups (above) but it enables individual discussion with the user and allows the user to discuss actual searching experience or to elaborate on answers given in a questionnaire. 

These too will be staged throughout the project, perhaps during the three phases and issues addressed will be user expectations, views about resources, content and accessibility. Users will be selected from across user groups, subject areas, type of user, institution etc. Two useful types of user to include in this activity will be academics and librarians who can give advice on content and interface, through their expertise and knowledge of the information and academic environments. 

The output from these will probably be internal reports or summaries, which will be used by the project team to tailor the product and make decisions about content and format. However the conclusions drawn from these summaries may well feed into official reports and summative evaluation in the final phase of the project. The final phase of interviews should help determine whether the system has lived up to or exceeded expectations. 

Resource and Systems Assessment 
This section describes evaluation of resources and testing of systems performed by the project team, or in consultation with the Steering Committee or other experts. 

Evaluation of Standards  
The project team will be study current standards, in order to find a suitable and relevant search protocol for the HeadLine project. Standards such as Athens will be examined as potential tools and the outcome of these studies will be the selection of a useful and relevant protocol which can be used in the project and which will make the HeadLine software compatible with that of other eLib projects. 

The output of this activity will be an internal report which will be used formatively in the design process. This may also feed into communication activities, as the findings of standards testing may be used in a conference paper or journal article. It will also be useful to monitor other JISC initiatives to ensure standards conform to those used in related projects. 

Any changes to external standards taking place throughout the duration of the project will also be monitored, and again this may lead into conference papers, or other communicable reports, as well as assisting design decisions. 

Site Resources  
The project team will undertake an evaluation of what resources are within the three sites, and how they compare in terms of main characteristics (content, subject matter, format, detail, accessibility, license agreements). This will be carried out in discussion with librarians and academics at the sites, in order to gain their input into the value and use of various sources in the partner sites. The output of this will be a list of the various resources needed for the project and selection of suitable range - it is hoped to include a range of subject, format, access type in order to make the system as heterogeneous as possible. 

System testing  
The system will be continually tested throughout its development by the project team, to ensure that it performs correctly and to specification. The prototype and integrated systems will be tested formatively (to aid design and modification) and summatively (to evaluate quality) at various stages of the project. There will be little formal output from the formative stage of testing - as the results will shape the design of the project, rather than provide conclusions outside the project. However summative testing may be written up and included in the final report. 

Project Evaluation 
The main strand of this evaluation process will be to test whether the project fulfilled its aims and what the outcomes were. Although project resources do not allow for any formal evaluation of the whole project, various means will be used to draw results from the project as a whole. The output of interviews, focus groups etc will be used to measure the outcomes of the project, and in some cases data which was collected in one particular situation will be used to assist the overall evaluation. For instance user logs may be used to measure how many searches were performed in a given month, which may provide an indication of the overall popularity of the project and assist any future cost benefit analysis. 

Similarly interviews with librarians (originally established to gain their advice on resource selection) may provide data about their role and how the HeadLine system has changed their role, again helping evaluate the impacts of such a project on its institution. 
 

When preparing questionnaires and other user consultation methods it is important to keep the whole evaluation aims in mind, in order to get information from stakeholders on a micro (opinions of the system) and macro (affects of the system on the environment) level. Output from this activity will be a report towards the end of the project. 

External Testing  
In order to achieve a degree of independent evaluation the project may engage in collaborative testing activities with another eLib project. In this way one project may provide a brief summative evaluation of another to test its progress and results against eLib expectations and the project's own aims. This will provide an objective assessment of the project without being too costly as the HeadLine team will offer reciprocal testing of a sister project. 

The team may also enlist the help of a Masters student who can evaluate the project (or a part of it) as part of his or her dissertation. This may be a student from one of the partner sites or from another institution, for instance City University (which runs a relevant Masters in information science course). 

Exit strategy  
The exit strategy will not incur particular evaluation activities but will analyse the project as a whole and translate the various evaluation results to form a strategy. This strategy's aim is to provide the most useful continuing service to users, after the completion of the project. Details may include estimating the types of costs and benefits involved in maintaining the service and demonstrating a way to end the project without disrupting the service. Also it will describe how dissemination and evaluation will continue so that the service can still be tailored to meet user needs, reflect changes in technology, and incorporate new services. 

The output of this exercise will be an exit strategy report, which may be incorporated into one of the final phase documents. 

Tools 
Various tools will be used to assist these evaluation techniques 

• Incentives will be used to encourage collaboration and help from stakeholders such as drinks at focus groups or access to demo sites for those willing to provide feedback or participate in interviews or focus groups.
• User community lists will be created and used to disseminate questionnaires, co-ordinate meetings, arrange interviews, as well as disseminating feedback or results and news about the project. The HeadLine users mailist (described in the Communications Plan) will be most heavily used for this, although some other internal communication gateways may be used where appropriate.
• A database of users will be maintained for mailshots and to record participation in evaluation exercises.
• Although the HeadLine Website will be used mainly to publicise the project, it will also assist evaluation by providing access to a demonstration site, with structured feedback mechanisms, (requests for feedback or an online questionnaire.) It will also allow interested parties to join the headline users mailing list and participate in discussion about the project and related issues.
• In addition to general user communities a target group will be recruited from amongst academic staff and research students. They should be representative of the various user community and subject interests within the HeadLine sites and if possible the same group will be used for the duration of project, in order to test the project against the same group of users. They will participate in interviews and focus groups as described above.
• A key feature of the evaluation techniques will be the HeadLine demonstration model. In the first phase there will be several separate systems for users to try, each with different facets of HeadLine in them; later the integrated HeadLine system will be launched for use. The various mechanisms for obtaining feedback will centre around these systems, users will be asked about them. Some feedback will be obtained via the systems themselves, as the models will incorporate questionnaires which users will be invited to complete, and the logging of user activity will play a major role in discovering how the systems are used.

The proposed timing for these activities follows the schedule set out in the Project Plan (HL-1998-02) 

Year one Evaluation Activities 
 

Date	Activity	Method
Spring 98	Evaluation plan	RSA
Spring 98	Evaluation of resources in sites, also the standards available, licensing agreements	RSA
Autumn 98	Recruitment of user panel and studies of user needs, views, existing use of services	SC
Spring/ Summer 98	Evaluation of desired content of HeadLine, finding out the following:  Profile of users, course, it literacy, level Current reading lists What sources are available What are used	RSA and SC
Autumn 98	Feedback from use by target groups of component parts of HeadLine service when launched, via logging usage, questionnaires, interviews, focus groups	SC
Dec 98	Review of year 1 including user feedback where possible	PE

Year Two Evaluation Activities 
 

Date	Activity	Method
Spring 99	Evaluation of hybrid library model , ie integration of components - testing by lib staff, academics and users	RSA
Ongoing	Logging usage	SC
Summer 99	Development of information content (interviews, focus groups with librarians)	SC
Spring/Summer 99	Discussion of experiences in using it - interviews, focus groups	SC
Winter 99	Detailed review of year 2 incorporating user feedback from deployment of hybrid library prototypes	PE

 
Year Three Evaluation Activities 
 

Date	Activity	Method
Spring/ Summer 2000	Integration of more sophisticated feedback systems into the model	SC
Ongoing	Logging usage and comparison of usage with responses from questionnaires interviews etc	SC
Autumn/ Winter 2000	Assessment of achievements of HL site  Use - summation of levels and type of use Critical mass of use of HL - ie enough users, quality of data Feedback from all users	PE
Autumn/ Winter 2000	Exit strategy - whether this can be continued - cost benefit issues etc	PE
Autumn/ Winter 2000	Evaluation of project - whether it achieved its aims etc	PE
Autumn/ Winter 2000	Implications for insitutions involved	PE
Autumn/ Winter 2000	Assessment of portability to other subjects, institutions	PE
Autumn/ Winter 2000	Evaluation of the project as a whole?  Methods used  Management of project Impacts for partner sites, environment, eLib and wider audience	PE

Key to abbreviations used in Method column  
(All described in Section 6 - Methods and Tools) 
 

Abbreviation	Method
RSA	Resource and Systems Assessment
SC	Stakeholder Consultation
PE	Project Evaluation

Issues for timing of Evaluation 
Evaluation must follow the overall timing of the project - evaluation must not drive the project's activities. 

It is important to involve users throughout - although the types of users who are needed most will vary through the phases. 

It will be necessary to plan an adequate period of testing so that users can choose the system or it can be modified according to results of evaluation. 

At every stage of evaluation it is necessary to disseminate information so that results of evaluation can be fed back into the design and maintenance of the project. 

It is important to co-ordinate evaluation activities - so that if students are to be approached for their views on accessibility and also content requirement that these two questions are asked at the same time. Any planned evaluation activity must consider all the questions that need to be asked so that time and resources are not wasted by duplicating interviews or questionnaires. This would also cause greater inconvenience to respondents of questionnaires, interviewees etc. 

8. Feedback Mechanisms  
Adequate procedures must be introduced to ensure that regular and relevant feedback from the evaluation activities is given to the appropriate stakeholders. The Evaluation Plan should operate in tandem with the Communication plan and evaluation should feed communication and dissemination activities. 

Although the only formal evaluation outputs are the annual reports and the final report for the project there are many other mechanisms for communicating evaluation data, both internally and externally. Much of the output of evaluation activity will not be formally published, as its purpose is to modify the product to match user needs and expectations. This information will therefore be collected and distributed to project team members in order for them to adapt the design where necessary or include extra content. However the conclusions drawn from this feedback will figure in the formal outputs, as summaries of user studies to answer specific evaluation questions. 

As different stakeholders will need different information at different times, the table below shows the mechanisms for achieving relevant and timely feedback. 

Mechanisms for feedback to various stakeholders 
 

Stakeholder	Information Requirements	Methods	Deliverables	Timing
Users	How to use HeadLine, new developments and how to input feedback etc	Site and dissemination materials online and in partner sites	Site and promotional literature	Ongoing
Partner Institutions	Effects of this service, demand for it, benefits and some idea of costs involved	Conclusions from evaluation activities (eg record logs, interviews etc) included in official reporting structure	Final report (or special report to partner sites)	Phase 3
Project Team	Information about how students use HL, what sources they use, frequency, technical problems in order to tailor product	Questionnaires. Logging records, interviews and focus groups,	Internal reports and summaries of evaluation (via meetings, team Website and HeadLine-techies mailing list	Phase 2
Project Team	General project information - administration of evaluation tasks and feedback of key stages, etc	Internal dissemination of all evaluation activities	Team Website, HeadLine-techies mailing list	Ongoing
Project Team	Resource information, standards available	Research	Internal reports and summaries of evaluation	Phase 1 & 2
Academics at partner sites	Whether access to information is improved, what is the content, effects on course, is there educational benefit	Conclusions from evaluation activities (eg record logs, interviews etc) included in official reporting structure	Final report (or special report to academics)	Phase 3

 

Stakeholder	Information Requirements	Methods	Deliverables	Timing
Academics at partner sites	How to access HeadLine and provide information for content (eg reading lists etc)	Site and dissemination materials online and in partner sites	Site, promotional literature and launch	Ongoing
Librarians at partner sties	How to access HeadLine and what it contains (in order to direct students), other administrative info (eg printing, access etc)	Site and dissemination materials - also consultation with librarians and possibly training session when it is launched	Site, promotional literature and launch	Ongoing
Librarians at partner sites	Effects of HeadLine - changes to information access and supply and implications for their role	Conclusions from evaluation activities (eg record logs, interviews etc) included in official reporting structure	Final report (or special report to partner sites)	Phase 3
Steering Committee	Whether project is fulfilling its aims, eLib concerns, outcomes of project	Annual report at end of each phase	Annual report	Phase 1, 2 and 3
Steering Committee	Day to day running of project	Steering Committee meetings, annual reports, other dissemination	Annual reports	Phase 1, 2 and 3
Wider audience - general public	General information - demonstration site	Website and other dissemination deliverables	Website, promotional literature	Ongoing
Wider audience - general public	Results, issues, implications of the project for library access, subject study	Website and other dissemination outputs	Website, conferences, promotional literature, journal articles	Phase 3
Wider audience - other eLib project	Progress of HeadLine, results, successes, demand for service, relevant issues	Conclusions from evaluation activities (eg record logs, interviews etc) communicated to other projects via conferences, mailing lists and journal articles	Website, conferences, promotional literature, journal articles	Phases 2 and 3
Wider audience - eLib and JISC	Is project meeting objectives, within budget, contributing knowledge to programme area, ongoing feasibility of HeadLine	Formal reporting from all evaluation activities	Final report, annual reports	Phases 1,2 and 3

 
9. Evaluation Activities and Deliverables.  
The table below summarises the various evaluation activities to be undertaken with the relevant stakeholders and deliverable from these. 

Internal reports will be used to summarise the findings of user focused methods, these will be used by appropriate team members. 

The three annual reports will aid evaluation as they will summarise the activities of the project as a whole (assisting project evaluation) as well as the results of user studies. 

The final reporting deliverables cannot be described fully here, as outputs will depend upon the direction evaluation takes throughout the project, however through a final report and possibly other deliverables (exit strategy report, report to partner sites), the evaluation results will be delivered to appropriate stakeholders. 

 

Activity	Stakeholders involved	Site	Time-scale	Deliverable
Focus Groups	Partner sites, Students, Librarians, Academics	All sites?	1 per year	Internal report
Activity Logs	All users of system, librarians, students, academics, external	All sites	Ongoing but data collated each month	Internal report
Questionnaires	All users of system, also external users (via Website)	All sites and externally	1 per year	Internal report
Interviews	Partner sites, Students, Librarians, Academics	All sites?	1 per year	Internal report
Evaluation of Standards	Project Team	To be confirmed	Phase 1	Fed into design of HeadLine (possibly an internal report)
Evaluation of Resources	Project Team with librarians and academic staff	All sites	Phase 1 (and some modification in Phase 2)	Fed into design of HeadLine (possibly an internal report)
System Testing	Project Team	To be confirmed	Phase 2 and 3	Fed into design of HeadLine
General project evaluation	Project Team	To be confirmed	All phases (most in Phase 3)	Formal reports (annual and final)
Exit Strategy	Project Team	To be confirmed	Phase 3	Exit strategy report

 
 
For paper copies of this document or any more information about the project please email the HeadLine Team.