 |
Methods of data collection. Doing evaluation:why and for whom?
|
|
|
|
Methods of data collection
To assess whether or not a programme or project performance is measuring up to what it says it is meant to be doing requires the collection of relevant data. This can be achieved in several different ways (Bartels and Richards, 2011; Queensland Government, 2016; Walter, 2013a). Using multiple sources and a triangulation (Walter, 2013b) or combining of different research methods (that is, using at least two different sources of information or types of data collection) increases the reliability (or consistency) of the findings and reduces the limitations of a single research method.
Quantitative
This type of data collection is basically concerned with counting and measurement. It provides an indication of broad statistical trends:
· numbers of people using a service (a simple numerical count of referrals to the programme, or who uses which specific types of service provision, such as coun- sellors, doctors, youth workers)
· extent of service use in particular geographical area (linking of numbers with particular location, or expressed as a proportion of possible user population, e. g. total prison releases in region)
· rate of service use (expressed in relation to time: fluctuations in number of users per week or month or year, and/or expressed in relation to total populations: relative to prison releases in region (which itself can fluctuate over time))
· trends in service use (broad changes in rate over specified time periods, down- wards or upwards).
This type of method may involve the use of formal pre-set questionnaires (e. g. asking people specific questions which have a limited number of optional answers: yes/no), as well as drawing upon formal records (e. g. the use of official prison statistics). Quantitative method thus lends itself to large-scale studies and statistical collection (Walter, 2013a).
Qualitative
This type of data collection is basically concerned with understanding. It is about gaining information about how people feel about things, and what their perceptions are about particular issues. It provides for more in-depth appreciation of how people make sense of their lives, and emphasizes their accounts of post-release experiences and service provision.
While the method is generally based on some form of interviewing, the findings may nonetheless be presented in statistical form (e. g. of the 50 people we spoke with, 25 thought that a pre-release plan was a good idea).
Qualitative methods generally involve a combination of description (based on observation) and relatively unstructured interviews (based around certain core questions or themes). The point is to gain information which will enable the researcher/evaluator to better appreciate the social contexts and social processes which inform how people feel about themselves, and their issues of concern. As such, qualitative method tends to be oriented toward small-scale research, involving interactions with small groups and in local contexts (Bartels and Richards, 2011).
|
|
|
Interpretive
This type of data collection is basically concerned with critical analysis. It is intended to be a form of reflection on the social meaning of official documents, existing sta- tistical collections, policy statements and media reports.
This form of research/evaluation is concerned with the identification of the per- spectives which underlie post-release issues. It is an attempt to expose the assumptions, discourses and ideological propositions which are embodied in post- release studies and policies. The ‘data’ which is collected is, in effect, the meanings assigned to particular forms and kinds of information, and the theories which are
implicit in the language used, and concepts employed, in the documentation of an issue or trend.
The contribution of this form of data collection, therefore, is to provide critical appraisal of taken-for-granted assumptions, and thereby to open the door for alterna- tive explanations, programmes for action or suggestions for reform (Hough, 2010; Tilley, 2000).
DOING EVALUATION: WHY AND FOR WHOM?
Different projects have different goals and may be informed by very different values, concepts, intentions and rationales. The main point of any particular intervention may range from trying to foster better processes of community interaction through to providing a substantial change in local crime rates.
Each project will, therefore, have its own focus and rationale. These, in turn, imply the use of evaluation criteria that best suit the purpose and context of the project. The model of evaluation that is used needs to be relevant and useful as it applies to specific types of projects or strategies. Choosing a particular evalua- tion model is thus inextricably linked to the nature of the project or programme itself.
There is a need to know concretely whether or not something is making a differ- ence. However, the criteria for success (from an evaluation viewpoint) should not be rigidly assumed either. For example, learning from errors is often a valuable experi- ence in its own right. It is also a normal part of developing the skills and methods that will enable better work in the future. A ‘mistake’ is an error that is not fixed.
Also, projects are not static but are in a continual state of flux or change (e. g. changes in personnel or in methods of service delivery). Process or continuous evaluation, therefore, is important to monitor such changes over time.
Who an evaluation is intended for (e. g. a government agency or third-sector organization) and why it is being carried out (e. g. to enhance service provision, or as a requirement for a funding agreement) are important questions, as is who is doing the evaluation (e. g. the agency itself or external evaluators). The intended outcome of the evaluation will also vary, from measuring the administrative competence of a funded agency through to identifying the future strategic goals and objectives of an organization. The values that underpin evaluation may mainly orient toward effi- ciency and effectiveness or place the greatest stress on social empowerment.
The diverse purposes, values and approaches to evaluation mean that, in practice, there will be considerable variation in how evaluation is carried out, and how it will be received and/or used. Evaluation is, therefore, an inherently political exercise. It can be managerialist and politically conservative in some instances, such as when government bodies demand ‘value for money’ from funded agencies offering mini- malist client servicing. On the other hand, it can be used progressively to extend
|
|
|
models of participatory practice amongst social reform-oriented agencies and thereby to enhance existing service provision. Evaluation is not intrinsically ‘good’ or ‘bad’. It all depends on what it is for, who is funding it, who undertakes it and under what conditions, and what the ultimate objective is.
|
|
|
