problems with current evaluation models

Most community arts practitioners that we know and have conducted research with are concerned by the kinds of evaluations that they are asked to do because:

Evaluations are based on one-off projects.

While it is true that some people do undergo remarkable changes in a few short weeks – they’re apparently transformed by the experience of participation – this is not the case for everyone. For some people, trying to get them to say what benefits they have gained at the immediate end of a project may actually reveal little. All that they may be able to say at the end of one project is that they had a good time, or that they’d quite like to be involved in another one.
People respond differently to the experience of community theatre, and for some people, some of the kinds of goals that funders are interested might just take a lot longer than one project. But this continued involvement may or may not happen, as it is very tied to funding, and there may well be a considerable time lapse between one project and the next, even if a repeat is funded.

They use outcomes which are hard to measure.

A lot of the things that funders looks for are big and general – a sense of well-being for example. What do you look for if you are trying to see if someone has ‘better well being’ than they had before they were involved in a community theatre project? And how do you know if it actually does generalize to other parts of their lives?
Evaluations typically rely on a few crude indicators and what participants say. But sometimes it is just plain hard to articulate what you’ve got from something. A lot of learning for example happens without us necessarily realizing it at the time, and it’s only later that we become conscious of the fact that we know things that we once didn’t.

They generally ignore the artistic dimensions of participation.

Projects that are funded in order to produce education, health or citizenship outcomes work as if the theatre is merely a means to something else – rather than a benefit in its own right. Extending the aesthetic and artistic possibilities of individuals and communities is too often ignored in required evaluations.

When we look at evaluations we are inclined to say that funders’ evaluation requirements could be said to operate with some of the same dimensions as the political agendas that produce their funding.

They suffer from short termism. They expect unrealistic outcomes and demand that these be promised at the outset. This places practitioners in the position of having to spin the evaluations in order to avoid punishment/secure more funding in the future.

They operate using a cause and effect rationality. It is as if community theatre is a kind of medicine which will cure social ills. All that needs to happen is that people take the pill on offer and they will be cured of whatever problem they are perceived to have.

They prefer what we might characterise as pseudo-scientific approaches to evaluation – numbers, graphs and pre and post tests are seen as reliable and valid, even though they are equally as subject to definition and manipulation as interviews and other kinds of documentation. We aren’t against numbers of course, but they can be used wisely – or not.They aren’t automatically better than words or pictures.

We know that community theatre practitioners are keen to get beyond these kinds of shallow evaluations. We also know that some already do work outside of funding guidelines to develop different ways of documenting and understanding the effects of what they do.

How do you manage these kinds of evaluation demands? Do you have alternative practices that you are prepared to share?