5.5.5 Evaluative Research

Evaluation is an area, that, I believe, organisations always need to be attentive to. It is inevitable that, over time, needs, staff, families, communities and society at large (and many other factors such as expectations of funders) will change.

While it is very affirming to reflect on our work from time to time in a positive manner among ourselves it is far more valuable, and indeed interesting to have an outsider undertake an evaluation.

Sometimes, in the hotel/catering/restaurant industry inspectors descend unexpectedly on an establishment to check kitchen cleanliness, health and safety, hygiene and general service.  I often wondered if inspectors arrived without warning into a family support agency and held up a mirror to our work would they find that we do what we say on the tin.

Now a very valuable form of evaluation of any programme is the uncensored opinion. 

While the uncensored opinion runs the danger of containing within it distortion brought about by a person’s own prejudice or prejudices – very often arising from their pain and emotional distress – there is often truth contained within it that is a very powerful comment on our effectiveness or otherwise.

Getting back to evaluation, there are links between research and evaluation but there are very distinct differences also. There are usually different reasons why they are done, and different results arise from both.

Evaluation is a kind of research into effectiveness, or into the methods of work of whatever is being evaluated.

Evaluative Research is, of course, not a new term, but it’s a term that I rarely see used. Remember the formative assessment that I described in a previous Sub-Chapter? Evaluative is similar, and implies formative evaluation.

That is 1): the evaluating includes those who are being evaluated, in real time, 2): the evaluation is written or presented in words and/or media that ordinary people understand, and 3): potential benefits are disseminated to those who matter most with the goal of improving their lives, i.e. alleviating their suffering.

I believe that 1), 2) and 3) are very important in what we want to achieve in our field of interest because what works and how it works are of huge importance. In fact, arguably, they are the most important things worth researching in respect of alleviation of suffering. 

Of course there is a need for occasional statistical research, or the indirect type research that I referred to above, and the academic research that universities love to do but which has little immediate impact on alleviation of suffering, but that type of research should constitute the minority of research – not the majority as it does now. 

In Evaluative Research I believe that it is very important that what is working well is affirmed and then grafted onto what is working well elsewhere – and if something completely new is gleaned, great!

Because of the complexity, the non-linear nature of growth, and the nature of the involvedness of the practitioner in the work, it is often difficult to come up with something really new.

But, because of the challenge of scaling-up processes that are culturally different, the grafting-on-to-what-is-working well-elsewhere (and existing-wisdom) will be an area where creativity, originality and newness may be really needed.

I can state with some confidence that much of what has been developed in the recent past in Ireland from research undertaken has not really led to something new.  The reason I say this is that the developments that came from the research done have rarely accommodated the chaos that I referred to earlier – if they had, I would be more inclined to class them as new.

If researchers journey (that is, immerse themselves mentally, physically and emotionally) with children and very distressed families in their enquiries there will be a far higher chance that solutions will break through the mainstream academic, educational and/or medical model paradigms that constantly try to put order on chaos but continually fail because the individuals who are the subject of the research cannot find true, felt, meaning in the solutions proposed. (How individuals find meaning and the impact on them of finding meaning, referred to earlier, is relevant here).

Also, in the journeying, it is more likely that the research will have an emotional impact on the researchers thereby optimising the probability of congruence in the outcomes, recommendations, conclusions etc.

Probably the most expensive example of the myth of newness in the area of child protection (which is the subject matter central to this book) over the past 20 years in Ireland has been the disbandment of the Regional Health Boards, the setting up of the HSE, and then the setting up of the dedicated child protection agency Túsla, added to the constitutional change in respect of the rights of children that arose from the Children’s Referendum in 2012, and mandatory reporting in 2015.

All that expense and ‘newness’ – yet most families and ordinary community workers (and I’m really trying to be fair here) would genuinely struggle to discern any noticeable change in the quality of the service that the most vulnerable children in our communities are getting over that time.

I will refer once again to the circular development that I mentioned earlier, where historical or even ancient wisdom is identified within the new development by a keen observer, and/or the new development is embedded in old wisdom and folk knowledge.

This description of emotional development fits in very well with our root foundations that we identified in the Chapter on the Universal Theories of Change.  After all, like gravity, time etc., in the physical world, the root foundations, with their high impact – low noticeability characteristics are central to our being in our emotional world, and have been as long as we have been human.

In fact, I believe that knowledge of them (whether conscious or unconscious) was the basis for the wisdom, stories, legends, parables etc. in ancient texts and the same knowledge always has been and still is hugely influential in art, music, drama, songs and folk memory in general.

So how do we research the extent to which emergence is nurtured, identity is honoured and fostered, the centrality or otherwise of relationship, the extent that affect is thought to be important, whether or not integration is modelled in an agency, and/or whether or not there is awareness of and thought put into phenomena such as how relationship, love and time are important, all of which we have identified as root foundations in our Universal Theory of Change.

When I am evaluating a new initiative or programme – and when I say evaluating I could mean forming an opinion from a quick visit and savouring the atmosphere or conducting a more forensic examination of all its aspects – I try and look at it through the eyes of not only the most distressed person that I know, but a person who, in the opinion of the vast majority of practitioners, is making no progress.

If I feel that the organisation will really make a difference in that person’s life, I might consider it as something that if I won the lottery (after paying off everyone I owe money to) I might consider investing in.

I will describe what I might see in the next post.

At the end of the last post I said that I’d describe what I’d like to observe in an organisation that is committed to engaging with and supporting people who are in our Focus Group and who have, perhaps, all the characteristics of same.

In my evaluation, whether brief or thorough, my first inclination is to assess how much I feel moved, and in what direction, by the people who are working within it.

In this, I immediately notice an atmosphere, an ambiance, or mood.  Do I feel welcome, or at home? I don’t have to try and do this. But I do take note of it. 

You may remember the description of the Sun in the Chapter on Cause and Effect – I am going to revisit it here to describe how I come to a decision as to whether or not I think that an atmosphere in a workplace will be attractive in the long term to the very hurt person who is making no discernible progress.

I described the difference between the objective (the description of the Sun that is full of facts and what sunlight does to the Earth) and subjective (how we are affected by the Sun, whether it makes us feel good or bad, safe or worried etc.)

I also said that one can apply the objective/subjective descriptions to a song, a rainbow, a competitive match, a decent pint, or thousands of other phenomena (both natural and man-made) that affect our feeling state on a daily basis.

If, on visiting an organisation, I perceive an atmosphere where staff appear to appreciate both, with a little bias towards the subjective then I feel that the very hurt person would have a good chance of finding the environment attractive. The subjective often manifests in people having strong opinions on things – they may not be right (or, indeed, wrong) but they have passion. It also manifests in people being curious, being affected, and taking personal interest.

Because of those factors, people will usually be better able to manage chaos, uncertainty, beauty, probability, ambiguity, indirectness, spontaneity, joy and many of the traits that may be part of the very hurt person’s family culture.

And how do I perceive that atmosphere?

Well, the principal indicator is the warmth of the welcome, followed by the ease and informality with which people communicate with each other and with visitors.  I also note whether the structure appears non-hierarchical (or not) – always a good indicator of at ease communication.

Finally, if I believe that there is two-way knowledge flow and glean that people are assessing the organisation as much as being assessed I will be more attracted to the agency because really this is the acid test for all organisations that aim to work with people in great distress.

But if I get an immediate impression that the general atmosphere leans too far towards the objective, and the objective will always trump the subjective in decision making, then I’m more likely to have the opinion that while the very hurt person might initially be attracted to it – the attraction will not survive the inevitable challenges that the person brings when the honeymoon period is over!

No matter how much we might like to think otherwise, the assessment that the person looking for help does of our agency is a lot more important than the assessment we the practitioners do of him!  (And who assesses the value of assessment)?

In the final post in this Sub-Chapter on Evaluative Research I would like to look again at stepped/linear and circular type development – in the context of a child growing up.

In his growth, we can evaluate his progress by measuring the extent of his growth by (at suitable intervals) measuring his height, weight, girth, heartbeat, eyesight, hearing, ability to walk, run, sprint, jump etc.  Let us say we did this at daily intervals over, say, 20 years, and then collated all the figures and graphically portrayed them on a page or computer screen.  We could use various forms of data analysis to paint a fascinating representation of his physical growth over 20 years.  (I am sure that this has already been done).

Then, in the formal education context, we could track his progress from being able to read the cat sat on the mat to making sense of obscure references in difficult-to-understand poetry.  We could track his being able to calculate 1 + 1 = 2 to his understanding of advanced mathematics.  We could track his ability to know the difference between a circle and a triangle to being a master craftsman, or his ability to sing do-re-me to understand the complex forms of a symphony.  (And we do, through our education systems – in different ways – in every country in the world).

But what kind of research would we do to evaluate/measure his cop-on-ness over many years?  Is there a way of measuring how he matures, or, to put it another way, how his maturity emerges?  Do we notice how he gains in wisdom?  Do we observe how he is able to make friends?  How about his willingness or unwillingness to be patient, take his turn, delay gratification?

I propose that parents and teachers and any others in close contact with (and in particular if they have responsibility for the child’s growth) qualitatively and informally measure all these factors (and more) on a continual basis as the child is growing.

It is very unlikely that notes are taken of his progress. And there are no exams in wisdom, maturity, cop-on-ness or ability to befriend. But they are still central concerns.

In fact, between two parents, the above matters are usually constant topics of conversation – particularly if a problem is observed.  They are also discussed at length in the extended family, e.g. grandparents and interested aunts and uncles, sisters and brothers.

With Mams and Dads in their own peer groups they are usually popular subjects too. 

With teachers, conversations in the staff rooms are as much about social progress of children, the barriers they face, their characteristics and personalities as much as their academic achievements and struggles.

I mention this because growth in cop-on-ness, wisdom etc. is far more circular than stepped/linear, and the wisdom that a young teenager needs – so that she can mature – has most likely filtered down through many generations!