Corps de l’article

Trying to Measure the Hard-to-Measure

Evaluating effectiveness [of social services] sits uncomfortably between … [the] imperative for measurement and efficiency within a market economy and agencies’ own need to reflect on their practice and respond to the views of their service users.

Cree et al., 2019, p. 599

There is an increasing need for social service providers and non-governmental organisations to measure and report on the results of the services they are providing, and furnish proof of efficiency gains and of whether they have ultimately had a positive impact on the lives of their intended beneficiaries. Public and private donors and “duty-bearers” (state and other actors with responsibility for human rights) have called in recent years for more rigorous use of evidence to prove results, as well as to inform decision-making and future investments in children and families in social services and development cooperation. This is also closely related to an increasing professionalisation of social services, using performance-based logistics, in a number of European welfare states (e.g., Albus & Ritter, 2018; Rogowski, 2011). At the same time, non-governmental organisations and service providers themselves, who in their very missions aim for a positive impact in the lives of the people they serve, strive to learn and improve through evidence measurement.

Social services, and interventions involving social development more broadly, are not easy to measure, however, and it is no simple feat to gauge their impact. In the development cooperation field, results measurement and evaluation have become standard practice, partly as a result of public administration reforms aimed at greater transparency regarding public spending (e.g., in the late 1980s and 1990s). The Millennium Development Goals were a pinnacle in this regard, and instigated a widespread shift from monitoring inputs to measuring quantifiable results (Organisation for Economic Co-operation and Development (OECD), 2014).

When it comes to social work practice, it is widely accepted that evaluation is integral (e.g., Shaw & Lishman, 1999). Nonetheless, ever since its origins as a profession in Europe, social work has been shaped by discussions related to the measurement of effectiveness and results, and that remains so today. Over the course of the 20th century, these debates were informed by studies that generally showed little positive effect from social work and lacked consensus regarding appropriate measurement models.

As the 21st century approached, evidence-based practice grounded in research and evaluation became more strongly embedded in social services. However, there is still no consensus concerning terms such as “evidence”, “effectiveness”, and “research”, and thus little agreement as to what should actually be measured and how measurements should be made. Social work professionals have concerns over being required to employ measurement criteria imposed on them by researchers, and the evaluation of their work being subject to political influences as well as management control (Davies, 1974; Shaw & Lishman, 1999). There is now greater acknowledgement of the fact that, with many definitions, approaches, and measurement methods to choose from, the measurement of effectiveness in social work requires a nuanced and critical view (Moriarty & Manthorpe, 2016; Shaw & Lishman, 1999).

When it comes to studies of the measurement of effectiveness and results, much of the research related to evidence-based social services is from Western Europe and North America. This is partly due to the fact that the social work profession in many parts of the world is still relatively young. For instance, as reported in Spitzer (2017), as of 2017 in the Eastern African Community Region there were only 44 Bachelor and Master’s programmes in social work and no PhD programme; Burundi, the last country in that region to establish social work training, did not take that step until 2004. Social work practice in the region still relies heavily on theory and models imported from the West, and there is little evidence regarding the effectiveness and results of services in different sociocultural contexts (Spitzer, 2017).

Such evidence is lacking for the two social services that are the focus of this article: family strengthening and alternative child care. Regarding the latter, there is a stark lack of data on the numbers and circumstances of children in the various forms of alternative care around the world, and especially in low- and middle-income countries (Better Care Network & UNICEF, 2009; Desmond et al., 2020; Martin & Zulaika, 2016; Nowak, 2019; Petrowski et al., 2017). Because sector-wide consensus definitions of the different care settings (e.g., residential care, institutional care, foster care, small group homes) are lacking (Cantwell et al., 2012), terms are often used interchangeably and inconsistently. This not only makes counting children across different care settings difficult (Desmond et al. 2020), but also means that research on different care settings and their suitability is not comparable (Gale, 2019). The evidence is also patchy when it comes to measuring outcomes of children who grew up in different care settings, due not only to differences in national contexts, policies, and amount of support when leaving care, but also to the wide array of research methodologies deployed, not all of which are sufficiently rigorous. In addition, many studies lack a holistic view of the child, and do not take into account contextual circumstances and previous care placements (Gale, 2019). “The consistent call for additional and improved research is a further indication of the lack of definitive knowledge regarding the comparable quality and effectiveness of alternative care settings” (Gale, 2019, p. 4).

The picture is also bleak when it comes to research on family strengthening. Also termed “family support” interventions, family strengthening services are generally understood as a set of public services aimed at improving the social and psychological well-being of families by promoting positive and healthy child development in a nurturing family environment. Families at risk are empowered to help themselves and their children through multidimensional services[1] (Daly et al., 2015; Dunst, 1995). Over the past two decades, family support services have found their place in the child welfare policies and practices of many states, and different models of family-related services have developed globally. Significant gaps remain, however, and policies to accelerate care reform through the deinstitutionalisation of large-scale residential care institutions have not always focused sufficiently on the simultaneous development of family- and community-based support (Chaitkin et al., 2017). This goes hand in hand with the need for more evidence on a global perspective of family support, how it can be contextualised adequately, and what works best for families and children on the ground; this can be achieved by including their voices and opinions in the discourse (Canavan et al., 2016; Daly et al., 2015). While there is a substantial body of evidence confirming the positive effects of broadly standardised parenting programmes in developing countries and in some low- and middle-income countries (see Knerr et al., 2013 and Rebello Britto et al., 2015 for some notable examples), there are still considerable gaps in evidence on family support interventions in low- and middle-income countries and their effects on various levels of society. In particular, there are shortcomings with respect to the conditions that are necessary for the sustainability of outcomes and services (Daly et al. 2015).

Against this backdrop, for an international organisation that provides social services and works in development cooperation, offering evidence-based and quality services for children can be challenging. By using the case example of SOS Children’s Villages International (SOS), this article aims to provide an insight into a measurement approach designed to gauge the impact of social services and related challenges. Readers should bear in mind that the authors are themselves co-workers with SOS.

Some Background: SOS Children’s Villages

SOS is a non-governmental organisation operating in almost 600 locations in 136 countries and territories across the Americas, Africa, Asia and Oceania, and Europe. The organisation provides the aforementioned social services, family strengthening and alternative care, to children who have lost parental care or are at risk of losing it (SOS Children’s Villages International, 2019), and works across a range of alternative care options, such as family-like care, foster care, small group homes, and short-term crisis care. Currently, approximately 70,000 children are reached through these services worldwide.

The most widespread care form is “family-like care”[2], which was established in post-World War II Central Europe as an alternative to the large-scale orphanages that were predominant at the time. With the growth of SOS in ensuing years, this care form was established in many countries across the globe (Honold & Zeindl, 2012). The children, who are often siblings, stay in small groups that resemble a family with a stable main caregiver. These smaller family groups are embedded in a psychosocial support structure, composed of a multiprofessional team and peer-support network. As opposed to foster care, SOS provides accommodation for a number of family groups, either clustered together or scattered throughout a town or city.

Family strengthening services for families at risk of separation have gained traction in SOS since the late 1970s as a complementary service to alternative care. In essence, the aim is to prevent child–family separation from the outset, and avoid placement in alternative care where possible. Family strengthening services have grown considerably, now reaching almost 500,000 children and their families across 112 countries (SOS Children’s Villages International, 2019). The primary objective — preventing child–family separation — is achieved through strengthening and improving parenting skills, child–parent and family relationships, and community-based support mechanisms. While this objective remains similar across all contexts, local adaptations of the services vary, depending on local resources, available partners, the social protection system, donor interests, and sociocultural realities. In this regard, some interventions are carried out by professional social workers and staff, but there are also community-based organisations with volunteer staff who carry out and coordinate formal and informal support services. The latter may include home visits; psychosocial, nutrition, and health services; capacity building; and peer support groups.

In SOS, initiatives to track the impacts of social services in the area of alternative care had already started several decades ago, most notably in a large-scale research project called “Tracking Footprints” (Lill-Rastern & Babic, 2010), which was carried out in more than 50 countries during 2002 to 2009. Young people with care experience from SOS services were interviewed through standardised questionnaires on their current life status and their experiences in care. Additionally, individual research projects and evaluations on former programme participants have taken place in and across specific countries and regions. However, until recently, there was no common approach to, or methodology for, measuring the long-term effects and impacts of services on beneficiaries and the wider community, as well as the social return on investment, in a way that allowed comparison across a variety of locations. In addition, a culture of results measurement is still relatively new in the organisation. This has led SOS to develop a comprehensive long-term social impact measurement approach for the two main services of family strengthening and alternative care, and to apply the approach in various locations across 15 low-, middle-, and high-income countries[3].

Finding Ways to Measure the Social Impact of Child Care Services

I have never been asked about my life experiences in so much detail before. In a way, I am feeling very rejuvenated to be part of this Social Impact Assessment and sharing my life experiences. I will eagerly look forward to understand about findings and conclusions of the assessment.

Young adult who grew up in alternative care, Nepal

The internal aim behind the development of a specialised impact measurement approach within SOS was to use such evaluations formatively, in order to inform service improvement for future service users and also give them the opportunity in the long term to voice their perspectives and concerns. On a higher level, the consolidation of findings across various locations enables the identification of trends and areas for learning and improvement. This type of information is very valuable for informing strategy, policy, and management decision-making, as well as further research. This section will detail the key terms, definitions, and principles used in this approach, and outline the key pillars of the methodology. Challenges and limitations related to the methodology and implementation will also be explored.

Here, the term “impact” refers to the long-term effect of an intervention on people’s lives. This is reflected in various definitions of impact, such as, “positive and negative, primary and secondary long term effects produced by a development intervention, directly or indirectly, intended or unintended” (OECD, 2013).

Impact is commonly defined as part of — or synonymous with — a results chain: from inputs and activities to outputs, to outcomes, to impact. As such, it is expected that the various outcomes will result in certain long-term effects on the beneficiaries of (or participants in) the intervention, and on broader society as well. The impact evaluation essentially also tests the validity of the theory of change of an intervention. While assessing impact by collecting evidence about the contribution of an intervention to observed changes, strategies must be found to determine causal attribution and eliminate the “attribution gap” (i.e., the extent to which results are due to factors other than the activities of the organisation itself, such as collective action or other external forces) as far as possible. In this sense, impact assessment can be regarded as the most challenging part of intervention evaluation, but is crucial in order to determine what remains and what follows after a participant is no longer receiving the service.

There are multiple ways of measuring “social impact” with a variety of degrees of rigour, ranging from randomised control trials and quasi-experimental designs, to non-experimental designs (e.g., Rogers, 2014). The initial methodology designed by SOS was based on an experimental design with the use of a control group. However, during the pilot assessments in Eswatini and Ethiopia it became apparent that it was practically impossible, especially for the alternative care service, to locate a sufficient number of individuals who may have needed alternative care when they were younger and who had a minimum set of shared characteristics. In addition, it was not possible to secure a sufficient number of care leavers of other service providers where the setting was sufficiently comparable and the sampling could be done credibly. The current methodology can therefore be classified as a non-experimental mixed-methods design.

Social impact at SOS Children’s Villages (2018) is evaluated in terms of:

  1. Impact on the lives of individuals: The long-term effects of the intervention[4] on former participants, whether these individuals are still dependent children (under the care of a primary caregiver responsible for supporting and guiding their development) or already independent adults (responsible for taking care of their own development needs, and above the legal age of adulthood);

  2. Impact in communities: The long-term effects of the intervention in the communities with which the services have been interacting and working;

  3. Social return on investment (financial): A forecast of the social return that can be expected, measured in monetary terms, for the amount spent in the programme.

The assessment methodology also covers the following areas, which go beyond impact.

  • Evaluation of relevance, effectiveness, efficiency, coherence, and sustainability: Complementary to the results of A, B, and C above, the effects of the intervention are compared with predictions of the theory of change, as outlined in the programme plan or log frame (OECD & DAC Network on Development Evaluation, 2019).

  • Analysis of the contribution to selected Sustainable Development Goals

  • Additional topics, per local need

While the methodology of evaluating A to C has been defined in considerable detail, it remains subject to continuous improvement. With this in mind, external researchers conducting the assessments were asked to put forward any recommendations for the further improvement of the methodology. The approach is modular; therefore, depending on the focus area, particular elements can be conducted separately or in combination. However, the module on the social return on investment cannot be performed as a standalone element, since it relies on the collection of primary data through A and B.

Five main principles guide the methodological approach. The approach must be:

  • Fair: Compliance with ethical standards related to research with children (Graham et al., 2013) is required, as well as adherence to internal policies and procedures related to Child Safeguarding and the Code of Conduct;

  • Useful: The learning needs of service participants, community, and staff are the highest priority; balance is required between local adaptation and international comparability for global learning;

  • Participatory: The voices of former participants and local staff must be considered throughout, from design to implementation of the findings. This is accomplished through kick-off workshops with local teams to adapt questionnaires and indicators to the local context, and through results validation and learning workshops with former participants and staff; evaluation tools are tailored to the ages of child respondents;

  • Systematic: Reliability, quality, accuracy, and validity of assessments is key; there is close monitoring of assessments by regional and international offices of the organisation;

  • Independent and impartial: External researchers are selected following a national and international tender process, and are responsible for the final report.

The assessment process usually lasts from 5 to 8 months. It includes the following key steps:

  1. Locations for the assessments are determined based on an evaluability assessment;

  2. Recruitment of researchers together with local staff (could be local or international research institutions);

  3. Local participatory workshops to adapt the indicators and related scales, and the assessment tools, to the local context and respondent groups;

  4. Field phase including individual interviews, focus group discussions, and further participatory methodologies as proposed by the researchers;

  5. Participatory results workshops to validate, discuss, and take forward the findings;

  6. Regular follow-up on the implementation of the evaluation results.

The next section will focus on methodological components A to C, as these form the heart of the social impact assessment methodology.

A: Impact in the Lives of Individuals

The social impact assessment approach measures the long-term well-being of children and families who benefitted from the services of SOS in the past. The majority of former participants from family strengthening tend to be dependent children and the majority from alternative care tend to be independent adults (care leavers), although there are cases where this pattern does not apply, such as in family reunifications.

For manageability purposes, there are slightly different sampling approaches for the two services types. For family strengthening, which has a relatively large number of participants, the expected number of former participants who have left services 1 to 5 years previously should be sufficiently high that a representative sample can be drawn. Although the sampling should be based on a random selection, it needs to reflect representativeness of the entire group of former participants, at least in terms of age (children below the age of 8 are usually not interviewed), gender, type of family, and reason for exiting the service. This means that the sample should include both those who have left “successfully” and those who did not, and the balance of reasons for “exit” should, as far as possible, be representative of the intervention reality. The only usual precondition for all selected former participants is that they have participated in the programme for at least 2 years, in order to be able to attribute impact.

For participants who exited alternative care, a census is taken of all former participants who exited from the programme within the last 2 to 6 years, with the precondition that they had participated in the programme for at least 2 years. The number in a given location is usually manageable in this regard; that is, the numbers usually range between 30 and 50 former participants that meet the criteria.

Although family strengthening and alternative care are distinct services with different target groups, the social impact assessment methodology measures the impact of the services across eight common dimensions of well-being: care, food security, accommodation, health, education and skills, livelihood, social protection and inclusion, and social and emotional well-being. SOS aims to have a positive impact in all these areas of well-being: individualised service, building on case management, aims to provide or coordinate services where needed across all areas of life, and across both the family strengthening and alternative care approaches. A key underlying assumption of the methodology is that children who grow up in inadequate care situations, including low-quality alternative care, will generally not do well across these eight dimensions of well-being, a belief supported by studies from around the world (see, e.g., Bicego et al., 2003; Cluver et al., 2008; Kang et al., 2014; Lionetti et al., 2015; Skinner et al., 2013; and Whetten et al., 2011, for some notable examples).

The eight dimensions of well-being and associated indicators (see Table 1) are largely based on dimensions, indicators, and rating scales used in the existing SOS monitoring and evaluation systems and the Child Status Index (CSI; O’Donnell et al., 2014). The CSI is a tool developed for service providers to assess the well-being of children systematically, in particular to:

assess the vulnerabilities and needs of children who have been orphaned or made vulnerable by HIV/AIDS. The Child Status Index, intended for use by governments, programs, or projects providing support to vulnerable children and their families, provides a framework for assessing child well-being and creating outcome-directed service plans for individual children and the households in which they live.

O’Donnell et al., 2014, p. 1

The CSI gauges child- and household-specific information, and can be used for service planning as well as evaluation. The CSI’s indicator matrix was adapted and incorporated into the SOS internal monitoring database, which is a case management tool for social workers in the various service types. In this way, the organisation can not only track progress of children and families over time in an intervention, but can use the same assessment dimensions in the impact evaluations. As sufficient data is captured in the central monitoring system, a “counterfactual analysis” (i.e., comparing actual results from an intervention with those expected had the intervention not occurred) can also be done during the social impact assessments.

As well as the dimensions of well-being in the CSI, other indicators were incorporated into the social impact assessment framework. These were based on the existing SOS internal monitoring and evaluation system and on knowledge requirements; they include indicators related to self-esteem, happiness, employability skills, social support networks, and fulfilment of parental obligations (if the care leaver is already a parent). In addition, the questionnaires cover the relationship to the family of origin, as well as to the former alternative caregiver. These additional indicators were based on assessment areas in commonly used assessment scales. For instance, the indicator of self-esteem was composed based on the Rosenberg self-esteem scale (Rosenberg, 1965); the indicator of happiness was largely inspired by the Oxford Happiness Questionnaire (Hills & Argyle, 2002) and by The Children’s Happiness Scale, which applies to children in care, receiving social care support, and living away from home in boarding or other residential schools or colleges (Morgan, 2014). In fact, during the pilot assessments the Rosenberg self-esteem and Oxford Happiness scales were used as supplementary questionnaires. However, their use was considered too resource intensive to include as standard instruments in every assessment.

Former participants are interviewed using semi-structured questionnaires that include open and closed questions. Researchers assess their status on a rating scale of predefined indicators ranging from 1 to 4. Participants with a rating of 1 and 2 are considered to be “doing well”, whereas those scoring a 3 or a 4 are “not doing well”. The complete data set of a dependent child includes both questions addressed to the former participant (e.g., education and health), and questions addressed to the caregiver (e.g., livelihood). For an independent adult, all questions are addressed to the former participant. Emerging and unexpected topics are captured by the researchers and further explored through participatory focus group discussions and results validation workshops with the former participants and staff. This enables a qualitative component of the methodology, permitting the analysis of open-ended answers and triangulation with the quantitative results according to the indicators.

Table 1

Overview of Dimensions and Indicators Measuring Impact in the Lives of Individuals

Overview of Dimensions and Indicators Measuring Impact in the Lives of Individuals

Note. See Willi et al., 2017a for guiding questions related to the dimensions and indicators.

-> Voir la liste des tableaux

The assessment results are also meant to be benchmarked against comparable national and regional statistics wherever possible. Generally, comparative data is more readily available for the following indicators: young people neither in education, training, nor employment (NEET); educational attendance; educational attainment; income; and employment status. While national averages are a good starting point, given the backgrounds of children who came from disadvantaged families or who grew up in alternative care, regional averages or certain lower quintiles (of, e.g., the income distribution) may be more appropriate reference points depending on the indicator. The contracted researchers are asked to check to what extent and for what development areas reliable data might be available. It is recommended that the check on reliability be done with local experts (either from the programme or community stakeholders), or with national labour or other statistics experts. In essence, this means a “virtual” comparison group is constructed by comparing participant outcomes with comparable groups in the population.

Prior to each evaluation, researchers are required to analyse the pool of services received by the former participants; if participants were not supported in certain areas (i.e.,, accommodation), impact cannot be attributed in those areas. Moreover, the organisation works in partnership with local stakeholders; therefore, the impact can never be attributed to the organisation alone (this is known as the “attribution gap”). Efforts have been taken to minimise this uncertainty by, for instance, taking the role of partners into consideration when calculating monetary impact and by involving them in the assessment (see SOS Children’s Villages International, 2019; Willi et al., 2017a).

B: Impact in Communities

The roles of sociocultural contextual factors and systems, such as communities with services and supports for families and children, and cultural contexts that provide positive standards, customs, norms, relationships, and supports, are crucial in the promotion of resilience (e.g., Masten, 2014) and thus are important to consider when measuring the social impact of a social service. Therefore, SOS decided to go beyond assessing its impact in the lives of individuals to assess the effects of its work in communities, since it aims at building up — strengthening — a stable network of stakeholders to ensure sustainability and adequate mechanisms to support children who are either without parental care or at risk of losing it. Consequently, a second key component of the social impact evaluation methodology is the assessment of key dimensions related to community awareness, social support mechanisms, and questions around the sustainability of services, as highlighted in Table 2.

Table 2

Dimensions and Indicators Measuring the Impact of SOS in Communities

Dimensions and Indicators Measuring the Impact of SOS in Communities

Note. See Willi et al., 2017a for guiding questions related to the dimensions and indicators.

-> Voir la liste des tableaux

Apart from the services per se and their potential influence on the community, individuals who participated in the services may also contribute to the community in various ways, which are also measured in the assessment approach. See Table 3.

Table 3

Dimensions and Indicators Measuring Contributions of Former Service Users to Community Wellness

Dimensions and Indicators Measuring Contributions of Former Service Users to Community Wellness

Note. See Willi et al. 2017a, for guiding questions related to the dimensions and indicators.

-> Voir la liste des tableaux

Information about the community impact of services is collected in a desk review of the initial situation in the community through needs assessments, baseline studies, research reports, and similar, and of the current situation in the community through national and local statistics. Information is also collected through semi-structured interviews with staff, community stakeholders, and former participants, and through further methodologies as proposed by the researchers (e.g., participatory focus groups, used in most cases). The ratings of the indicators (also on a scale of 1 to 4) are based primarily on the findings of the primary research. The ratings are supplemented by illustrations or case studies, providing more qualitative information to explore the “how” and “why” of the impacts. Ratings should also be made for the situation before and the situation after the engagement of SOS services with the community.

The above dimensions were developed based on consultations with practitioners across the organisation. These dimensions are also adjusted to the local context in a participatory workshop, as they strongly depend on the local context, and the existing child welfare system and related mechanisms; certain dimensions may thus not be fully relevant in a given context.

C: Social Return On Investment (SROI)

During the last decade, the use of social return on investment (SROI) calculations for development cooperation and social services has become more widespread. First pioneered and developed between 1996 and 2001 by the Roberts Enterprise Development Fund (now REDF), the approach seeks to express in monetary terms the values created by social organisations and returned to communities. In ensuing years, the approach was further refined by the New Economics Foundation, which is based in the United Kingdom, and various practitioner groups. Today, there are multiple frameworks and approaches to measuring SROI (Brouwers et al., 2010).

The SROI aims to measure social impact in the community in financial terms. It compares the total costs of services with the projected financial benefits to society. In contrast to the return on investment (ROI), which is often measured in a business context, the SROI does not account for the profits accruing to the organisation itself, but rather evaluates the benefits accruing to its beneficiaries and society at large.

As a social development organisation, SOS lacked expertise in the measurement of SROI. This was addressed by working with the Boston Consulting Group (BCG) in the development of the SROI aspect of the SOS social impact assessment methodology. BCG, an international consulting firm, had developed an SROI framework for complex interventions in the development cooperation field (BCG, 2008; Villis et al., 2009). This framework was then adapted to the specific context of the services SOS provides, enhancing the assessment of impact on the lives of individuals and on communities.

Figure 1 outlines the key parameters that are measured to gauge the benefits and costs of interventions.

Figure 1

Components of SROI Calculation

Components of SROI Calculation

Note. Adapted from Willi et al., 2017a.

-> Voir la liste des figures

Various factors are applied to maximise the robustness and validity of the calculation. For instance, the approach assumes that related benefits to society can only be expected to be sustained by former participants who have positive educational and livelihood outcomes, these being reliable predictors for the economic success of individuals over time. Therefore, when calculating the social benefits of an intervention, only the benefits for those individuals who are doing well in these two areas of well-being are monetised, whereas the costs of all participants are used in the calculation, including costs incurred for former participants who may not be doing as well as hoped in terms of their job status, educational outcomes, and livelihood. Further factors include: the application of a discount factor, which allows for comparison of monetary flows occurring at different times; an attribution factor if other service providers are providing similar or associated services in the same location; benchmarking with local income and education levels, as well as other comparable external data; and finally the application of three scenarios to ensure sensitivity of results, in particular for those benefits where there is no definitive projected value (for more information, see Willi et al., 2017a).

It must be noted that non-financial benefits in the lives of children, their families, and their communities that cannot be quantified are excluded, but remain important. In this sense, SOS never conducts the SROI calculation as a standalone element and always includes it as part of a broader impact assessment.

Challenges and Limitations

Applying this evaluation approach across various locations and contexts while working with a wide array of different researchers has not been without challenges, and certain limitations of the methodology have become evident. As mentioned previously, SOS is taking the approach that measuring long-term effects is a continuous learning process, meaning that the methodology is adapted and refined with each assessment.

Individual Impact

Predetermined well-being dimensions: By predetermining the assessment areas, the participatory scope of the assessment is limited, as is the potential for unexpected findings that are sensitive to the local context in terms of social and cultural specificity. In more recent assessments, children and parental caregivers have been asked to validate and rate the different areas of well-being, as well as to suggest additional ones. However, in future, this element will have to be streamlined more consistently in the methodological approach to ensure that the voices of children and young people are also considered in the design.

Trade-offs between the range and depth of the dimensions: Covering eight dimensions of well-being, and multiple indicators, unfortunately limits how deeply each can be explored and assessed if the individual interviews are to be of a manageable length, especially when interviewing children. This means that the assessment of dimensions such as social protection and inclusion, and social and emotional well-being, is only possible on a high level. Also, certain topics related to a particular target group (e.g., those that particularly affect care leavers) cannot be explored deeply.

Rating dimensions and their interpretation: Dimensions related to emotional well-being and protection are particularly difficult for interviewers to rate; in-depth training is required. In addition, an adult’s or child’s description of emotional difficulties or problems can be strongly affected by cultural or social expectations, language, and understanding. Therefore, it is very important for the care workers to discuss the ratings and possible nuances related to child behaviour with the researchers to ensure a consensus on which types of behaviour are “normal” or “worrisome” in a particular context (see also O’Donnell, 2014).

Interviewing children with their parental caregiver: Having a parental caregiver present may sometimes influence the answers children give related to their care and well-being. However, since it cannot always be guaranteed that the recruited researchers are experts in interviewing children of various age groups, this approach was favoured in order to protect the rights of the child.

Working with different researchers: Working with different researchers, with various disciplinary backgrounds, knowledge, and skills has particular challenges and requires close monitoring to ensure methodological consistency for the consolidation of results across different assessments, as well as to maintain general quality standards.

Reaching former participants and potential for bias: Former participants cannot always be located due to missing or incorrect contact information or to having moved. An analysis of the effects of these missing participants on the representativeness of the sampled participants vis-à-vis the whole population is a requirement. For former participants who do not wish to take part in the evaluation, the reasons for their abstention must be recorded and included in the analysis of the findings, as far as possible, in order to take account of potential bias.

Securing baseline data: In some locations, securing baseline data for the former participants has been difficult and, in some cases, the data had to be reconstructed by the researchers.

Data constraints when benchmarking: The ability to benchmark the results vis-à-vis comparable local statistics is sometimes limited depending on the availability and robustness of local data collection frameworks.

Consolidating findings from different contexts: The consolidation of results across locations carries the risk of leading to generalisations and misinterpretations, and is therefore only possible to a limited extent. One benefit of SOS is that the services themselves are relatively comparable across locations, due to their similar scope, target groups, and service designs. In this phase, the focus is on statistical consolidation of the indicator ratings across the dimensions, as well as qualitative content analysis outlining high-level trends, further illustrated by anecdotal examples related to individual case stories.

The methodology is not all-encompassing: While the analysis of contextual factors and their effects on social services should be taken into consideration, the extent to which this can be done is restricted by the limited timeframe.

Impact in Communities

Measuring community impact where SOS is still engaged: In communities where the organisation is still operating, researchers do not have the advantage of being able to assess the situation after exit. In these cases, researchers are asked to carry out a comparative analysis of the initial and current situation based on a methodology of their choosing.

Social Return on Investment

Use of the SROI: For many practitioners on the ground, the SROI is controversial, as many benefits, such as self-esteem, happiness, and family relationships, cannot be monetised. Therefore, it is crucial that the SROI component of the evaluation does not play an exclusive role in the methodology and in the communication of the findings. Further limitations are that calculating the SROI is resource intensive and adds a layer of complexity to the assessment.

Attribution of the findings to SOS: There is an unavoidable degree of uncertainty regarding the extent to which the impact of an intervention can be attributed to the work of SOS and its partners, and how much results from external factors. Attribution of impact is more credible when it comes to the alternative care services, since these are more comprehensive (providing 24-hour care and a range of support services directly to a child) than the family strengthening services. For family strengthening, the attribution is more difficult to determine due to the more flexible service approach and the involvement of more stakeholders. Therefore, in spite of the calculation of an attribution factor in the SROI analysis, some degree of uncertainty will necessarily remain in the attribution of impact to the organisation. The results are therefore largely built on the experiences of the beneficiaries and stakeholders themselves and what they regard as the impact of the services, as well as the observations of independent researchers.

Overall Impact

Resource intensity of the impact assessments: The social impact assessments are quite resource intensive in terms of human resources and financial costs.

Current and Future Outlook

In general, the social impact assessement methodology has generated valuable learnings and insights for service development and improvement, locally, nationally, and more widely across the organisation. Following each assessment, action plans are formulated by the affected national organisation to take forward the relevant recommendations in the assessment location as well as other locations. The findings are also used for fundraising, external communication, and advocacy. At the same time, the methodology is contributing to capacity building in results measurement across the organisation.

The results are regularly consolidated on an international level. So far, two consolidation rounds have taken place. The first, in 2016, brought together the findings from social impact assessments across seven locations in Africa and Asia: Abobo-Gare (Côte d’Ivoire), Dakar (Senegal), Hawassa (Ethiopia), Kara (Togo), Mbabane (Eswatini), Surkhet (Nepal), and Zanzibar (Tanzania; Willi et al., 2017b). The second consolidation took place in 2019, bringing together the findings of an additional eight assessments in Abomey-Calavi (Benin), Santa Cruz de la Sierra (Bolivia), Sarajevo (Bosnia and Herzegovina), Vicenza (Italy), Tete (Mozambique), Bethlehem and Rafah (Palestine), Lima (Peru), and Nuwara Eliya (Sri Lanka; Willi et al., 2019). This consolidation also brought together the results of the previous impact surveys (“Tracking Footprints”) for the occasion of the organisation’s 70th anniversary (Willi et al., 2019)[5]. In terms of the international level, the findings have informed the development of the organisation’s strategy towards 2030 and programme policies, as well as the determination of the global research agenda, which aims to improve the organisation’s global impact.

While the existing methodology has its limitations, it is continually being refined based on learning through experience. In this sense, it is also beneficial to work with a range of different researchers, as this helps to bring fresh perspectives, insights, and ideas to improve the methodology.

Currently, four additional social impact assessments in Kyrgyzstan, Indonesia, Nicaragua, and the Philippines are ongoing. Future possibilities include returning to some countries where assessments have been previously carried out, to compare the status of former participants over time and to assess service improvements.

Moreover, it is envisaged that the methodology can be used to evaluate results in and across various types of services and with different service providers. In Latin America, some assessments have already evaluated different approaches to family strengthening. When it comes to formal alternative care, so far only one type has been assessed. It would be interesting to evaluate different types of alternative care, such as family-based care and other alternative care options, to better inform the efforts of SOS and other service providers to engage with the range of different care options, especially in the context of care reform. As more evidence is gathered on different responses to the situation of children without adequate parental care, it becomes increasingly possible to support children to stay in the care of their families, and to offer the most suitable alternative care for those who cannot.

“What counts in life is not the mere fact that we have lived. It is what difference we have made in the lives of others that will determine the significance of the life we lead.”

Nelson Mandela