Introduction
After decades of conflict, an agreement in 2005 set in motion the processes that would lead South Sudan to become an independent nation-state in 2011. After an initial period of optimism, conflict re-emerged; first over control of oil resources in 2012, and then in the form of a civil war, starting in 2013. The conflict has caused the displacement of millions of people internally and internationally as refugees. Compounded by the lack of basic infrastructure and services, limited capacity, and minimal governmental presence outside of Juba, South Sudanese experienced widespread food insecurity, leading to emergency and famine conditions. The international community responded to the crises with humanitarian assistance. South Sudan has become one of the world’s largest humanitarian emergencies. Over the duration of the conflict, optimism and support of the international community shifted, to the extent that many donors were/are no longer willing to support the government. As violence escalated, the government and specific individuals were sanctioned. The Government of South Sudan (GoSS) responded with restrictions on non-governmental organisations (NGOs) and levying of unreasonable fees for NGO personnel visas. After multiple attempts of peace negotiations, forms of power-sharing agreements have been attempted, yet these remain fragile and contested.
One of the challenges for donors and organisations seeking to work in such a complex operational environment is the lack of available evidence to support decision making alongside the lack of experiential lessons for learning from practice. On the former, basic data is absent in nearly all sectors; 45 indicators in UNDP’s Human Development Index were missing as of 2018. On the latter, as far as we are aware, no documents have brought together lessons from multiple donors and NGOs, and thus learning has remained donor-specific or project-based. A further challenge, as evidenced in the evaluations analysed in this study, is that actors have yet to act upon lessons they have learned, suggesting that other barriers need to be overcome in implementation. As of 2018, there were 194 organisations registered with the United Nations Office for the Coordination of Humanitarian Affairs (UN OCHA) in South Sudan (69 national, 36%; 125 international, 64%). Of the activity that has taken place in the pre- and post-independence periods, there is a wealth of lessons that have been learned about how to work effectively and appropriately in this complex operational environment. However, no available publications have reviewed or synthesised evaluation reports to enable learning to transcend individuals, organisations and donors. While recognising that evaluations are not always well done (Raifman et al., 2018), nor always made public, this article seeks to address existing knowledge gaps by identifying all publicly available evaluation reports, analysing the landscape of evaluation in South Sudan, and synthesising the lessons learned. In so doing, this article attempts to synthesise evidence and lessons from evaluation reports, as a means to support better-informed decision making and to facilitate learning across donors and organisations. In addition to applicability in South Sudan, the lessons may be useful for engagement in other fragile and conflict-affected contexts.
Unlike academic repositories, there are few comprehensive databases for ‘grey’ literature, including evaluation reports. The available resources regarding evaluations in South Sudan suggest that only a few evaluations are publicly available. For example, one evaluation database (discussed in more detail below), which is updated to 2015, identifies only three evaluations related to South Sudan. In 2016, the Norwegian Agency for Development Cooperation (Norad) compiled an ‘Evaluation Portrait’, outlining 24 evaluations (covering 2010 to 2015), which appears to be the most extensive collection of evaluations conducted in South Sudan (Sorbo et al., 2016). Given the wide range of humanitarian and development activity in the country, at the outset of this study it was assumed that many more evaluations have been published, but innovative methods would be required to identify them. Due to the challenges of identifying and tracking evaluation reports, this article presents methodological reflection and learning regarding how systematic reviews and syntheses of this nature can be conducted. The article begins with a detailed outline of the methods, specifying which approaches worked more effectively than others. The results section is presented in two parts, the first is a landscape analysis of the trends of evaluation reports in South Sudan, while the second presents a synthesis of lessons learned categorised by thematic areas. A discussion presents some reflections on both the methods and the findings, in light of enhancing the usefulness and impact of evaluation reports.
Methods
This assessment of evaluation trends and synthesis of learning from South Sudan focuses upon evaluation reports (often referred to as ‘grey literature’). This differs from studies that focus exclusively on peer-reviewed academic literature, as is done in many systematic reviews. The period of study is primarily that following independence, from 2011 to 2018. However, evaluation reports from the transition period (2005–11) were included for two reasons; firstly, some projects began in the transition period and ended after 2011, when an evaluation was conducted, and secondly, that projects done in what was then Southern Sudan (and can be identified as such) offer valuable insight into a range of relevant fields, from agricultural interventions to donor granting lessons. Although the 2005–11 period was included within the inclusion criteria, a limited number of reports were identified (11 evaluations of a total of 98; see Appendix for complete listing) and few contributed specific insight for this synthesis (three evaluations). This assessment focuses upon the transition and independence period because previous to that, projects were approved in Khartoum and often had coverage that extended beyond what is today South Sudan (particularly projects operating at scale). Geographic divisions of the past do not necessarily align with those of today, which is compounded by the fact that evaluations often do not distinguish between what is now South Sudan and Sudan in their reporting. The use of this inclusion criteria ensures that the evaluations assessed are relevant to the study area.
Focusing upon evaluation reports presents some limitations. One might suggest that this presents an ‘inverse publication bias’. We recognise this, however, given the amount of material combined with the lack of engagement with it in the academic literature, we have opted to focus upon these reports. Future studies might conduct reviews of academic works, about which a comparative assessment to these findings could be offered. There are two key limitations when aiming to use evaluations: (1) a lack of peer review or quality control, and (2) a lack of platforms or databases that identify evaluations, as is available for academic literature (such as the Web of Science or Google Scholar). For the first, researchers could conduct their own quality assessment of evaluations and exclude reports. This was not done because many reports do not provide sufficient detail of methodologies to standardise the assessment of evaluations. One reason for this is that such details are sometimes listed in Terms of Reference or within Appendices that are not included in the report or have not been made public. A further barrier is that not all evaluations are made public, often because transparency opens donors and implementing organisations to criticisms. While more donors have enacted policies to make all evaluations public, this is less common for implementing organisations. As a result, we were only able to critically analyse the reports identified. The expert-driven process of synthesis allowed for those reports with more rigorous methodologies to be emphasised over those with questionable methodologies and/or that did not collect primary data. Regarding the second limitation, this presents a methodological challenge for research using evaluation reports. The following section outlines the experimental methods utilised in this article, particularly in identifying evaluation reports, which highlights the difficulties of obtaining reports in a systematic fashion. The approaches attempted to amass as large a set of evaluation reports as possible in order to assess trends and synthesise findings. It is noteworthy that South Sudan presents some unique methodological challenges. One reason for this is that it is a nation that has relatively recently become independent, meaning that the identification of material was not as straightforward as it may be for other fragile and conflict-affected states. Synthesis efforts conducted for other contexts may not need to adopt all of the methods described below in seeking to identify relevant evaluations.
Finding Evaluations
The first point of departure in seeking to obtain evaluation reports was a joint project between the Centres for Learning on Evaluation and Results (CLEAR) and the Centre for Research on Evaluation, Science and Technology (CREST), respectively based at Witwatersrand University and Stellenbosch University, both in South Africa. The project aimed to develop a database of articles and reports on evaluation from Anglophone Africa.1 While the database did collect over 9,000 academic articles and 2,600 evaluation reports, South Sudan does not feature strongly in the database. The three reports it contains on South Sudan focus on refugee assistance programmes outside of the country. At the time of querying the database (February 2018), it was updated to 2015. The low number of results for evaluation reports was surprising, because the database included evaluation reports from 3ie, ALNAP, DANIDA, DFID, government websites, IFAD, MEASURE, Norad, OECD, SIDA, UNAIDS, UNEG, UNICEF, USAID and the World Bank. The academic literature identified in the database was also limited for South Sudan, finding only 14 articles, however of these only four focused on South Sudan primarily, while the majority made minor or comparative reference to the country.
The next set of methodological activities included those that aimed to identify organisations that operate in South Sudan. UN OCHA reports on operational presence of NGOs and international agencies were analysed, and a list of all organisations operating in the country was compiled. A total of 194 organisations were identified as being registered with UN OCHA between January 2014 and February 2018. Organisations were categorised as being national or international, as per the UN OCHA categorisation (69 national, 36%; 125 international, 64%). Next, an attempt was made to contact all of the identified organisations in order to request the sharing of evaluation reports. Of the 194 organisations, a total of 166 (86%) were contacted regarding their organisation’s evaluation work and availability of reports (only those organisations for which contact information could not be identified were not included). There was a 23% response rate to these inquiries (38 of 166), however only four of these organisations were able to provide evaluation reports (the majority that did reply had not conducted evaluations of their activities or had not made the reports public). Given the time-intensive nature of this component of the process, and the limited results it offered, identifying and contacting individual organisations would not be undertaken in future efforts of this nature. Attempting to contact all organisations operational in South Sudan was a fruitful experience, as it highlighted how few organisations are conducting evaluations and that the evaluations which are done are often not available to the public. This result highlighted how a synthesis of existing evaluations may support these organisations to make better-informed decisions.
Thereafter, the following steps of identifying evaluation reports from South Sudan were undertaken, which were more informal in nature. Requests for evaluation reports were sent to two mailing lists of development practitioners (Pelican, which is UK-based, and the Canadian Association of International Development Professionals). This process resulted in the sharing of 15 reports, some of which had been previously identified. ReliefWeb, DFID, and USAID’s Development Experience Clearinghouse websites were also explored, as well as the websites of other major donors and international NGOs, in order to acquire evaluation reports specific to South Sudan (e.g. FAO, UNICEF, WFP). An ‘Evaluation Portrait’ published by Norad (Sorbo et al., 2016) emerged during this process, which highlighted 17 previously unidentified reports. After attempting these processes, a degree of ‘saturation’ emerged, as no new reports were being identified that had not already been identified via the search processes outlined above. In total, 98 reports were identified that focused on activities within South Sudan that were efforts to evaluate humanitarian or development activity (some reports used ‘evaluation’ as synonymous with a situation assessment, which were utilised for contextualisation but not included as an ‘evaluation report’). Due to the limited number of external evaluations that have taken place in South Sudan, internal evaluations were included (e.g. a comprehensive final report with evaluation components) as well as documents that specifically outline lessons learned, as that is information specifically sought in this study. While a collection of 98 evaluations is a relatively limited set of reports given the scale of donor and NGO activity, in the context of South Sudan it is a substantial set of material, particularly since other databases, such as the CREST and CLEAR-AA database, had identified very few.
Drawing out Lessons
In order to assess the landscape of evaluation in South Sudan, each report was reviewed and categorised by year, implementing organisation, donor, sector (following the UN OCHA sector/cluster categorisation), and report type. The categorisation allowed for a basic landscape assessment by year and sector. For the synthesis of lessons learned, the reports were assessed qualitatively by sector. Content was coded as lessons learned, challenges, insights and positive practices. While NVivo (as do other content-coding software) allows for more systematic approaches (e.g. quantifying usage frequency of specific terms to indicate level of importance, or lack thereof; e.g. Cochrane et al., 2017), this approach was not taken for this synthesis. The framing of this work was intentionally as a synthesis, not as a systematic review, and as a result relies upon a degree of expert analysis and ‘saturation’ with regard to critical assessment of the set of data (although, we are somewhat critical of the idea of ‘saturation’; see Cochrane, 2017), we could alternatively suggest that critical engagement of ideas in evaluation reports emphasised importance in synthesising the data to arrive at the findings presented). This was determined to be a more appropriate approach because unique insight, for example, is important not because of its frequency but because of its rarity and uniqueness, and therefore requires an approach different to that which is typically taken in systematic reviews. The reliance upon expert judgement introduces subjectivity, however this subjectivity was engaged with intentionally. As systematic reviews have their limitations, so too does the subjectivity of expert judgement. Nonetheless, a synthesis approach that relied upon content coding and expert judgement lends itself toward the objectives of this article, more so than the alternatives. Readers should engage with the lessons and recommendations with this limitation in mind; lessons learned from complex operational environments will, at best, act as a means to make better-informed decisions. They will not transform a complex situation into a complicated technical one. As a result of this process, and due to the diversity, and sometimes project-specific nature, of evaluations and their content, not all of the lessons were included. For the sake of readability, not all of the reports that mentioned a particular lesson, challenge, insight or positive practice were cited (lest the text become difficult to follow with lines of references reported in-text). However, a complete bibliography of all reports (with links to each report) is included as an Appendix.
Results
Landscape of Evaluation
It appears that few evaluation reports are available for the pre-independence (2005–10) transition period, however this may not be representative as activities occurring in southern Sudan at that time would have reported as Sudan, without necessarily specifying South Sudan or being categorised as such. As the focus of this study is on synthesising lessons learned for the current operational environment, an effort was not made to identify all evaluation reports on Sudan and include those that implemented activities in what is now South Sudan. Thus, by design as well as by international donor funding trends, the majority of the evaluation reports are from the post-independence period (2011–17). The significant rise of evaluations in 2015 appears to align with the end of a project cycle that began shortly after independence, and the decline of evaluations in 2013 and 2014 being related to the violence and uncertainty that occurred during these years (see Figure 1).
To give some context to the publication trends of evaluation reports, South Sudan received US$1.7 billion in official development assistance from OECD countries in 2015, and US$1.6 billion in 2016 (OECD, 2018). Taking a more in-depth view of 2016, the largest amount of funding went into humanitarian aid (approximately US$950 million), followed by social infrastructure (approximately US$400 million), while relatively limited amounts of funding went into aspects of production, such as agriculture or livestock (OECD, 2018). According to the Global Humanitarian Assistance Report of 2016, South Sudan was the second largest recipient of humanitarian funding globally (GHA, 2016). The OECD figures do not include non-OECD donors, which are significant contributors to humanitarian funding (GHA, 2016), but nonetheless highlight the scale of official development assistance in South Sudan. Since independence in 2011, official development assistance has been steadily increasing (Norad, 2016).
The sectors for which evaluations have taken place appear to differ significantly from the funding flows. This requires some context, as the OECD classifications of funding do not necessarily align with the UN OCHA sectors utilised in this study. For example, the UN OCHA sectors/clusters of ‘health’ and ‘nutrition’ might be funded under humanitarian assistance, along with ‘coordination’, ‘disaster management’, ‘post emergency recovery’ and ‘protection’. If these sectors are combined, 53 (54%) of the evaluation reports might be considered within the humanitarian funding stream (as a rough estimate). It thus appears that publicly available evaluation reports to-date have focused more on development-oriented activities than upon the humanitarian ones. This study did not undertake a geographic assessment of projects and/or evaluations. This would be a fruitful endeavour for future research, which would highlight some of the challenges of operating within a politically contested operational environment, wherein one actor in the conflict has the ability to direct or restrict activity.
In order to contextualise these results, consider some comparative results from the evaluation repositories made available by the CREST and CLEAR-AA African Evaluation database. While South Sudan has three evaluation reports available, neighbouring countries have many more (figures adjusted to represent those between 2011 and present, a similar time period as South Sudan): Ethiopia (375), Kenya (404), Uganda (321). The USAID Development Experience Clearinghouse and the reports made available by ReliefWeb show similar disparities.2 Several factors contribute to this: South Sudan is a relatively new nation, it has a relatively small population (12 million) in comparison to Ethiopia (100 million), Kenya (48 million) and Uganda (41 million), and since independence it has been plagued with instability, uncertainty and economic failure. Thus, the relatively low number of evaluation reports available for South Sudan is as expected. Moreover, the number of evaluation reports identified by this study are higher than anticipated in comparison to the existing compilations (e.g. report by Norad and the CREST and CLEAR-AA database). One reason that one might expect to see a rise of evaluation work over time within South Sudan is because of a recognition by donors in the 2000s that thousands of interventions had taken place but there was a dearth of evidence about what impacts had resulted (Bennett et al., 2010; Norad, 2016). These donors have continued to operate in the newly independent Republic of South Sudan, many of which have published evaluation reports identified by this study.
Synthesis of Lessons Learned
The following synthesis of lessons learned is presented based on themes, as they emerged from the evaluation reports included in this study. The objective of this section is to synthesise learning, not to delve into debates about which sector or project ought, or ought not, be prioritised. Instead, the objective is to better inform decision makers regarding their funding, planning, implementation and learning processes. In seeking to synthesise and share the lessons learned, the following presentation aims for readability and clarity, and therefore references are only provided when needed (meaning not every report that mentions a particular lesson is referenced in the Bibliography). This is not meant to emphasise one report or organisation over another, rather it is to enhance the readability of the synthesis and enable learning, while the full list of all reports is available as an Appendix.
Information. All projects operating in South Sudan over the last decade have worked in difficult circumstances. The conflict, combined with limited capacity and infrastructure, have prevented the collection of rigorous data. This is compounded by the fact that much programming was emergency/humanitarian oriented, which does not have the same expectations of needs assessments before funding and implementation as development programming does. For many projects, the required data to support appropriate proposal development, planning and implementation are not available (AET, 2014; Johnson et al., 2013). Efforts to fill these information gaps during the project cycle often fall short (UNICEF, 2015; UNIDO, 2014; World Bank, 2011). Some projects have performed unsatisfactorily, or had questionable relevance, due to insufficient information and/or incorrect assumptions (Bennett et al., 2010; USAID, 2015; IOM, 2016; Norad, 2016). This is not an organisation-specific challenge. As of 2018, UNDP’s Human Development Index was missing data for 45 indicators. Basic data is lacking in all sectors.
Needs assessments should become more routine to inform projects at the outset and to create feedback mechanisms for adjusting to changing contexts during implementation (ACF, 2011; EC, 2017; GoC, 2017). Ensuring that activities are appropriately conceptualised and implemented may require a greater investment into data collection and research, a step in the project cycle that will require flexibility from donors and new skills by implementing organisations (USAID, 2016). For all involved, this also includes recognising the complexity of working in South Sudan and investing time and resources into better understanding the cultural, political and economic arenas that influence activities, outcomes and impacts (Caldwell and Oliver-Burgess, 2014; Morrison-Métois, 2017). A Norad report notes that ‘context analysis, rather than the availability of advisors and/or bilateral interests, should dictate the choice of sector priorities’ (Norad, 2009: 60–1), while an OECD report notes that ‘simple contexts do not exist among fragile states, only situations simplified by donor countries’ (Morrison-Métois, 2017: 20). The All-Party Parliamentary Group for Sudan and South Sudan similarly states that ‘conflict analysis must not be a tick box exercise but a real and ongoing endeavour that has the power to shift priorities and change programme direction’ (APPGSS, 2015: 37).
Systems thinking. Not all projects can engage entire systems (e.g. agricultural, legal, health), but systems thinking is required so that activities are understood as occurring within multi-scale, complex systems. Identifying the ‘weak link’ in the system may allow for proactive programming to address potential barriers to impact and sustainability (USAID, 2012a). Taking a broader systems perspective may result in pragmatic programme adjustments, as basic infrastructure are lacking throughout the majority of the country. Systems thinking helps to identify barriers and potential enablers, reducing ‘blind spots’ in design and implementation. For example, a systems thinking approach can identify how activities might contribute to conflict, and thereby adjust accordingly or introduce mechanisms to reduce this potential outcome. Another example is that systems thinking may identify important interconnectivity, such as the how, in some (semi-)pastoralist areas, income generation projects are linked to livestock, as the latter is how wealth is protected and assets accumulated. Protecting the gains of income generation, therefore, may need to consider linkages to other programming on livestock management and vaccination, lest the newfound economic gains be lost due to animal disease. Developing theories of change that outline the linkages and pathways of change can help organisations better grapple with integrating systems thinking into their practice, and in so doing improve the quality of their activities (Jeene et al., 2013; Johnson et al., 2013; Mansilla and Turnbull, 2016; SSRF, 2012; UNICEF, 2013). More donors should make systems approaches and theories of change a common practice for their implementing organisations. This will strengthen proposal design, metric selection, implementation and learning.
Gender. The lack of available data contributes to an overall paucity of activity that address the gendered nature of many of the development activities undertaken. Many projects have not committed the required resources to understanding, let alone addressing, the gendered roles, constraints, opportunities and impacts of the activities they implement (Jeene et al., 2013; USAID, 2011; UNICEF, 2013). While there are a number of projects that focus explicitly on women and girls (Nonviolent Peaceforce, 2017; UN Women, 2012), such a focus ought to engage with the broader socio-cultural environment that shapes ideas of masculinity, power and decision making (UNICEF, 2016; USAID, 2012b), which includes boys and men, as well as community and traditional leaders (Nonviolent Peaceforce, 2017; UN Women, 2012). Many of the projects that did focus on women and girls were stand-alone projects that commonly have been implemented in parallel to other interventions, detached from the broader set of activities (Norad, 2016; UNICEF, 2015). What can be drawn from the literature, therefore, is not a specific lesson, challenge, insight or practice per se, but what has been routinely absent. As a means to address these limitations, a number of reports note that greater integration is required so that the gendered nature of all activities are better understood and engaged with (e.g. Jeene et al., 2013; USAID, 2011; UNICEF, 2013). Such integration would enable projects to better consider the gendered impacts of activities. For example, agricultural improvements may result in improved indicators of yield, but may also introduce new, labour-intensive burdens on women (Cochrane Consulting, 2018). The status quo has been programming that is gender blind or minimally gender aware, whereas gender transformative programming requires that these absences be more proactively considered from design to exit planning.
Seasonality. Nutrition, livelihoods, access and conflict are seasonal in South Sudan (ACF, 2011; Bennett et al., 2010; FAO, 2016; OCHA, 2015). Donors and NGOs need to take the role of seasonality as a first point of annual planning, not their own fiscal or calendar years, lest major opportunities be missed. For example, livelihood projects not aligned with the agricultural season, and without lead time for procurement, may result in a ‘lost’ year as the critical entry point in the agricultural cycle is missed. Donors and NGOs must acknowledge ‘the primacy of the seasons in annual planning’ (IAHE, 2015: 9).
Flexibility. The complexity and uncertainties of the operational environment in South Sudan necessitates adaptiveness in programming and an ability to adjust in a swift manner. Donors and implementing organisations need to plan for unexpected changes and need to have partnership agreements whereby the modality of working can be responsive in a timely manner (Morrison-Métois, 2017; SSRF, 2012; UNHCR, 2013; USAID, 2012c; see also Coghlan, 2017). This could take the form of ‘crisis modifiers’ within project implementation plans as one means to ensure rounds of further approvals are not required when rapid shifts take place. This could be supported by new directions of nexus programming (although practical tools for reconceptualising and redesigning programming based upon this idea are limited). Donors that have separate lines of funding for humanitarian and development activity may consider the Australian Aid ‘early recovery’ or the Canadian ‘whole-of-government’ or ‘whole-of-department’ approaches as potential ways of working, or using the ‘resilience’ framework to enable programmatic flexibility across humanitarian and development funding lines (FAO, 2016; GoC, 2017; IAHE, 2015; Jeene et al., 2013; Morrison-Métois, 2017; OCHA, 2015). To-date, this flexibility has been widely recognised as being required, but insufficiently acted upon. The Norwegian Agency for Development Cooperation concluded that the ‘donor community mostly failed to adapt their development interventions to the volatile and fragile South Sudanese context’ (Norad, 2016: 3). Flexibility, however, ought not become the modus operandi of all activities, but rather occur within a set of broader objectives and act as a means to strengthen the pathways to the sought after change (i.e. a greater acknowledgment of the needs for responsive processes, while not becoming overly reactive and losing programmatic cohesion; Yang and Logo, 2015).
Communication. Some members of the GoSS have experience working with international organisations, however many are new to engaging with these relationships. One of the reasons for unsatisfactory processes and outcomes is misunderstandings between government staff and personnel of international partners (UNDP, 2013; World Bank, 2011). Ensuring that the roles, responsibilities and expectations are clear and that requirements are understood requires an investment of time and resources for building relationships, common ground and mutual understanding. Donors and large organisations that have specific processes and requirements may need to work with a greater degree of flexibility, as the institutions and capacity within the country may pose barriers for meeting typical conditions, and thus delay implementation (GoC, 2017).
Collaboration. In the absence of strong governmental systems, myriad parallel systems are being supported and developed by NGOs. Organisations should ‘seek out opportunities to collaborate with complementary activities of other stakeholders where the potential benefits of collaboration outweigh the costs’ (USAID, 2012a: 40). In practice, collaboration has been insufficient and often limited to information sharing (Bennett et al., 2010; IOM, 2016; Norad, 2016; UNDP, 2013), sometimes creating duplication (see Coghlan, 2017). Experience at the donor level has not been promising (Bennett et al., 2010; DFID, 2010; Norad, 2009), and thus collaboration based on sector or geographic area may be more fruitful entry points for time investment (IAHE, 2015). Rather than developing a detailed common strategy, and risk lengthy disagreement and negotiation, collaboration might take the form of activity linking and coordination at sub-national levels or across sectors (Baker et al., 2017; Norad, 2009; UNESCO, 2016). UN OCHA clusters/task forces/working groups are examples that are widely recommended in the evaluation reports (FAO, 2015; Norad, 2017; OCHA, 2015). In parts of the country, additional NGOs are less of a priority than cooperation and collaboration between existing ones (DRC, 2015). As an initial entry point, collaboration should be fostered within donor and organisational portfolios (ACF, 2011; GoC, 2017; Mansilla and Turnbull, 2016). Dedicated resources and staffing are required to enable improved coordination and collaboration (EC, 2017; USAID, 2012a). Collaboration can also be used to leverage systemic shifts, such as in governance and accountability; to-date this has not be effectively utilised by the international community. Former Canadian Ambassador to South Sudan Nicholas Coghlan writes ‘the humanitarian community’s inability to display solidarity (even in a restricted area and for a short period) was a fatal flaw in our collective attempts to hold the governmental authorities to account. We allowed ourselves to be morally blackmailed and our calls for accountability reaped steadily diminishing returns’ (2017: 199). In instances where collaboration is not possible or feasible (including due to serious concerns of corruption), an alternative to active collaboration with the government is alignment (outlined next).
Alignment. As of 2018, the operational environment was slowly moving from one of emergency response to development activity (or at least the introduction of both sets of activities in different parts of the country). As this transition occurs, there is a greater need to align activities with the GoSS to avoid duplication and improve the efficiency of resource use (USAID, 2012a, 2012c). The experience leading up to the Comprehensive Peace Agreement (up to 2005) demonstrated a period of coherence within the donor community, as the objective was clear and shared. However, following the signing of the agreement (post 2006) the Joint Donor Team has not been able to maintain alignment, within the donor community or with the GoSS (Bennett et al., 2010). The lack of cohesion and alignment continues to be the norm, with few exceptions. Federal plans (e.g. the New Deal Compact and South Sudan Development Plan) or sector plans (e.g. Comprehensive Agriculture Master Plan) are documents around which alignment could materialise, however donors and NGOs need to ensure their planning involves the GoSS and there is need for the GoSS to take a stronger leadership role in alignment and coordination. While not possible for all projects, a degree of alignment is also needed across levels of governance within the GoSS (central, state, county, payam/district, boma/community) to ensure vertical alignment within the country, as well as horizontal alignment between the GoSS and its partners (GoC, 2017). For clarity, the focus here is upon alignment because budgetary support or sector-wide approaches that providing funding to the GoSS is not feasible. In this usage, alignment refers to an orientation of working akin to collaboration, and the avoidance of duplication and parallel systems.
Commitment. Due to the extent of existing challenges and the low level of capacity, the transformative objectives of nearly all developmental activities will require long-term commitments of technical support and funding, lest the sustainability of short-term gains be lost (ACF, 2011; GoC, 2017; IAHE, 2015; USAID, 2012a). Experience to-date is that programming is too short, and as a result focuses on filling gaps and meeting urgent needs, rather than building local capacity (Morrison-Métois, 2017). In recommending longer-term funding and programming, this study is in agreement with Bennet et al. who argued that ‘the disbursement of these funds – whether bilateral, multilateral or through pooled funds – should be dependent on at least bi-annual (twice yearly) updates of events on the ground’ (2010: xxi) to ensure effective resource utilisation. Conditional modalities need to be carefully considered, as some ways of working (e.g. payment for results) may not be suitable or have negative, unintended consequences (Clist and Dercon, 2014). In recognising uncertainty and complexity in the South Sudanese context, and the probability of non-linear outcomes, donors should encourage an on-going open dialogue with implementing partners about the challenges and lessons learned (lest these emerge only in final reports or by external evaluations). As Bennett et al. argued, there has been an ‘over-use of “good practice”, particularly with respect to ownership and harmonization, at the expense of field knowledge and engagement that was required (and welcomed) from 2005 onwards’ (2010: xx). Projects have routinely introduced best practices from other countries and contexts, sometimes because of the lack of available information and sometimes because the organisation opted not to obtain the required information (e.g. needs assessments, participatory input on programme design) or draw upon local knowledge. This has resulted in projects lasting only as long as foreign funding keeps them afloat, often of questionable immediate impact and being unsustainable. Changing course in South Sudan will require reflection about common practice: incorporating in-depth assessments, opting for ‘best fit’ over ‘best practice’, and amidst uncertainty a willingness to engage in more flexible, but also long-term, programming.
Recognise trade-offs. Many projects struggle with the challenge of quantity versus quality: broad coverage at a basic level or deeper change at smaller scales. This is a long-running debate in development practice that will not be resolved based on lessons from South Sudan. However, what the available evaluation reports suggest is that the trade-offs of these decisions, whichever they may be, ought to be done in a more informed fashion – particularly as they relate to ‘do not harm’ programming. Decisions need to be informed by access, geographic coverage and scale to ensure resources are not spread too thin and logistical costs are appropriately taken into account (ACF, 2011; USAID, 2012b). Programmes with a specified geographic area can be effective, as they seek to address needs of a region (e.g. state-level), as opposed to scaling-up a specific activity across a broad region (e.g. USAID, 2017). In line with the spirit of the 2005 Paris Declaration on Aid Effectiveness and the 2008 Accra Agenda for Action, this synthesis will not posit which modality ought to be pursued, and leave these decisions with the GoSS and its partners.
Transition. Humanitarian and development activities, in the absence of government systems, often operate in parallel to what ought to be a government system. The long-term objective of activities should include the building of capacity and processes for transitioning resources and responsibilities to local actors to ensure the work extends beyond funding cycles and is not threatened by shifts in official development assistance priorities (ACF, 2011; Bennett et al., 2010; LWF, 2016; Johnson et al., 2013; SSRF, 2015; UNDP, 2013). This should be informed by an overarching vision or strategy to ensure cohesion and consistency (GoC, 2017; UNICEF, 2015). In many instances, this will involve building the capacity of government personnel – from extension workers at the community level to the Ministry in Juba – necessitating a shift of modality (UNESCO, 2016; USAID, 2012a). Bypassing the government, Norad argues, will have negative consequences (Norad, 2016). Alternative approaches to enhance sustainability could include building linkages to national institutions (e.g. the Yei Agricultural College and the Wau Mentor-Teachers’ Union) (USAID, 2012b), supporting civil society (USAID, 2016), and engaging in more community-based participatory approaches (Save the Children, 2011; UNDP, 2012). The donor community has not developed an ‘overall strategic plan for recovery and development for itself or in collaboration with the government’, raising serious questions about the sustainability of action to-date (Norad, 2016: 3, also UNICEF, 2015).
Conclusion
The emergence of new initiatives to collect and share evaluation reports are encouraging. Donors are making their reports easier to find (e.g. DFID) and creating searchable repositories of evaluation documents (e.g. USAID’s Development Experience Clearinghouse). Database style efforts are emerging (e.g. ReliefWeb and the CREST/CLEAR African Evaluation Database). However, much more is needed if evaluation reports are going to be publications that are used beyond individuals and organisations. Evaluation reports remain difficult to find and are hosted on a wide range of websites. More fundamentally, many evaluation reports are not public and thus cannot be accessed at all. Usable evaluation that shares best practices and lessons learned requires changes to how reports are stored and shared. More recognition of evaluation reports is required such that their value is captured within systematic reviews and syntheses, further enabling the use of evaluation reports. The landscape assessment and synthesis of learning from South Sudan in this study exemplifies the challenges of identifying evaluation reports, but also presents methodological learning on how studies of this material might be conducted and the potential for improved collective learning, if these reports were more accessible.
With regard to this synthesis of learning from South Sudan, a summary report by Norad found that the ‘aid architecture was inconsistent and lessons learned did not alter approaches’ (Norad, 2016: 3). This failing is partially because the lessons learned are not available, easily accessible, or presented in a synthesised fashion. The lack of access, however, does not explain why actors are not responding to their own learning, repeating the modalities and activities found to be ineffective or harmful. While it is important to synthesise, further critical reflection is required on the implementation barriers that exist. This study has experimented with methods regarding how systematic reviews and syntheses might be conducted for evaluation reports, which may provide methodological guidance for additional studies of this nature. The utilisation of existing databases, seeking input from relevant networks and obtaining reports from targeted websites proved most useful, while contacting individual organisations seeking evaluations was time consuming and of limited use.
The high-level synthesis of lessons learned from humanitarian and development activity in South Sudan presents key areas of action for improvement. The synthesis finds that projects would be better designed if they obtained the required information, integrated systems thinking to account for the broader context, and better took into account the gendered nature of activities and impacts. Implementation can be strengthened if seasonality is taken into account, if modalities are more flexible, and if a greater degree of communication and collaboration between partners develops. Sustainability and long-term impact require that there is a higher degree of alignment with the government, longer-term commitments in programming, a recognition of trade-offs, and a clear vision and strategy for transitioning capacities and responsibilities to national actors. The lessons drawn from evaluation reports in South Sudan offer direction for new ways forward, many of which have been concurrently learned by a diverse set of donors and organisations. These lessons need to be more widely shared and available lest they be re-learned by other donors and organisations in the years to come.
Notes
https://crest2.sun.ac.za/african_evaluation_db/default/african_eval_db_01 (accessed 7 January 2020).
The details are analysed in this article, as the categorization of ‘evaluation’ by ReliefWeb differs from that used in this report (being much more flexible) and the USAID search function appears to include text references to nations, and the results require assessment, as was done for the evaluation reports on South Sudan included in this study.
Bibliography
ACF [Action Against Hunger] . (2011), An Evaluation of ACF Operational Approach and Strategy Warrap and Northern Bar el Ghazal States, South Sudan (Patrick Andrey). ACF International: Paris.
AET . (2014), Primary Education Programme: Africa Educational Trust Jigsaw Consult.
APPGSS . (2015), Bridging the Gaps: Lessons from International Engagement with South Sudan 2011–2014. All-Party Parliamentary Group for Sudan and South Sudan January 2015.
Baker, J., Thiele, A. and Saint-Mleux, C. (2017), Disasters Emergency Committee – East African Response Review: South Sudan. Results Matter Consulting August 2017.
Bennett, J., Pantuliano, S., Fenton, W., Vaux, A., Barnett, C. and Brusset, E. (2010), Aiding the Peace: A Multi-Donor Evaluation of Support to Conflict Prevention and Peacebuilding Activities in Southern Sudan 2005–2010. ITAD Ltd, UK.
Caldwell, A. and Oliver-Burgess, A. (2014), Real-Time Evaluation of IOM’s Humanitarian Response to the South Sudan Crisis. Final Report November 2014.
Clist, P. and Dercon, S. (2014), 12 Principles for Payment by Results (PbR) in International Development https://assets.publishing.service.gov.uk/media/57a089d2e5274a27b20002a5/clist-dercon-PbR.pdf (accessed 7 January 2020).
Cochrane, L. (2017), Strengthening Food Security in Rural Ethiopia. Dissertation submitted to the University of British Columbia, Kelowna, Canada.
Cochrane, L., Corbett, J., Evans, M. and Gill, M. (2017), ‘Searching for Social Justice in Crowdsourced Mapping’ Cartography and Geographic Information Science, 44:6, 507–20.
Cochrane Consulting . (2018), Comparative Study of Activities and Modalities for Resilience Building in South Sudan. Cochrane Consulting: Ottawa, Canada.
Coghlan, N. (2017), Collapse of a Country: A Diplomat’s Memoir of South Sudan. McGill-Queen’s University Press: Montreal and Kingston.
DFID . (2010), Country Programme Evaluation – Sudan. Evaluation Report EV708 Department for International Development March 2010.
DRC . (2015), Food Security and Livelihoods Assessment in Maban County, Upper Nile. Danish Refugee Council January 2015.
EC . (2017), Evaluation of the European Commission’s Interventions in the Humanitarian Health Sector, 2014–2016. Final Report, European Commission September 2017.
FAO . (2015), FAO in South Sudan: Emergency Livelihood Response Programme – A Review of 2015 and Planning for 2016. Food and Agriculture Organization: Rome.
FAO . (2016), Evaluation of FAO’s Contribution in South Sudan. Office of Evaluation, Country Programme Evaluation Series, Food and Agriculture Organization: Rome.
GHA . (2016), Global Humanitarian Assistance Report 2016. Development Initiatives: Bristol.
GoC . (2017), Evaluation of Canada’s Development and Stabilization and Reconstruction Taskforce (START) Programming in South Sudan. Evaluation Report January 2017.
IAHE . (2015), Report of the Inter-Agency Humanitarian Evaluation (IAHE) of the Response to the Crisis in South Sudan. Final Evaluation Report November 2015.
IOM . (2016), If We Leave We Are Killed: Lessons Learned from South Sudan Protection of Civilian Sites 2013–2016. International Organization for Migration South Sudan.
Jeene, H., Sabu, J. and RALSA Foundation . (2013), South Sudan Early Recovery and Humanitarian NGO Funding Round. INJ734/11A458, Evaluation Report Australian Government.
Johnson, R., Ockelford, J. and Power, T. (2013), Learning from BSF: Lessons from the Basic Services Fund, South Sudan, 2006 to 2012. Final Report February 2013.
LWF . (2016), Final Evaluation of Malaria Prevention Project in Jonglei State, South Sudan. Lutheran World Federation April 2016.
Mansilla, C. and Turnbull, M. (2016), NRC Final Evaluation Report: South Sudan Emergency Response December 2013 – December 2015. Integrated Risk Management Associates: Singapore.
Morrison-Métois, S. (2017), Responding to Refugee Crises: Lessons from Evaluations in South Sudan as a Country of Origin September 2017. OECD/Norad: Paris.
Nonviolent Peaceforce . (2017), Promoting Women’s Role in Peace Building and Gender Based Violence Prevention in South Sudan. Mid-term Evaluation Report October 2017.
Norad . (2009), Mid-Term Evaluation of the Joint Donor Team in Juba, Sudan. Evaluation Report 2/2009 Evaluation Department, Norwegian Agency for Development Cooperation.
Norad . (2016), Country Evaluation Brief: South Sudan. Report 6/2016 Evaluation Department, Norwegian Agency for Development Cooperation.
Norad . (2017), Realising Potential: Evaluation of Norway’s Support to Education in Conflict and Crisis through Civil Society Organizations. Report 9/2017 Evaluation Department, Norwegian Agency for Development Cooperation.
OCHA . (2015), Evaluation of the Common Humanitarian Fund. Country Report: South Sudan, United Nations Office for the Coordinator of Humanitarian Affairs May 2015.
OECD . (2018), South Sudan Country Profile: www2.compareyourcountry.org/aid-statistics?cr=302&cr1=oecd&lg=en&page=1 (accessed 7 January 2020).
Raifman, J., Lam, F., Keller, J. M., Radunsky, A. and Savedoff, W. (2018), ‘How well are Aid Agencies Evaluating Programs? An Assessment of the Quality of Global Health Evaluations’. Journal of Development Effectiveness, doi: 10.1080/19439342.2018.1452779.
Save the Children . (2011), An Evaluation of Save the Children’s Cash Transfer Project in Aweil East County, Northern Bahr el Ghazal State, South Sudan. Save the Children May–June 2011.
Sorbo, G. M., Schomerus, M. and Aalen, L. (2016), South Sudan Evaluation Portrait. Report 6/2016 Country Evaluation Brief, Evaluation Department, Norwegian Agency for Development Cooperation.
SSRF . (2012), Lessons Learned Exercise: Final Report. South Sudan Recovery Fund September 2012.
SSRF . (2015), South Sudan Recovery Fund – Round 3: UN Joint Stabilization Programmes. South Sudan Recovery Fund, Outcome Evaluation August 2015.
UN Women . (2012), Sudan and South Sudan Programme Evaluation Report: Building Capacities for Gender Equality and Protection of Women’s Rights in Sudan 2008–2011. UN Women June 2012.
UNDP . (2012), Evaluation of Sudan Disarmament, Demobilization and Reintegration (DDR) Programme. United Nations Development Programme Sudan Final Report November 2012.
UNDP . (2013), Final Evaluation of Disarmament, Demobilization and Reintegration Programme Individual Reintegration Project Component (2009–2012). South Sudan Disarmament, Demobilization and Reintegration Commission (SSDDRC) and United Nations Development Programme (UNDP) September 2013.
UNESCO . (2016), What to Prioritize When Everything is a Priority? Crisis-Sensitive Education Sector Planning in South Sudan. International Institute for Educational Planning: Paris.
UNHCR . (2013), Flooding Across the Border: A Review of UNHCR’s Response to the Sudanese Refugee Emergency in South Sudan. PDES/2013/08 United Nations High Commission for Refugees.
UNICEF . (2013), Evaluation of UNICEF Programmes to Protect Children in Emergencies: South Sudan Country Case Study. Evaluation Report, Evaluation Office, UNICEF April 2013.
UNICEF . (2015), Most Significant Change and Lessons Learnt - YouthLEAD Initiative. Final Evaluation Report, UNICEF South Sudan September 2015.
UNICEF . (2016), Multi-Country Real Time Evaluation of UNICEF Gender-Based Violence in Emergencies Programmes: South Sudan Country Report. Child Protection Section, Evaluation Report July 2016.
UNIDO . (2014), Integration and Progress through Protection and Empowerment of Displaced Groups in South Sudan. United Nations Industrial Development Organization: Vienna.
USAID . (2011), End-of-Project Review: Southern Sudan Technical Assistance Project. United States Agency for International Development September 2011.
USAID . ( 2012a), Food, Agriculture, and Rural Markets (FARM) Project Mid-Term Evaluation Report. United States Agency for International Development.
USAID . ( 2012b), Gender Equity through Education (GEE) – End of Project Performance Evaluation Report. United States Agency for International Development June 2012.
USAID . ( 2012c), Sudan Health Transformation Project Phase II. End of Project Performance Evaluation Report United States Agency for International Development July 2012.
USAID . (2015), Mid-Term Performance Evaluation of Health Activities under the Integrated Service Delivery Project. Evaluation Report United States Agency for International Development July 2015.
USAID . (2016), Evaluation of USAID/South Sudan’s Democracy and Governance Activities under the IRI Project – 2012–2014. Evaluation Report United States Agency for International Development April 2016.
USAID . (2017), Abyei Rehabilitation Initiative (ARI): Mid-Term Evaluation and Phase III Baseline Survey. United States Agency for International Development June 2017.
World Bank . (2011), Implementation Completion and Results Report (TF-57638) on a Grant in the Amount of US$ 13.49 million to the Government of South Sudan for a Livestock and Fisheries Development Project. World Bank: Washington, DC.
Yang, S. X. and Logo, H. K. (2015), UNDP South Sudan Access to Justice and Rule of Law Project. Mid-Term Evaluation Report. United Nations Development Program (UNDP).
Appendix: Bibliography of Evaluations
Action Against Hunger . (2004) Evaluation of the Nutritional Surveillance Activities of ACF-USA and Analysis of the Nutritional Situation in South Sudan 2004. https://www.actionagainsthunger.org/sites/default/files/publications/Evaluation_of_the_nutritional_surveillance_activities_of_ACF-USA_and_Analysis_of_the_Nutritional_Situation_South_Sudan_12.2004.pdf
Action Against Hunger . (2005) Evaluation of the Nutritional Surveillance Activities of ACF-USA and Analysis of the Nutritional Situation in South Sudan 2005. https://www.actionagainsthunger.org/sites/default/files/publications/Evaluation_of_the_Nutritional_Surveillange_Activities_of_ACF-USA_and_Analysis_of_the_Nutritional_Situation_South_Sudan_12.2005.pdf
Action Against Hunger . (2006) Evaluation of the Nutritional Surveillance Activities of ACF-USA and Analysis of the Nutritional Situation in South Sudan 2006. https://www.actionagainsthunger.org/sites/default/files/publications/Evaluation_of_the_Nutritional_Surveillance_Activities_of_ACF-USA_and_Analysis_of_the_Nutritional_Situation_South_Sudan_12.2006.pdf
Action Against Hunger . (2007) Evaluation of the Nutrition Surveillance Activities of ACF-USA and Analysis of the Nutrition Situation in South Sudan in 2007. https://www.actionagainsthunger.org/sites/default/files/publications/Evaluation_of_the_Nutrition_Surveillance_Activities_of_ACF-USA_and_Analysis_of_the_Nutrition_Situation_South_Sudan_12.2007.pdf
Action Against Hunger . (2008) Evaluation of the Nutritional Situation in South Sudan in 2008. http://www.actionagainsthunger.org/sites/default/files/publications/ACF-NUT-South-Sudan-2008-12-EN.pdf
Action Against Hunger . (2011) An Evaluation of ACF Operational Approach and Strategy in Warrap and Northern Bar el Ghazal States, South Sudan. https://reliefweb.int/sites/reliefweb.int/files/resources/Evaluation_of_ACF_Operational_Approach_and_Strategy_in_South_Sudan_2011_0.pdf
Africa Educational Trust (AET) . (2014) Primary Education Programme Evaluation. Nairobi: Africa Educational Trust.
Africa Educational Trust, Comic Relief. (2014) Final Evaluation of Speak Up: South Sudan. https://africaeducationaltrust.org/wp-content/uploads/2014/07/2014-Speak-Up-Evaluation-2011-2013.pdf
APPGSS . (2015) Bridging the Gaps: Lessons from International Engagement with South Sudan 2011–2014. http://southsudanhumanitarianproject.com/wp-content/uploads/sites/21/formidable/APPGSouthSudanReport_Jan2015.pdf
Australian Aid . (2013) South Sudan Early Recovery and Humanitarian NGO Funding Round. https://beta.dfat.gov.au/sites/default/files/south-sudan-early-recovery-humanitarian-funding-round-evaluation.pdf
CARE . (2017) Adaptation Learning Programme for Africa (ALP). http://www.careevaluations.org/wp-content/uploads/Adaptation-Learning-Programme-for-Africa-Narrative-Report.pdf
CARE . (2017) Integrated Smart Nutritional Anthropometric Survey. http://www.careevaluations.org/wp-content/uploads/evaluations/final-abiemnom-smart-survey-report-2017.pdf
Central Emergency Response Fund . (2013) Impact Evaluation Report of the South Sudan Education Cluster. https://reliefweb.int/sites/reliefweb.int/files/resources/South%20Sudan%20EiE%20Impact%20Report%20FINAL.pdf
Central Emergency Response Fund . (2015) Independent Review of the Value Added of the Central Emergency Response Fund (CERF) in the Countries Affected by the South Sudan Crisis. https://cerf.un.org/sites/default/files/resources/Indpendent%20review%20of%20value%20added%20South%20Sudan%20Crisis.pdf
Danish Demining Group and the Swedish International Development Agency . (2010) SIDA DDG Evaluation: Final Report South Sudan. https://www.gichd.org/fileadmin/GICHD-resources/rec-documents/Evaluation-DDG-South-Sudan-GICHD-Feb2010.pdf
Danish Refugee Council (DRC) . (2014) Post Relocation Evaluation: Key Findings & Life in the New POC. https://reliefweb.int/sites/reliefweb.int/files/resources/2._post_relocation_assessement_report_final.pdf
Danish Refugee Council . (2015) Food Security and Livelihoods Assessment in Maban County, Upper Nile. https://reliefweb.int/sites/reliefweb.int/files/resources/FoodSecurityandLivelihoodsAssessmentinMaban2015.pdf
DFID . (2010) Country Programme Evaluation, Sudan. Evaluation Report EV708 https://www.oecd.org/countries/sudan/45025294.pdf
DFID . (2013) Learning from BSF: Lessons from the Basic Services Fund, South Sudan 2006 to 2012. http://oro.open.ac.uk/38023/1/2013.pdf
DFID . (2014) Girls’ Education South Sudan. http://iati.dfid.gov.uk/iati_documents/4783937.odt
DFID. (February 2015) Girls’ Education South Sudan. http://iati.dfid.gov.uk/iati_documents/4870618.odt
DFID . ( March 2015) Girls’ Education South Sudan. http://iati.dfid.gov.uk/iati_documents/5098223.xlsx
DFID . (2015) Health Pooled Fund: South Sudan. https://assets.publishing.service.gov.uk/media/57a0896ae5274a31e0000090/South-Sudan-pooled-health-fund-midterm-review.pdf
DFID . (2016) Girls’ Education South Sudan. http://iati.dfid.gov.uk/iati_documents/5711960.odt
DFID . (2017) Girls’ Education South Sudan. http://iati.dfid.gov.uk/iati_documents/22859169.odt
Disasters Emergency Committee . (2017) Disasters Emergency Committee – East Africa Response Review: South Sudan. www.alnap.org/help-library/disasters-emergency-committee-east-africa-response-review-south-sudan
European Union . (2017) Evaluation of the European Commission’s Interventions in the Humanitarian Health Sector, 2014–2016. https://ec.europa.eu/echo/sites/echo-site/files/3._health_evaluation_en_exec_summary_europa.pdf
FAO . (2015) FAO in South Sudan: Emergency Livelihood Response Programme, Lessons Learned in 2015. http://www.fao.org/fileadmin/user_upload/emergencies/docs/Emergency%20Livelihood%20Response%20Programme_Lessons%20Learned%20in%202015.pdf
FAO . (2015) Final Evaluation of Agriculture and Food Information Systems for Decision Support in South Sudan (AFIS). http://www.fao.org/3/a-bd435e.pdf
FAO . (2016) Evaluation of FAO’s Contribution in South Sudan. http://www.fao.org/3/a-bq506e.pdf
FAO . (2016) FAO in South Sudan: Emergency Livelihood Response Programme – A Review of 2015 and Planning for 2016. http://www.fao.org/fileadmin/user_upload/emergencies/docs/Emergency%20Livelihood%20Response%20Programme%202015-16.pdf
Global Affairs Canada . (2017) Evaluation of Canada’s Development and Stabilization and Reconstruction Taskforce (START) Programming in South Sudan. http://international.gc.ca/gac-amc/publications/evaluation/2017/sudan_development-evaluation-soudan_developpement.aspx?lang=eng
Global Nutrition Cluster . (2014) Lessons Learned in South Sudan Nutrition Cluster: Exercise Conducted by the Global Nutrition Cluster. https://reliefweb.int/sites/reliefweb.int/files/resources/Country-Lessons-Learned-South-Sudan-Nutrition-Cluster_final_10-10-14.pdf
International Alert . (2011) Evaluating the EU’s Role and Challenges in Sudan and South Sudan. http://www.international-alert.org/sites/default/files/publications/092011IfPEWSudan_0.pdf
International Development Law Organization . (2015) Evaluation of the Project ‘Enhancing the Capacity of the Judiciary of South Sudan’. http://www.idlo.int/system/files/event-documents/IDLO%20Evaluation%20Brief_South%20Sudan-2.02.16.pdf
International Organization for Migration . (2014) Real-Time Evaluation of IOM’s Humanitarian Response to the South Sudan Crisis: Final Report. https://www.sheltercluster.org/sites/default/files/docs/rte_south_sudan_-_final_report_141126.pdf
International Organization for Migration . (2016) If We Leave We Are Killed: Lessons Learned from South Sudan Protection of Civilian Sites 2013–2016. https://publications.iom.int/system/files/pdf/if_we_leave_0.pdf
Lutheran World Federation . (2016) Final Evaluation of Malaria Prevention Project in Jonglei State [not available online at present].
Multi-Donor Trust Fund South Sudan . (2013) Independent Evaluation of the Multi-Donor Trust Fund-South Sudan (MDTF-SS). http://www.worldbank.org/content/dam/Worldbank/document/Africa/South%20Sudan/Report/independent-evaluation-of-the-mdtf-ss-report-july-2013.pdf
Netherlands Ministry of Foreign Affairs . (2010) Aiding the Peace: A Multi-Donor Evaluation of Support to Conflict Prevention and Peacebuilding Activities in Southern Sudan 2005–2010. https://www.oecd.org/countries/southsudan/46895095.pdf
Nonviolent Peaceforce . (2017) Promoting Women’s Role in Peace Building and Gender Based Violence Prevention in South Sudan. www.nonviolentpeaceforce.org/images/NPSS_Mid-Term_Evaluation_Report_-28319.pdf
Norad . (2009) Mid-Term Evaluation of the Joint Donor Team in Juba, Sudan. Evaluation Report 2/2009. https://www.oecd.org/countries/sudan/42682206.pdf
Norad . (2013) Evaluation of Five Humanitarian Programmes of the Norwegian Refugee Council (NRC) and of the Standby Roster NORCAP. www.oecd.org/derec/norway/1_Evaluation%20of%20Five%20Humanitarian%20Programmes%20of%20the%20Norwegian%20Refugee%20Council%20and%20of%20the%20Standby%20Roster%20NORCAP.pdf
Norad . (2016) Country Evaluation Brief. Report 6/2016: South Sudan. https://www.norad.no/contentassets/8bbce9a79c8d44959a07de10caa2d1dc/6.16-ceb_south_sudan.pdf
Norad . (2017) Realising Potential: Evaluation of Norway’s Support to Education in Conflict and Crisis through Civil Society Organisations. Report 9/2017. https://www.norad.no/contentassets/5490809999e545be8257166aaa01ffd9/south-sudan_desk-study-report.pdf
Norwegian Refugee Council . (2016) NRC Final Evaluation Report: South Sudan Emergency Response, December 2013 – December 2015. https://www.nrc.no/globalassets/pdf/evaluations/final-evaluation-report-nrc-south-sudan-emergency-response-2013-15.pdf
OCHA, Common Humanitarian Fund . (2015) Evaluation of the Common Humanitarian Fund. Country Report: South Sudan. http://www.unocha.org/sites/dms/Documents/OCHA%20CHF%20South%20Sudan%20Report.pdf
OECD, NORAD . (2017) Responding to Refugee Crises: Lessons from Evaluations in South Sudan as a Country of Origin. https://reliefweb.int/sites/reliefweb.int/files/resources/south_sudan_-_responding_to_refugees_in_crisis_-_lessons_from_evaluations_in_south_sudan_as_a_country_of_origin_.pdf
OXFAM . (2012) Evaluation of Oxfam’s South Sudan Humanitarian Response Using Oxfam’s Global Humanitarian Indicator Tool. https://policy-practice.oxfam.org.uk/publications/effectiveness-review-evaluation-of-oxfams-south-sudan-humanitarian-response-312691
Save the Children . (2004) Water and Sanitation Project Review. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/3858.pdf
Save the Children . (2011) An Evaluation of Save the Children’s Cash Transfer Project in Aweil East County, Northern Bahr el Ghazal State, South Sudan. https://reliefweb.int/sites/reliefweb.int/files/resources/evaluation-cash-transfer-echo-sdn-bud-2010-01043.pdf
Save the Children . (2016) Final Evaluation of the Joint Response for South Sudan under the Dutch Relief Alliance 2015. https://www.savethechildren.nl/sci-nl/files/58/58457cf8-ea2e-4bb3-b537-1c0334239447.pdf
Search for Common Ground . (2017) Strategic Communications for Peace Building in South Sudan; Promoting Localized Understanding and Peaceful Coexistence in South Sudan. https://www.sfcg.org/wp-content/uploads/2017/09/SFCG-ILMC-Evaluation-Final-ILT-approved.pdf
South Sudan Recovery Fund . (2012) South Sudan Recovery Fund Lessons Learned Exercise Final Report. http://mptf.undp.org/document/download/10126
South Sudan Recovery Fund . (2015) South Sudan Recovery Fund Round 3: UN Joint Stabilization Programmes. https://info.undp.org/docs/pdc/Documents/SSD/SSRF%20OUTCOME%20EVALUATION%20REPORT.pdf
UN Women . (2012) Sudan and South Sudan Programme Evaluation Report: Building Capacities for Gender Equality in Governance and Protection of Women’s Rights in Sudan 2008–2011. http://gate.unwomen.org/EvaluationDocument/Download?evaluationDocumentID=3574
UNDP, Millennium Development Goal Fund. (2012) Sustained Peace for Development: Conflict Prevention and Peace-Building in Sudan through Targeted Intervention in Selected Communities along the 1-1-1956 Border. http://www.mdgfund.org/sites/default/files/Sudan%20-%20CPPB%20-%20Mid-term%20Evaluation%20Report.pdf
UNDP, UNMISS . (2013) Final Evaluation of Disarmament, Demobilization, and Reintegration Programme Individual Reintegration Project Component (2009–2012). https://erc.undp.org/evaluation/documents/download/7636
UNDP . (2011) UNDP Southern Sudan: Crisis Prevention and Recovery Programme Mid-Term Outcome Evaluation. https://erc.undp.org/evaluation/documents/download/5295
UNDP . (2012) Evaluation of Sudan Disarmament, Demobilization and Reintegration (DDR) Programme. http://www.smallarmssurveysudan.org/fileadmin/docs/facts-figures/ddr/UNDP-Sudan-DDR-Evaluation-Nov-2012.pdf
UNDP . (2015) UNDP South Sudan Access to Justice and Rule of Law Project. Mid-Term Evaluation Report. https://erc.undp.org/evaluation/documents/download/9321
UNEP, DFID. (2013) Republic of the Sudan Country Programme Evaluation. https://wedocs.unep.org/rest/bitstreams/978/retrieve
UNESCO . (2014). TVET Policy Review. South Sudan. http://unesdoc.unesco.org/images/0023/002312/231287e.pdf
UNESCO . (2016) What to Prioritize When Everything is a Priority? Crisis-Sensitive Education Sector Planning in South Sudan. http://unesdoc.unesco.org/images/0024/002460/246090e.pdf
UNHCR . (2008) Evaluation of UNHCR’s Returnee Reintegration Programme in Southern Sudan. http://www.unhcr.org/48e47f972.pdf
UNHCR . (2011) An Independent Impact Evaluation of UNHCR’s Community Based Reintegration Programme in Southern Sudan. http://www.unhcr.org/research/evalreports/4e41237a9/independent-impact-evaluation-unhcrs-community-based-reintegration-programme.html
UNHCR . (2013) Flooding across the Border: A Review of UNHCR’s Response to the Sudanese Refugee Emergency in South Sudan. PDES/2013/08. http://www.unhcr.org/research/evalreports/51e94e689/flooding-across-border-review-unhcrs-response-sudanese-refugee-emergency.html
UNICEF . (2011) Improvement of Health and Livelihood of Rural Communities in Southern Sudan and the Three Transitional Areas of Abyei, Blue Nile State and South Kordofan through Increased Access to Safe and Sustainable Water, Sanitation and Hygiene Facilities Project. https://www.unicef.org/evaldatabase/files/2011_South_Sudan_-_FINAL_EC_WASH_EVALUATION_REPORT.pdf
UNICEF . (2013) Evaluation of UNICEF Programmes to Protect Children in Emergencies: South Sudan Country Case Study. https://www.unicef.org/evaldatabase/files/CPiE_Evaluation_South_Sudan_country_case_study_final.pdf
UNICEF . (2015) Youthlead Initiative: Most Significant Change and Lessons Learnt. https://www.unicef.org/evaldatabase/files/YouthLEAD_Initiative_Final_Evaluation_Report_Most_Significant_Change_and_Lessons_Learnt_South_Sudan_2015-001.pdf
UNICEF . (2016) Multi-Country Real Time Evaluation of UNICEF Gender-Based Violence in Emergencies Programmes: South Sudan Country Report. https://www.unicef.org/evaldatabase/files/GBV_South_Sudan_final__2016-005.pdf
UNICEF, Search for Common Ground . (2016) Communicating for Peace in South Sudan: A Social and Behaviour Change Communication Initiative, Final Evaluation. https://www.sfcg.org/wp-content/uploads/2016/09/SFCG_Final-_Report_Forcier2016.pdf
UNIDO . (2014) Integration and Progress through Protection and Empowerment of Displaced Groups in South Sudan. https://www.unido.org/sites/default/files/2014-03/PRO-South_Sudan_TFSUD12001-Integr-progr-displ-groups_Final_Eval_Report_0.pdf
UN OCHA . (2015) Report of the Inter-Agency Humanitarian Evaluation (IAHE) of the Response to the Crisis in South Sudan. Final Evaluation Report. http://www.unocha.org/sites/dms/Documents/26042016_final_south_sudan_iahe.pdf
USAID, Education Development Center . (2012) South Sudan Interactive Radio Instruction Performance Evaluation Report. https://pdf.usaid.gov/pdf_docs/PDACT575.pdf
USAID, Management Sciences for Health . (2012). Sudan Health Transformation Project Phase II: End of Project Performance Evaluation Report. https://pdf.usaid.gov/pdf_docs/PDACU122.pdf
USAID, WHO . (2011) Mid-Term Evaluation of the Integrated Disease Surveillance and Response Project. https://pdf.usaid.gov/pdf_docs/PDACT326.pdf
USAID, World Vision . (2014) Maternal and Child Health Transformation Project in Warrap State, South Sudan. https://www.mcsprogram.org/wp-content/uploads/2015/08/WV-S.-Sudan-FE-Report-2.pdf
USAID . (2011). End-of-Project Review: Southern Sudan Technical Assistance Project. https://pdf.usaid.gov/pdf_docs/PDACT272.pdf
USAID . (2012) An Assessment of the Sudan HIV And AIDS Program (SHAP). https://pdf.usaid.gov/pdf_docs/PDACU380.pdf
USAID . (2012) Food, Agriculture, and Rural Markets (FARM) Project: Mid-Term Evaluation Report. https://pdf.usaid.gov/pdf_docs/pdacu715.pdf
USAID . (2012) Gender Equity through Education (GEE): End of Project Performance Evaluation Report. https://pdf.usaid.gov/pdf_docs/PDACU197.pdf
USAID . (2012) Sudan Rural Land Governance (SRLG) Project: Ensuring the Inclusion of Gender and Vulnerable Groups in Land Reform Processes in South Sudan: A Strategy. https://pdf.usaid.gov/pdf_docs/pa00ht7r.pdf
USAID . (2013) Mid-Term Performance Evaluation of the Sustainable Water and Sanitation in Africa (SUWASA) Project. https://pdf.usaid.gov/pdf_docs/PDACY091.pdf
USAID . (2015) Engineering Services for South Sudan Feeder and Trunk Roads: Yambio Country Feeder Roads Inventory and Mapping of Settlement and Potential Agriculture Areas February–March 2015. https://pdf.usaid.gov/pdf_docs/PA00MJFB.pdf
USAID . (2015) Evaluation of USAID/South Sudan Electoral Support Initiatives. https://pdf.usaid.gov/pdf_docs/PA00KMR4.pdf
USAID . (2015) Evaluation of USAID/South Sudan’s Democracy and Governance Activities under NDI Project 2009–2014. https://pdf.usaid.gov/pdf_docs/pa00kwft.pdf
USAID . (2015) Mid-Term Performance Evaluation of Health Activities under the Integrated Services Delivery Project. https://pdf.usaid.gov/pdf_docs/PA00KRW6.pdf
USAID . (2015) Performance Evaluation of Economic Growth Activities under Responsive Assistance for Priority Infrastructure Development (RAPID) Program. https://pdf.usaid.gov/pdf_docs/pa00kq48.pdf
USAID . (2015) USAID South Sudan Health Learning Assessment White Paper. https://pdf.usaid.gov/pdf_docs/PA00KM48.pdf
USAID . (2016) Evaluation of USAID/South Sudan’s Democracy and Governance Activities under the IRI Project 2012–2014. https://pdf.usaid.gov/pdf_docs/PA00MNXR.pdf
USAID . (2017) Abyei Rehabilitation Initiative (ARI): Mid-Term Evaluation and Phase III Baseline Survey. https://pdf.usaid.gov/pdf_docs/PA00N6Q7.pdf
USAID . (2017) Assessment of USAID/South Sudan’s Viable Support to Transition and Stability (VISTAS) Report. https://pdf.usaid.gov/pdf_docs/PA00N6JC.pdf
World Bank . (2011) Implementation Completion and Results Report (TF-57638). https://reliefweb.int/sites/reliefweb.int/files/resources/Full_Report_3334.pdf
World Food Programme . (2017) South Sudan: An Evaluation of WFP’s Portfolio (2011–2016). https://docs.wfp.org/api/documents/WFP-0000022545/download/
World Vision, Ploughshares. (2009) Addressing Armed Violence in East Africa: A Report on World Vision Peacebuilding, Development and Humanitarian Assistance Programmes. http://ploughshares.ca/wp-content/uploads/2012/08/WorldVisionPloughsharesEastAfrica.pdf
World Vision . (2016) Cash-based Programming to Address Hunger in Conflict-Affected South Sudan: A Case Study. http://www.cashlearning.org/downloads/dm2020-case-study-cash-based-programming-south-sudan-final.pdf
