In order to elicit the knowledge, experience, and attitudes of individuals involved in disaster response with regard to evidence-based best practices, Evidence Aid and its institutional partners, Georgetown University and the Uniformed Services University of the Health Sciences, carried out a Policy Delphi study in 2015–2016.
Methods
Purposive and snowball methods were used to select study participants. The Delphi study comprised two rounds of iterative questions, with the questionnaires completed online. In addition, participants at the Evidence Aid conference in November 2016 discussed the findings in focus groups. Excel was used to analyze the quantitative data and Glaser and Strauss (1967) to analyze the qualitative data.
Results
Thirty-six participants responded to the first round of the study, 165 responded to the second round, and 30 participated in the focus group discussions. The salient findings include 1) ensuring that all key stakeholders are engaged in planning for and responding to disasters in a collaborative, coordinated manner—including local community members; 2) using, insofar as possible, evidence-based responses; 3) increasing and strengthening research to ensure that such data are available; and 4) addressing ethical, legal and social issues throughout the planning, immediate response, and post-disaster periods.
Conclusions
Recent humanitarian disasters, due to natural and man-made hazards or a combination of the two, reinforce the need for more effective, efficient, humane responses at the local, national and international levels. This study has yielded findings that can be used to strengthen planning and response by taking into account, where possible, evidence based on research that has been carried out with the engagement of community members and with support by key stakeholders. The most effective means of facilitating the development and implementation of consistent, coordinated policies and practices might be for the United Nations Office for Disaster Risk Reduction to take the lead in engaging key organizations in the required discussions and collaborations.
Background
Disaster planning and response has been largely based on long-standing practices, notwithstanding the fact that for more than a decade there have been calls for evidence-based practices. The patchwork of practices and preferences is often anecdotal, based on the history and perceptions of the host country’s leadership, non-governmental organizations (NGOs), and external aid organizations, rather than on the use of evidence-based interventions and practices. In an environment that increasingly recognizes the importance of accountability and the need to base actions on evidence, this raises the question of how practices that are supported by science and the main gaps in disaster response can be identified.
We investigated this broad question using the Policy Delphi technique, a policy analysis method designed to engage groups of experts (with experts being broadly defined) in dialogue regarding a single issue or multiple issues related to a particular topic. The Policy Delphi approach entails a series of rounds of iterative, structured dialogues (through mail or e-mail, questionnaires, meetings or a combination of these), with each building on the other. Initially, questions are posed and the respondents answer anonymously. The responses are then summarized and used as a basis for the questions posed in the succeeding rounds. Questions may be added to move toward a rich and substantive dialogue in the study.
Policy Delphi has been widely used in disaster management since the early 1970s, when it was first developed by the US Office of Emergency Preparedness. In contrast to the traditional Delphi approach, it does not seek consensus, but rather explores alternatives and their implications. In the health field, Policy Delphi has been used to address ethical, legal and social issues in biomedical and behavioral research, to forecast medical technological developments, to consider drug abuse policy, to plan for organizational and professional priorities, to develop national health priorities, and, at the community level, to engage communities in developing local health systems plans.
The purpose of this study was to engage a wide range of key stakeholders in dialogue regarding various elements of disaster response, including, for example, evidence for best practice, approaches to improving investment with systematic review analysis of evidence, identification of gaps in the evidence base; and factors that impact on effective disaster response. It builds on earlier needs assessment and priority setting exercises by Evidence Aid which is an international organization registered as a charity in the United Kingdom, dedicated to strengthening decision making in the disaster sector through the use of reliable and robust evide.
Methods
This Policy Delphi comprised two rounds, with the first and second being completed online using SurveyMonkey. In addition, participants at the Evidence Aid conference in November 2016, in Washington DC, engaged in focus groups to discuss the findings. The Policy Delphi was designed collaboratively by the planning group for the study, which comprised individuals from Evidence Aid, Georgetown University (GU) and the Uniformed Services University of the Health Sciences. The GU Institutional Review Boards (IRB) approved the study, and GU disseminated the two rounds of questionnaires through SurveyMonkey, conducted the analysis, and served as the location for the Evidence Aid conference. The broad research questions were:
- To what extent is evidence for best practices in disaster response available to a wide range of stakeholders?
- To what extent is Cochrane-style (systematic review) analysis used to assess evidence for best practices in disaster response?
- What are the most effective approaches to improving the cost-effectiveness of investments in disaster response?
- How can the ethical, legal and social issues related to disaster response decision-making be most effectively addressed?
- What are the factors that impact on effective disaster response decision-making?
Participants were selected by the study-planning group using a purposive sampling method; that is, we identified individuals who had published in the field of disaster/humanitarian response, who were public officials or representatives of NGOs engaged in the field, or who had participated in Evidence Aid conferences or other events related to disaster/humanitarian response. For the first round, 135 individuals were selected as potential respondents. For the second round, 237 potential respondents were selected, including the original 135 to whom the first round was sent.
An invitation email that included an informed consent script was sent to the initial 135 potential respondents, with a link to the questionnaire. The first-round, semi-structured questionnaire which was developed for this project (Additional file 1) comprised 24 questions in four categories: a) demographics; b) nature and quality of research-based evidence for disaster response; c) social returns of investments in disaster response; and d) effectiveness of current efforts in disaster response. Thirty-six individuals responded, a response rate of 27%. The second round of the Policy Delphi was disseminated by email (which included an invitation letter and informed consent script) to the 237 individuals who had been identified, with links to the round two questionnaire and the report of the first round. The second-round, semi-structured questionnaire, which was developed for this project (Additional file 2), comprised 20 questions in three categories: a) demographics; b) evidence for best practices in disaster planning and response; and c) effectiveness of current efforts in disaster response. These questions were based primarily on the results of the first round. Over two-thirds of the sample (165, 70%) responded to the second round. Thirty individuals who attended the Evidence Aid conference on November 16–18, 2016 participated in focus groups to discuss key questions related to the findings.
Results
Respondent professional demographics
Respondents to the first and second rounds of the Policy Delphi had similar professional demographic characteristics. Most respondents (37% in the first round and 52% in the second) reported working for an institution other than those listed (e.g., international other than development, scientific research, diplomatic corps). Nearly one-third (31%) of respondents in the first round reported working primarily at an academic institution or university, while 25% of respondents did so in the second round. Other respondents described their primary place of work as being involved in international development (11 and 6%), private sector non-profit organizations, other than development assistance (11% in both the first and second rounds), national government work generally (11 and 15%), government agencies involved in the provision of or support for international aid or development projects (9 and 1.3%), or private sector non-profit development assistance (9%). The categories mentioned by less than 9% (i.e., 3 respondents) but at least two were national government, health agency and national government other. In both the first and second round, some respondents indicated multiple roles.
When asked to specify their role in the organization with which they worked, 35% of first round and just 11% of second round respondents described themselves as researchers; 29 and 12% as professors or teachers; 21 and 13% as consultants or advisors; 18 and 28% as being involved in either administration or management; and 12 and 10% as clinicians. Nearly one-fifth (18%) in the first round and more than one-fourth (28%) in the second round selected the “Other” option, with the self-identified descriptions being primarily the same in both rounds: program manager/emergency manager, technical specialist, director of research, public health program coordinator, science and technology adviser.
There was a difference between the two rounds in geographic location of the respondents: in the first round, one-fourth were based in the United Kingdom (25%), followed by the United States (17%), India (8%), and Switzerland (8%). In the second round, most reported being based in the United States (77%) followed by the United Kingdom (4%). Eleven participants did not respond to this question in the second round.
Summary of findings by round and focus group
Round one
There was agreement among the 36 respondents that international and multi-national organizations, as well as national governments and NGOs could strengthen their response to disasters by using available evidence, and that systematic reviews should be used to synthesize evidence of disaster response effectiveness. Most believed that, although research-based evidence is preferable, for the most part it is ‘best practice’ information that serves currently as the basis for disaster responses determined by decision-makers and practitioners. Most respondents also believed that improvements in co-operation/co-ordination among disaster relief agencies in the collection, analysis and dissemination of data and information regarding disaster response would improve disaster response. Indeed, co-operation and co-ordination among all key stakeholders was a consistent theme through the first round responses, with respondents noting that this would contribute to improving social return on investments in disaster response, for example. Respondents believed that, although there are major impediments to effective and equitable disaster response that derive from both the country in which the disaster occurred (e.g., economic and political factors) and external factors (e.g., global economic influence and post-colonial linkages), these impediments can be addressed through the use of evidence-based policymaking and specific disaster-responses.
Round two
The 165 respondents to the second round consistently identified three factors that facilitate the development of ‘best practices’ for disaster planning and response: 1) increased accessibility to scientific evidence; 2) improved communication among stakeholders; and 3) engagement of multiple beneficiaries in the decision-making process. Most respondents believed that policy and decision-makers need to be educated about research to effectively implement evidence-based policy. Many respondents believed that accessibility to information at all levels can be challenging. They also believed that overcoming communication and financial barriers as well as problems with distribution of information can improve accessibility of information to ensure evidence-based decision-making. Furthermore, co-operation and co-ordination among all key stakeholders and beneficiaries was a consistent theme through the responses.
Focus group discussions
Participants suggested that policy makers should focus on dissemination of evidence related to disaster planning and response, communicating data, and creating a clear chain of command. Funding for research regarding disaster response and strengthened systems was also a concern. This would necessitate a clear plan and strengthening the capacity of all those involved, based on evidence. Participants identified lack of co-ordination between and among NGOs, stakeholders, local authorities and donors as both a key issue and a need that must be addressed with evidence insofar as possible. This included co-ordination regarding 1) assessment of needs and evidence-based practices (and appropriate analysis and dissemination of findings); 2) strategic planning for disasters; and 3) responses and follow-up assessments. Educating the stakeholders on the evidence-based responses and creating a universal standard (or models) were also essential to better planning for and responding to disasters, as was promoting a free flow of information to all, in particular populations affected by disasters.
Results by topic areas addressed
In the following sections, the results of the two rounds of the Policy Delphi and the focus groups are presented by the topic areas addressed:
- Use of Research-based Evidence to Strengthen Disaster Responses;
- Perceptions of Ways that Ethical, Legal and Social Issues Related to Disaster Response Decision-Making be Most Effectively Addressed;
- Factors that Impact Effective Disaster Response Decision-Making; and
- Challenges and Approaches to Overcoming the Challenges of the Expected Outcomes of the United Nations International Strategy for Disaster Reduction (UNISDR) Road Map for Implementation of Sendai
Use of research-based evidence to strengthen disaster responses
Overall, respondents agreed (mean: 4.0–4.5 out of maximum of 6) with the following statements:
- The nature of the evidence for disaster response is primarily best practice information rather than research-based evidence.
- It is possible for national governments to achieve better social return on investment in disaster response by implementing actions or interventions that are based on research evidence.
- It is possible for international and multinational organizations to better achieve social return on investment in disaster response.
- Cochrane-style systematic reviews should be used in a standardized way to synthesize evidence to inform contextually specific evidence of effectiveness in disaster response.
The analysis of answers to questions related to research-based evidence to strengthen disaster responses yielded four themes: 1) availability of evidence for best practices in disaster response; 2) most effective ways to improve the applicability of research-based evidence for pre- and post-disaster planning; 3) most effective ways to ensure that ‘best practices’ inform the design of disaster-related research to prove evidence of effectiveness; and 4) improving the applicability of research-based evidence for pre- and post-disaster planning. If a question was posed in only one round, this is mentioned in the findings. Otherwise, comparisons between rounds are presented.
Availability of evidence for best practices in disaster response
Individual participants mentioned increased research- and evidence-based case studies, more specific and thoroughly researched interventions, and universal access to this information as examples of how this could be accomplished. Responses about the nature and quality of research-based evidence for disaster response suggested that the disaster response community may be divided regarding their perceptions of the reliability of research based-evidence as the basis for strategic planning by national governments or international development agencies.
Of the 36 respondents to Round 1, 21 (58%) either completely or strongly agreed with the statement that ‘[t]he nature of disaster response is primarily ‘best practice’ information rather than research-based evidence’ (58%). The remaining participants ‘somewhat agreed’ (six participants), ‘somewhat disagreed’ (two participants), or ‘completely disagreed’ (one participant) with the statement. Three participants did not provide a response. An overall mean rating of 5 suggests that the respondents believe that best practice information rather than research comprises the basis of ‘evidence’ currently used in disaster response strategic efforts and planning. There was no consensus regarding the statement that research-based evidence for disaster response is sufficiently reliable to warrant its use as the basis for strategic planning by national governments. Indeed, one respondent suggested that international agencies should use evidence “only when it has been peer reviewed, published, and critiqued”.
Most effective ways to improve the applicability of research-based evidence for pre- and post-disaster planning
Responses suggested that the disaster response community may be divided regarding their perceptions of the reliability of research-based evidence as the basis for strategic planning by either national governments or international development agencies.
Of the 165 respondents who participated in the second round, 109 (83%) thought that one of the most effective methods to address this issue was for potential users of research related to pre-disaster planning (i.e. policy makers, planners, emergency response personnel) to be involved in the design and/or implementation of the research. This response was also selected as one of the most effective ways to improve applicability of research-based evidence for post-disaster planning by 104 (80%) respondents. Ensuring that research related to planning is decision-linked was chosen by 74 (57%) respondents for pre-disaster planning and by 70 (54%) respondents for post-disaster planning. Ensuring that comprehensive research is carried out (i.e. addressing socio-economic, life sciences, governance, and other aspects) was selected by 77 (59%) respondents as another effective way to improve the applicability of research-based evidence for post-disaster planning and by 65 (50%) respondents for pre-disaster planning.
Open-ended responses to this item suggest both an emphasis on collaboration with various parties and the use of gap analyses to identity areas of vulnerability for pre-disaster planning. 45 (34%) participants selected and wrote that multiple research methods should be used for pre-disaster planning, with one stating, “When talking about evidence, we need to make sure that it is context-specific” for pre-disaster planning.
Respondents’ suggestions for post-disaster planning emphasized ways to disseminate information and the importance of using evidence-based research to improve policy. Ensuring that the research is guided by priority setting or gap map exercises with a substantial methodology was a goal chosen by 48 (37%) participants.
Many respondents mentioned the importance of having researchers ‘directly engage and actively participate with ‘the local community’, ‘subject matter experts’, policy makers, and planners to improve the applicability of research-based evidence for post-disaster planning’.
Most effective ways to ensure that ‘best practices’ inform the design of disaster-related research to prove evidence of effectiveness
Of the 66 respondents for this question, three quarters (76%) chose ‘ensuring that research-based evidence is available at no or minimal cost’ as one of the most effective ways to ensure that ‘best practices’ inform the design of disaster-related research to prove evidence of effectiveness’.
More than half (55%) also chose the dissemination of ‘best practice information’ related to disaster planning and response through international and national associations of related professionals (e.g., World Medical Association, International Nurses’ Association, International Association of Emergency Managers) only if that information had been critically reviewed by an approved body.
Nearly half the respondents (48%) selected ‘dissemination of ‘best practice information’ related to disaster planning and response through a central, global network only if they have been critically reviewed by an approved body.’
One respondent suggested that, rather than focusing on disseminating information, there should be an emphasis to “work with one group to make a good example of use of best practice information, and then publish the case study”.
As a follow-up question, respondents were asked whether evidence regarding disaster planning and response should be used only if it is peer reviewed, published and critiqued.
Many (42%) indicated that all those processes should occur if the evidence is going to be used. Others (29%) indicated that it should be peer reviewed, while 22% of participants chose that it should always be published and 20% that it should be critiqued.
