Developing and Assessing Digital Public Health Interventions: A Comprehensive Framework
About this framework
Purpose and scope of the framework
1st version, 7/2022
Public Health aims to promote and enhance the health status of individuals and communities through collective societal efforts. Recently, several digital technologies have emerged, pursuing the same goal, developing a novel concept: Digital Public Health. Given the rapidly increasing number of health-related digital technologies, a systematic framework is necessary to assess their values from a public health perspective.
The present framework aims to assist developers, evaluators, policymakers and researchers in the systematic development and evaluation of digital public health interventions in public health by providing a comprehensive overview of criteria to assess digital public health interventions.
The framework comprises a set of criteria framed as open-ended questions clustered within domains that will lead interested parties through a broad spectrum of crucial elements when developing and evaluating digital public health interventions.
In total, it consists of 210 questions, structured by 13 domains: 1) Health Conditions and Current Public Health Interventions, 2) Functionality of the Health Technologies, 3) Software Properties, 4) Human-Computer Interaction, 5) Infrastructure and Organization, 6) Implementation 7) Health-related Effects, 8) Social, Cultural and Gender Aspects, 9) Cost and Economics, 10) Legal and Regulatory, 11) Ethics, 12) Data Security and Data Protection, and 13) Sustainability.
Here you can download the complete framework.
Chen-Chia Pan, Nuria Pedros Barnils, Dorothee Jürgens, Saskia Muellmann, Sarah Janetzki, Jonathan Kolschen, Merle Freye, Hans-Henrik Dassow, Oliver Lange, Sarah Forberger, Tina Jahnel, Benjamin Schüz, Ansgar Gerhardus
PLEASE CITE THIS PUBLICATION AS:
Pan, C. C. ; Pedros Barnils, N.; Jürgens, D.; Muellmann, S.; Janetzki, S.; Kolschen, J.; Freye, M.; Dassow, H. H.; Lange, O.; Forberger, S.; Jahnel, T.; Schüz, B.; Gerhardus, A. (2022) Developing and assessing Digital Public Health Interventions: A comprehensive framework. 1st version. Leibniz Science Campus Digital Public Health, Bremen. 7/2022
We welcome feedback. Please contact us: framework(@)lsc-digital-public-health.de
Currently, there are frameworks that systematically assess the use of health-related technologies such as the Health Technology Assessment Core Model (Lampe et al. 2009, EUnetHTA 2016), frameworks on digital health (without considering specifically public health) and frameworks on public health. Given the rapidly increasing digital technologies designed for public health interventions, a framework for digital public health interventions is deemed necessary. The current framework aims to be comprehensive; therefore, users need not draw on multiple frameworks for their assessments. Among others it considers technical aspects as well as e.g. issues around implementation, ethics or data security.
Each criterion in the framework is framed as a question. Users are encouraged to apply these questions to the digital public health intervention they want to develop and assess. Not all questions will be equally important or pertinent for all interventions, and some may not be applicable. There may not be any robust evidence available to answer the question or no available information at all.
The framework does not provide methodologies related to the questions. For some questions, it might be enough to use common sense; for others, specialist expertise may be necessary.
For an intervention under development, a first orientation might be enough to understand if it is worth continuing along the determined path or if adjustments might be necessary.
In summary, the application of the framework is primarily user-led. At a minimum, it can serve as a checklist that helps avoid overlooking key issues with relevance to the performance of the intervention.
The proposed framework is the result of a three-step process. First, a scoping review was conducted to identify existing Public Health and Digital Health frameworks for developing and evaluating health interventions in primary prevention and health promotion and their assessment criteria (OSF scoping review protocol registration). Second, all assessment criteria collected from the scoping review were analysed and mapped into domains based on the structure of the Health Technology Assessment (HTA) Core Model (Lampe et al. 2009; EUnetHTA, 2016). Finally, multidisciplinary Digital Public Health experts from the LSC were invited to a scientific consensus meeting to discuss and validate each framework domain.
The information sources for the scoping review included articles published in scientific journals. Journal articles were identified using the electronic literature search function of international bibliographic databases alongside the manual search of relevant reviews’ reference lists. The electronic bibliographic databases used for the literature search were MEDLINE (via PubMed), Scopus, IEEE, CINAHL (via EBSCO) and PsycINFO (via Ovid). The search syntax in the bibliographic databases included the following basic keywords, specific search fields, and Boolean operators: (“Public Health” [Title/Abstract] OR “Digital Health” [Title/Abstract]) AND Evaluation [Title] AND Framework [Title]. The search syntax included synonyms and the most relevant subject terms of our primary keywords in each concept. A truncated term with the wildcard character was used when appropriate to maximize sensitivity while striving for reasonable precision. The final search was executed on the 12th of April 2022 with no language or publication date limitations.
The inclusion criteria included English-written journal articles, reports or thesis that depicted a descriptive framework. The aim of the frameworks had to be to develop, monitor, validate, evaluate, or implement interventions related to public health or digital health, focusing on primary prevention or health promotion at a population level. A total of 9,011 articles were identified after searching in the databases. After deduplication, 4,830 titles and abstracts were screened by two researchers independently. Following, 433 full texts were assessed by two independent researchers for potential extraction. Disagreements between the two researchers at both stages were discussed among them. If no agreement was achieved, a third researcher was involved in making a conclusive decision. After the full-text screening, 68 articles were included for data extraction.
Data from all 68 articles were extracted, specifically including the framework criteria and sub-criteria. These were inductively analysed as a first step into clustered domains suggested by the EUnetHTA Health Technology Assessment (HTA) Core Model (Lampe et al. 2009, EUnetHTA 2016). The multidisciplinary experts from the Leibniz Science Campus (LSC) were assigned to domains to which they conferred and restructured according to specific characteristics of digital public health interventions. Where necessary additional literature was consulted.
A first version of the framework was sent to a multidisciplinary expert panel consisting of 105 members of the LSC. They were invited to give written feedback and to participate in an online scientific consensus meeting to discuss and validate each domain of the framework. A total of 25 members participated in the meeting on the 19th of July 2022.
As the field of Digital Public Health is rapidly evolving, the framework has been designed as a living framework. In the coming months, it will be applied and tested on a heterogeneous set of digital public health interventions. As this occurs, additional literature will be integrated, and experts beyond the LSC will be consulted.
If you find items missing, questions difficult to understand or operationalize or if you have applied the framework and would like to share your experiences, we are very keen to hear from you. Also, if you are interested in participating as an expert in the external consensus meeting, contact us with your expertise and a brief overview of your area of interest.
In these cases, or if there is other relevant feedback, please send an e-mail to: