Currently, there are frameworks that systematically assess the use of health-related technologies such as the Health Technology Assessment Core Model (Lampe et al. 2009, EUnetHTA 2016), frameworks on digital health (without considering specifically public health) and frameworks on public health. Given the rapidly increasing digital technologies designed for public health interventions, a framework for digital public health interventions is deemed necessary. The current framework aims to be comprehensive; therefore, users need not draw on multiple frameworks for their assessments. Among others it considers technical aspects as well as e.g. issues around implementation, ethics or data security.

Each criterion in the framework is framed as a question. Users are encouraged to apply these questions to the digital public health intervention they want to develop and assess. Not all questions will be equally important or pertinent for all interventions, and some may not be applicable. There may not be any robust evidence available to answer the question or no available information at all.

The framework does not provide methodologies related to the questions. For some questions, it might be enough to use common sense; for others, specialist expertise may be necessary.

For an intervention under development, a first orientation might be enough to understand if it is worth continuing along the determined path or if adjustments might be necessary.

In summary, the application of the framework is primarily user-led. At a minimum, it can serve as a checklist that helps avoid overlooking key issues with relevance to the performance of the intervention.

The proposed framework is the result of a three-step process. First, a scoping review was conducted to identify existing Public Health and Digital Health frameworks for developing and evaluating health interventions in primary prevention and health promotion and their assessment criteria (OSF scoping review protocol registration). Second, all assessment criteria collected from the scoping review were analysed and mapped into domains based on the structure of the Health Technology Assessment (HTA) Core Model (Lampe et al. 2009; EUnetHTA, 2016). Finally, multidisciplinary Digital Public Health experts from the LSC were invited to a scientific consensus meeting to discuss and validate each framework domain.

The information sources for the scoping review included articles published in scientific journals. Journal articles were identified using the electronic literature search function of international bibliographic databases alongside the manual search of relevant reviews’ reference lists. The electronic bibliographic databases used for the literature search were MEDLINE (via PubMed), Scopus, IEEE, CINAHL (via EBSCO) and PsycINFO (via Ovid). The search syntax in the bibliographic databases included the following basic keywords, specific search fields, and Boolean operators: (“Public Health” [Title/Abstract] OR “Digital Health” [Title/Abstract]) AND Evaluation [Title] AND Framework [Title]. The search syntax included synonyms and the most relevant subject terms of our primary keywords in each concept. A truncated term with the wildcard character was used when appropriate to maximize sensitivity while striving for reasonable precision. The final search was executed on the 12th of April 2022 with no language or publication date limitations.

The inclusion criteria included English-written journal articles, reports or thesis that depicted a descriptive framework. The aim of the frameworks had to be to develop, monitor, validate, evaluate, or implement interventions related to public health or digital health, focusing on primary prevention or health promotion at a population level. A total of 9,011 articles were identified after searching in the databases. After deduplication, 4,830 titles and abstracts were screened by two researchers independently. Following, 433 full texts were assessed by two independent researchers for potential extraction. Disagreements between the two researchers at both stages were discussed among them. If no agreement was achieved, a third researcher was involved in making a conclusive decision. After the full-text screening, 68 articles were included for data extraction.

Data from all 68 articles were extracted, specifically including the framework criteria and sub-criteria. These were inductively analysed as a first step into clustered domains suggested by the EUnetHTA Health Technology Assessment (HTA) Core Model (Lampe et al. 2009, EUnetHTA 2016). The multidisciplinary experts from the Leibniz Science Campus (LSC) were assigned to domains to which they conferred and restructured according to specific characteristics of digital public health interventions. Where necessary additional literature was consulted.

A first version of the framework was sent to a multidisciplinary expert panel consisting of 105 members of the LSC. They were invited to give written feedback and to participate in an online scientific consensus meeting to discuss and validate each domain of the framework. A total of 25 members participated in the meeting on the 19th of July 2022.

As the field of Digital Public Health is rapidly evolving, the framework has been designed as a living framework. In the coming months, it will be applied and tested on a heterogeneous set of digital public health interventions. As this occurs, additional literature will be integrated, and experts beyond the LSC will be consulted.

If you find items missing, questions difficult to understand or operationalize or if you have applied the framework and would like to share your experiences, we are very keen to hear from you. Also, if you are interested in participating as an expert in the external consensus meeting, contact us with your expertise and a brief overview of your area of interest.

In these cases, or if there is other relevant feedback, please send an e-mail to:
framework(@)lsc-digital-public-health.de