Latest publications - 2022

Franklin, M. et al (2022) Estimating Cost-Effectiveness Using Alternative Preference-Based Scores and Within-Trial Methods: Exploring the Dynamics of the Quality-Adjusted Life-Year Using the EQ-5D 5-Level Version and Recovering Quality of Life Utility Index

Authors - Matthew Franklin, Rachael Hunter, Angel Enrique, Jorge Palacios, and Derek Richards


This study aimed to explore quality-adjusted life-year (QALY) and subsequent cost-effectiveness estimates based on the more physical health–focused EQ-5D 5-level version (EQ-5D-5L) value set for England or cross-walked EQ-5D 3-level version UK value set scores or more mental health recovery-focused Recovering Quality of Life Utility Index (ReQoL-UI), when using alternative within-trial statistical methods. We describe possible reasons for the different QALY estimates based on the interaction between item scores, health state profiles, preference-based scores, and mathematical and statistical methods chosen. QALYs are calculated over 8 weeks from a case study 2:1 (intervention:control) randomized controlled trial in patients with anxiety or depression. Complete case and with missing cases imputed using multiple-imputation analyses are conducted, using unadjusted and regression baseline-adjusted QALYs. Cost-effectiveness is judged using incremental cost-effectiveness ratios and acceptability curves. We use previously established psychometric results to reflect on estimated QALYs. A total of 361 people (241:120) were randomized. EQ-5D-5L crosswalk produced higher incremental QALYs than the value set for England or ReQoL-UI, which produced similar unadjusted QALYs, but contrasting baseline-adjusted QALYs. Probability of cost-effectiveness <£30 000 per QALY ranged from 6% (complete case ReQoL-UI baseline-adjusted QALYs) to 64.3% (multiple-imputation EQ-5D-5L crosswalk unadjusted QALYs). The control arm improved more on average than the intervention arm on the ReQoL-UI, a result not mirrored on the EQ-5D-5L nor condition-specific (Patient-Health Questionnaire-9, depression; Generalized Anxiety Disorder-7, anxiety) measures. ReQoL-UI produced contradictory cost-effectiveness results relative to the EQ-5D-5L. The EQ-5D-5L’s better responsiveness and “anxiety/depression” and “usual activities” items drove the incremental QALY results. The ReQoL-UI’s single physical health item and “personal recovery” construct may have influenced its lower 8-week incremental QALY estimates in this patient sample.


Young, T. et al (2022) Proceedings of the Patient Reported Outcome Measures (PROMs) Research Conference, Sheffield 2021

Authors - Tracey Young, Jill Carlton, Phil Powell, Laura Flight, Alexis Foster, Lucy Musson, Amanda Lane, Kathryn Mackellar, Patricia Holch, Grace Turner


The 5th UK Patient Reported Outcome Measures (PROMs) Research Conference (#UKPROMS) held on the 16th and 17th June 2021 was attended by 154 academics, clinicians, PhD students and patients, representing 105 organisations. As the conference was held online, due to the Covid-19 pandemic, it attracted an international audience, including delegates from the United States, Canada, Australia, Europe, Hong Kong, Jordan and Africa. The aim of the conference was to bring together leading experts and early career researchers to engage in the latest advances in the field of PROMs research and implementation. The conference focused on researcher-led activities on methodologies around the development, testing and use of PROMs in different contexts and settings. The conference made a return to Sheffield having previously been hosted by the Universities of Oxford, Birmingham and Leeds Beckett.


Carlton, J. et al (2022) Generation, selection and face validation of items for a new generic measure of quality of life, the EQ Health and Wellbeing (EQ-HWB).

Authors - Jill Carlton, Tessa Peasgood, Clara Mukuria, Janice Connell, John Brazier, Kristina Ludwig,

Ole Marten, Simone Kreimeier, Lidia Engel, Maria Belizán, Zhihao Yang, Andrea Monteiro, Maja Kuharic, Nan Luo, Brendan Mulhern, Wolfgang Greiner, Simon Pickard, Federico Augustovski


This article aims to describe the generation and selection of items (stage 2) and face validation (stage 3) of a large international (multilingual) project to develop a new generic measure, the EQ-HWB (EQ Health and Wellbeing), for use in economic evaluation across health, social care, and public health to estimate quality-adjusted life-years. Items from commonly used generic, carer, social care, and mental health quality of life measures were mapped onto domains or subdomains identified from a literature review. Potential terms and items were reviewed and refined to ensure coverage of the construct of the domains/subdomain (stage 2). Input on the potential item pool, response options, and recall period was sought from 3 key stakeholder groups. The pool of candidate items was tested in qualitative interviews with potential future users in an international face validation study (stage 3). Stage 2 resulted in the generation of 687 items. Predetermined selection criteria were applied by the research team resulting in 598 items being dropped, leaving 89 items that were reviewed by key stakeholder groups. Face validation (stage 3) tested 97 draft items and 4 response scales. A total of 47 items were retained and 14 were modified, whereas 3 were added to the candidate pool of items. This resulted in a 64-item set. This international multiculture, multilingual study with a common methodology identified many items that performed well across all countries. These were taken to the psychometric testing along with modified and new items for the EQ-HWB. Keywords: EQ-HWB, face validity, interviews, item selection, quality of life. VALUE HEALTH. 2022; 25(4):512–52

Mukuria, C. et al (2022) A targeted review of qualitative evidence on domains of quality of life important for patients, social care users and carers to inform the development of the EQ Health and Wellbeing (EQ-HWB)

Authors – Clara Mukuria, Janice Connell, Jill Carlton, Tessa Peasgood, Alison Scope, Mark Clowes ,Stacey Rand, Karen Jones, John Brazier


To identify the themes to inform the content of a new generic measure, the EQ-HWB (EQ Health and Wellbeing), that can be used in economic evaluation across health, social care, and public health, based on the views of users and beneficiaries of these services including informal carers. A qualitative review was undertaken. Systematic and citation searches were undertaken focusing on qualitative evidence of the impact on quality of life from reviews for selected health conditions, informal carers, social care users, and primary qualitative work used in the development of selected measures. A subset of studies was included in the review. Framework analysis and synthesis were undertaken based on a conceptual model. A total of 42 reviews and 24 primary studies were selected for inclusion in the review. Extraction and synthesis resulted in 7 high-level themes (with subthemes): (1) feelings and emotions (sadness, anxiety, hope, frustration, safety, guilt/shame); (2) cognition (concentration, memory, confusion, thinking clearly); (3) self-identity (dignity/respect, self-esteem); (4) “coping, autonomy, and control” relationships; (5) social connections (loneliness, social engagement, stigma, support, friendship, belonging, burden); (6) physical sensations (pain, discomfort, sleep, fatigue); and (7) activity (self-care, meaningful activities, mobility, communication, hearing, vision). Apart from physical sensations, most of the other themes and subthemes were relevant across both health and social care, including for informal carers. The findings from this broad review identified themes that go beyond health and that are relevant to patients, informal carers, and social care users. The themes and subthemes informed the domains for the EQ-HWB.

Augustovski, F. et al (2022) The Development of a New International Generic Measure (EQ Health and Wellbeing): Face Validity And Psychometric Stages In Argentina

Authors – Federico Augustovski, Fernando Argento, Rocío B.Rodríguez, Luz Gibbons, Clara Mukuria, María Belizán


This study aimed to present the face validity and psychometric stages performed in Spanish in Argentina, the only Spanish-speaking country of an international collaboration that undertook the construction of a new measure that can be used in economic evaluation across health, social care, and public health, the EQ EQ-HWB (EQ Health and Wellbeing). We also explored the relationship among 3-level version EQ-5D (EQ-5D-3L), 5-level version EQ-5D (EQ-5D-5L), and EQ-HWB. Face validity was based on semistructured face to face interviews of a purposive sample to explore translatability of language and concepts of 97 candidate items, translated into Argentina Spanish. The psychometric evaluation using an online panel assessed the psychometric properties of 64 items that were carried forward (floor and ceiling effects, item correlations, known-group differences in relevant prespecified subgroups by the international and local teams, exploratory and confirmatory factor analysis, and item response theory). EQ-5D-3L, EQ-5D-5L, and EQ-HWB correlations were explored. In the face validity stage, 24 interviews with carers, general public, patients, and users of social services were included. Most items showed adequate face validity. In the psychometric assessment, 497 participants were recruited (64% reporting a long-term health condition). Most of the items showed adequate psychometrics in an Argentinian context. EQ-5D-3L and EQ-5D-5L had strong correlations, and EQ-HWB was moderately correlated to EQ visual analog scale. The Argentina team recommended 23 of the final 25 items. The assessment of Spanish items contributed to the overall development of EQ-HWB and helped inform the design of an internationally relevant 25-item and a short 9-item measure intended to be used in economic evaluations.

Peasgood, T. et al (2022) Developing a new generic health and wellbeing measure: psychometric survey results for the EQ Health and Wellbeing (EQ-HWB)

Authors – Tessa Peasgood, Clara Mukuria, John Brazier, Ole Marten, Simone Kreimeier, Nan Luo, Brendan Mulhern, Wolfgang Greiner, Simon Pickard, Federico Augustovski, Lidia Engel, Luz Gibbons, Zhihao Yang, Andrea L. Monteiro, Maja Kuharic, Maria Belizan, Jakob Bjørner


The development of measures such as the EQ-HWB (EQ Health and Wellbeing) requires selection of items. This study explored the psychometric performance of candidate items, testing their validity in patients, social carer users, and carers. Article and online surveys that included candidate items (N = 64) were conducted in Argentina, Australia, China, Germany, United Kingdom, and the United States. Psychometric assessment on missing data, response distributions, and known group differences was undertaken. Dimensionality was explored using exploratory and confirmatory factor analysis. Poorly fitting items were identified using information functions, and the function of each response category was assessed using category characteristic curves from item response theory (IRT) models. Differential item functioning was tested across key subgroups. There were 4879 respondents (Argentina = 508, Australia = 514, China = 497, Germany = 502, United Kingdom = 1955, United States = 903). Where missing data were allowed, it was low (UK article survey 2.3%; US survey 0.6%). Most items had responses distributed across all levels. Most items could discriminate between groups with known health conditions with moderate to large effect sizes. Items were less able to discriminate across carers. Factor analysis found positive and negative measurement factors alongside the constructs of interest. For most of the countries apart from China, the confirmatory factor analysis model had good fit with some minor modifications. IRT indicated that most items had well-functioning response categories but there was some evidence of differential item functioning in many items. Items performed well in classical psychometric testing and IRT. This large 6-country collaboration provided evidence to inform item selection for the EQ-HWB measure.

Bywater, T. et al (2022)A randomized controlled trial of a proportionate universal parenting program delivery model (E-SEE Steps) to enhance child social-emotional wellbeing

Authors - Tracey Bywater ,Vashti Berry, Sarah Blower, Matthew Bursnall, Edward Cox, Amanda Mason-Jones, Sinead McGilloway, Kirsty McKendrick, Siobhan Mitchell, Kate Pickett, Gerry Richardson, Kiera Solaiman, M. Dawn Teare, Simon Walker, Karen Whittaker


Evidence for parenting programs to improve wellbeing in children under three is inconclusive. We investigated the fidelity, impact, and cost-effectiveness of two parenting programs delivered within a longitudinal proportionate delivery model (‘E-SEE Steps’). Eligible parents with a child ≤ 8 weeks were recruited into a parallel two-arm, assessor blinded, randomized controlled, community-based, trial with embedded economic and process evaluations. Post-baseline randomization applied a 5:1 (intervention-to-control) ratio, stratified by primary (child social-emotional wellbeing (ASQ:SE-2)) and key secondary (maternal depression (PHQ-9)) outcome scores, sex, and site. All intervention parents received the Incredible Years® Baby Book (IY-B), and were offered the targeted Infant (IY-I)/Toddler (IY-T) program if eligible, based on ASQ:SE-2/PHQ-9 scores. Control families received usual services. Fidelity data were analysed descriptively. Primary analysis applied intention to treat. Effectiveness analysis fitted a marginal model to outcome scores. Cost-effectiveness analysis involved Incremental Cost-Effectiveness Ratios (ICERs). The target sample (N = 606) was not achieved; 341 mothers were randomized (285:56), 322 (94%) were retained to study end. Of those eligible for the IY-I (n = 101), and IY-T (n = 101) programs, 51 and 21 respectively, attended. Eight (of 14) groups met the 80% self-reported fidelity criteria. No significant differences between arms were found for adjusted mean difference scores; ASQ:SE-2 (3.02, 95% CI: -0.03, 6.08, p = 0.052), PHQ-9 (-0.61; 95% CI: -1.34, 0.12, p = 0.1). E-SEE Steps had higher costs, but improved mothers’ Health-related Quality of Life (0.031 Quality Adjusted Life Year (QALY) gain), ICER of £20,062 per QALY compared to control. Serious adverse events (n = 86) were unrelated to the interventions. E-SEE Steps was not effective, but was borderline cost-effective. The model was delivered with varying fidelity, with lower-than-expected IY-T uptake. Changes to delivery systems and the individual programs may be needed prior to future evaluation.


Hinde, S. et al (2021) The cost effectiveness of ecotherapy as a healthcare intervention, separating the wood from the tree

Authors - Sebastian Hinde, Laura Bojke, Peter Coventry


Internationally, shifts to more urbanised populations, and resultant reductions in engagements with nature, have been a contributing factor to the mental health crisis facing many developed and developing countries. While the COVID-19 pandemic reinforced recent trends in many countries to give access to green spaces more weight in political decision making, nature-based activities as a form of intervention for those with mental health problems constitute a very small part of patient pathways of care. Nature-based interventions, such as ecotherapy, are increasingly used as therapeutic solutions for people with common mental health problems. However, there is little data about the potential costs and benefits of ecotherapy, making it difficult to offer robust assessments of its cost-effectiveness. This paper explores the capacity for ecotherapy to be cost-effective as a healthcare intervention. Using a pragmatic scoping review of the literature to understand where the potential costs and health benefit lie, we applied value of information methodology to identify what research is needed to inform future cost-effectiveness assessments. We show that there is the potential for ecotherapy for people with mild to moderate common mental health problems to be cost-effective but significant further research is required. Furthermore, nature-based interventions such as ecotherapy also confer potential social and wider returns on investment, strengthening the case for further research to better inform robust commissioning.

Brown, L. et al (2021) Community Ageing Research 75+ (CARE75+) REMOTE study: a remote model of recruitment and assessment of the health, well-being and social circumstances of older people

Authors - Lesley Brown, Anne Heaven, Catherine Quinn, Victoria Goodwin ,Carolyn Chew-Graham, Farhat Mahmood, Sarah Hallas, I khlaq Jacob, Caroline Brundle, Kate Best, Amrit Daffu-O'Reilly, Karen Spilsbury, Tracey Anne Young, Rebecca Hawkins, Barbara Hanratty ,Elizabeth Teale, Andrew Clegg


The Community Ageing Research 75+ (CARE75+) study is a longitudinal cohort study collecting extensive health and social data, with a focus on frailty, independence and quality of life in older age. CARE75+ was the first international experimental frailty research cohort designed using trial within cohorts (TwiCs) methodology, aligning epidemiological research with clinical trial evaluation of interventions to improve the health and well-being of older people. CARE75+ REMOTE is an extension of CARE75+ using a remote model that does not require face-to-face interactions for data collection in the current circumstances of a global pandemic and will provide an efficient, sustainable data collection model. Prospective cohort study using TwiCs. One thousand community-dwelling older people (≥75 years) will be recruited from UK general practices by telephone. Exclusions include: nursing home/care home residents; those with an estimated life expectancy of 3 months or less; and people receiving palliative care. Assessments will be conducted by telephone, web-submission or postal questionnaire: baseline, 6 months, 12 months, 18 months, 24 months, 30 months and 36 months. Measures include activities of daily living, mood, health-related quality of life, comorbidities, medications, frailty, informal care, healthcare and social care service use. Consent will be sought for data linkage and invitations to additional studies (sub-studies). CARE75+ was approved by the National Research Ethics Service (NRES) Committee Yorkshire and the Humber-Bradford Leeds 10 October 2014 (14/YH/1120). CARE75+ REMOTE (amendment 13) was approved on the 18th November 2020. Consent is sought if an individual is willing to participate and has capacity to provide informed consent. Consultee assent is sought if an individual lacks capacity. Results will be disseminated in peer-reviewed scientific journals and conferences. Results will be summarised and disseminated to study participants via newsletters, local engagement events and on a bespoke website.

Franklin, M. et al (2021) Correspondence on "How can we estimate QALYs based on PHQ-9 scores? Equipercentile linking analysis of PHQ-9 and EQ-5D" by Furukawa et al

Authors – Matt Franklin, Tracey Young


Furukawa et al posed the question: how can we estimate quality-adjusted life years (QALYs) based on Patient Health Questionnaire-9 (PHQ-9) scores? They recommend equipercentile linking analysis between the depression severity PHQ-9 and preference-based EQ-5D three-level version (EQ-5D-3L; UK value set), the latter used to estimate utility data for QALYs.

Franklin, M. et al (2021) Psychometric assessment of EQ-5D-5L and ReQoL measures in patients with anxiety and depression: construct validity and responsiveness

Authors - Matthew Franklin, Angel Enrique, Jorge Palacios & Derek Richards


Generic health measures have been questioned for quantifying mental-health-related outcomes. In patients with anxiety and/or depression, our aim is to assess the psychometric properties of the preference-based EQ-5D-5L (generic health) and ReQoL-UI (recovery-focussed quality of life) for economic evaluation against the PHQ-9 (depression) and GAD-7 (anxiety). EQ-5D-5L anxiety/depression item and ReQoL-10 are also assessed. A 2:1 (intervention: control) randomised controlled trial collected measures at baseline and 8 weeks post baseline; in the intervention arm, data were also collected 3, 6, 9, and 12-months post baseline. EQ-5D-5L preference-based scores were obtained from the value set for England (VSE) and ‘cross-walked’ EQ-5D-3L United Kingdom (UK) value set scores. ReQoL-UI preference-based scores were obtained from its UK value set as applied to seven ReQoL-10 items. EQ-5D-5L and ReQoL measures’ construct validity and responsiveness were assessed compared against PHQ-9 and GAD-7 scores and group cut-offs. 361 people were randomised to intervention (241) or control (120). ReQoL-UI/-10 had better construct validity with depression severity than the EQ-5D-5L (VSE/cross-walk scores), which had relatively better construct validity with anxiety severity than the ReQoL-UI/-10. Across all intervention-arm time-points relative to baseline, responsiveness was generally better for EQ-5D-5L (VSE in particular) than ReQoL-UI, but worse than ReQoL-10. There is insufficient evidence to recommend the ReQoL-UI over EQ-5D-5L for economic evaluations to capture anxiety severity. However, there may be rationale for recommending the ReQoL-UI over the EQ-5D-5L to capture depression severity given its better construct validity, albeit poorer responsiveness, and if recovery-focussed quality of life relative to condition-specific symptomology is the construct of interest.

Pilbery, R. et al The effect of a specialist paramedic primary care rotation on appropriate non-conveyance decisions (SPRAINED) study: a controlled interrupted time series analysis

Authors - Richard Pilbery, Tracey Young, Andrew Hodge


NHS ambulance service conveyance rates in the UK are almost 70%, despite an increase in non-emergency cases. This is increasing the demands on crowded emergency departments (ED) and contributes to increased ambulance turnaround times. Yorkshire Ambulance Service introduced a specialist paramedic (SP) role to try and address this, but non-conveyance rates in this group have not been as high as expected. We conducted a controlled interrupted time series analysis using data from incidents between June 2017 and December 2019, to study appropriate non-conveyance rates before and after a GP placement. A costing analysis examined the average cost per appropriate non-conveyance achieved for patients receiving care from intervention group SPs pre- and post-placement was also conducted. 7349 incidents attended by intervention group SPs were eligible for inclusion. Following removal of cases with missing data, 5537/7349 (75.3%) cases remained. Post-placement, the intervention group demonstrated an increase in appropriate non-conveyance rate of 35.0% (95%CI 23.8–46.2%, p<0.001), and a reduction in the trend of appropriate non-conveyance of −1.2% (95%CI −2.8–0.5%, p=0.156), relative to the control group. Post-placement, the cost per appropriate non-conveyance for intervention group SPs was a mean of £509.38 (95% bootstrapped CI £455.32–£564.59) versus £1258.04 (95% bootstrapped CI £1232.64–£1284.04) for the same group in the pre-placement phase. This represents a mean saving of £748.66 per appropriate non-conveyance (95% bootstrapped CI £719.45–£777.32) and a cost-effectiveness ratio of £2141.15 per percentage increase in appropriate non-conveyance (95% bootstrapped CI £2057.62–£2223.12). In this single UK NHS ambulance service study, we found a clinically important and statistically significant increase in appropriate non-conveyance rates by specialist paramedics who had completed a 10-week GP placement. This improvement persisted for the 12-month period following the placement and demonstrated cost savings compared to usual care.

Peasgood, T. et al (2021) The Role of Response Domain and Scale Label in the Quantitative Interpretation of Patient-Reported Outcome Measure Response Options

Authors - Tessa Peasgood, Jen-Yu Chang, Robina Mir, Clara Mukuria, Philip A. Powell


Uncertainties exist in how respondents interpret response options in patient-reported outcome measures (PROMs), particularly across different domains and for different scale labels. The current study assessed how respondents quantitatively interpret common response options. Members of the general public were recruited to this study via an online panel, stratified by age, gender, and having English as a first language. Participants completed background questions and were randomised to answer questions on one of three domains (i.e. loneliness (negatively phrased), happiness or activities (positively phrased)). Participants were asked to provide quantitative interpretations of response options (e.g. how many times per week is equal to “often”) and to order several common response options (e.g. occasionally, sometimes) on a 0–100 slider scale. Chi-squared tests and regression analyses were used to assess whether response options were interpreted consistently across domains and respondent characteristics. Data from 1377 participants were analysed. There was general consistency in quantifying the number of times over the last 7 days to which each response option referred. Response options were consistently assigned a lower value in the loneliness than happiness and activities domains. Individual differences, such as age and English as a second language, explained some significant variation in responses, but less than domain. Members of the public quantify common response options in a similar way, but their quantification is not equivalent across domains or every type of respondent. Recommendations for the use of certain scale labels over others in PROM development are provided.

Hinde, S. et al (2021) Delayed Transfers of care for older people: a wider perspective

Authors - Sebastian Hinde, Laura Bojke, Gerry Richardson, Yvonne Birks, William Whittaker, Mark Wilberforce, Andrew Clegg


Delayed transfers of care (DTOC), often unhelpfully referred to as ‘bed blocking’, has become a byword for waste and inefficiency in healthcare systems throughout the world. An estimated 2.7 million bed days are occupied each year in England by older people no longer in need of acute treatment, estimated to cost £820 million (2014/15) in inpatient care. Policy and media attention have often been drawn to this narrative of financial waste, resulting in policy setting that directly targets the level of DTOC, but has done little to put patient health first. These figures and policies portray a misleading image of the delays as primarily of concern in terms of their financial burden on acute hospital care, with little consideration given to the quantification on patient health or wider societal impacts. In spite of the multi-factorial decision-making process that occurs for each patient discharge, current evaluation frameworks and national policy setting fail to reflect the complexity of the process. In this commentary, we interrogate the current approach to the quantification of the DTOC impact and explore how policies and evaluation methods can do more to reflect the true impact of the delays.

Candio, P. et al (2021) Copula Models for Addressing Sample Selection in the Evaluation of Public Health Programmes: An Application to the Leeds Let’s Get Active Study

Authors - Paolo Candio, Andrew J. Hill, Stavros Poupakis, Anni-Maria Pulkki-Brännström, Chris Bojke & Manuel Gomes


Sample selectivity is a recurrent problem in public health programmes and poses serious challenges to their evaluation. Traditional approaches to handle sample selection tend to rely on restrictive assumptions. The aim of this paper is to illustrate a copula-based selection model to handle sample selection in the evaluation of public health programmes. Motivated by a public health programme to promote physical activity in Leeds (England), we describe the assumptions underlying the copula selection, and its relative advantages compared with commonly used approaches to handle sample selection, such as inverse probability weighting and Heckman’s selection model. We illustrate the methods in the Leeds Let’s Get Active programme and show the implications of method choice for estimating the effect on individual’s physical activity. The programme was associated with increased physical activity overall, but the magnitude of its effect differed according to adjustment method. The copula selection model led to a similar effect to the Heckman’s approach but with relatively narrower 95% confidence intervals. These results remained relatively similar when different model specifications and alternative distributional assumptions were considered. The copula selection model can address important limitations of traditional approaches to address sample selection, such as the Heckman model, and should be considered in the evaluation of public health programmes, where sample selection is likely to be present.

Kwon, J. et al (2021) Qualitative research to inform economic modelling: a case study in older people’s views on implementing the NICE falls prevention guideline

Authors - Joseph Kwon, Yujin Lee, Tracey Young, Hazel Squires, Janet Harris


High prevalence of falls among older persons makes falls prevention a public health priority. Yet community-based falls prevention face complexity in implementation and any commissioning strategy should be subject to economic evaluation to ensure cost-effective use of healthcare resources. The study aims to capture the views of older people on implementing the National Institute for Health and Care Excellence (NICE) guideline on community-based falls prevention and explore how the qualitative data can be used to inform commissioning strategies and conceptual modelling of falls prevention economic evaluation in the local area of Sheffield. Focus group and interview participants (n = 27) were recruited from Sheffield, England, and comprised falls prevention service users and eligible non-users of varying falls risks. Topics concerned key components of the NICE-recommended falls prevention pathway, including falls risk screening, multifactorial risk assessment and treatment uptake and adherence. Views on other topics concerning falls prevention were also invited. Framework analysis was applied for data analysis, involving data familiarisation, identifying themes, indexing, charting and mapping and interpretation. The qualitative data were mapped to three frameworks: (1) facilitators and barriers to implementing the NICE-recommended pathway and contextual factors; (2) intervention-related causal mechanisms for formulating commissioning strategies spanning context, priority setting, need, supply and demand; and (3) methodological and evaluative challenges for public health economic modelling. Two cross-component factors were identified: health motives of older persons; and professional competence. Participants highlighted the need for intersectoral approaches and prioritising the vulnerable groups. The local commissioning strategy should consider the socioeconomic, linguistic, geographical, legal and cultural contexts, priority setting challenges, supply-side mechanisms spanning provider, organisation, funding and policy (including intersectoral) and health and non-health demand motives. Methodological and evaluative challenges identified included: incorporating non-health outcomes and societal intervention costs; considering dynamic complexity; considering social determinants of health; and conducting equity analyses. Holistic qualitative research can inform how commissioned falls prevention pathways can be feasible and effective. Qualitative data can inform commissioning strategies and conceptual modelling for economic evaluations of falls prevention and other geriatric interventions. This would improve the structural validity of quantitative models used to inform geriatric public health policies.

Blower, S. et al (2021) Enhancing Social-Emotional Outcomes in Early Years (E-SEE): Randomized Pilot Study of Incredible Years Infant and Toddler Programs

Authors - Sarah L. Blower, Vashti L. Berry, Matthew C. Bursnall, Judith Cohen, Nicole Gridley, Amanda Loban, Laura Mandefield, Amanda J. Mason-Jones, Sinéad McGilloway, Kirsty L. McKendrick, Siobhan B. Mitchell, Kate E. Pickett, Gerry A. Richardson, M. Dawn Teare, Louise C. Tracey, Simon M. Walker, Karen A. Whittaker, Jessica Wright & Tracey J. Bywater


Social emotional development in infancy is a predictor of outcomes in later life, yet there is little evidence of effectiveness for parenting interventions designed to enhance social emotional wellbeing in infancy. An 18-month two-arm randomized controlled pilot trial evaluated the feasibility of a definitive trial of Incredible Years (IY) Infant and Toddler parent programs delivered in a proportionate universal model, called Enhancing Social-Emotional Health and Wellbeing in the Early Years (E-SEE) Steps. Intervention families received an IY Babies book (universal dose), followed by the IY Infant and/or the Toddler group-based programs, based on parent depression (PHQ-9) and/or child social emotional development (ASQ:SE-2) scores. Control parents received services as usual. Parents from two English local authorities with a child eight-weeks-old or younger participated, and were block randomized using a web-based system. Primary endpoints for the study were feasibility parameters relating to recruitment, retention, intervention fidelity and appropriateness of measures. 205 participants were randomized (152:53, intervention:control). Our target was 288 parents. Trial retention rate was higher than expected, with a completion rate of 88% (n = 181, 137:44) at follow-up 3; equating to 94% of 192 expected participants. Intervention uptake was lower than expected. Fidelity of delivery was acceptable and measures were deemed appropriate. A definitive trial is feasible with design amendments to include: introduction of a child screener for intervention eligibility; enhanced intervention material; revised sample size and random allocation ratio. Our internal pilot became an external pilot due to these changes.

Peasgood, T. et al (2021) What is the best approach to adopt for identifying the domains for a new measure of health, social care and carer-related quality of life to measure quality adjusted life years? Application to the development of the EQ-HWB?

Authors - Tessa Peasgood, Clara Mukuria, Jill Carlton, Janice Connell, Nancy Devlin, Karen Jones, Rosemary Lovett, Bhash Naidoo, Stacey Rand, Juan Carlos Rejon-Parrilla, Donna Rowen, Aki Tsuchiya & John Brazier


Economic evaluation combines costs and benefits to support decision-making when assessing new interventions using preference-based measures to measure and value benefits in health or health-related quality of life. These health-focused instruments have limited ability to capture wider impacts on informal carers or outcomes in other sectors such as social care. Sector-specific instruments can be used but this is problematic when the impact of an intervention straddles different sectors. An alternative approach is to develop a generic preference-based measure that is sufficiently broad to capture important cross-sector outcomes. We consider the options for the selection of domains for a cross-sector generic measure including how to identify domains, who should provide information on the domains and how this should be framed. Beyond domain identification, considerations of criteria and stakeholder needs are also identified. This paper sets out the case for an approach that relies on the voice of patients, social care users and informal carers as the main source of domains and describes how the approach was operationalised in the ‘Extending the QALY’ project which developed the new measure, the EQ-HWB (EQ health and wellbeing instrument). We conclude by discussing the strengths and limitations of this approach. The new measure should be sufficiently generic to be used to consistently evaluate health and social care interventions, yet also sensitive enough to pick up important changes in quality of life in patients, social care users and carers.

Mukuria, C. et al (2021) Applying EuroQol Portable Valuation Technology to the EQ Health and Wellbeing Short (EQHWB-S) : a pilot study

Authors – Clara Mukuria, Tessa Peasgood, John Brazier


The aim was to assess whether existing valuation methods were suitable for the nine item EQ Health and Wellbeing ShortTM (EQ‐HWB‐S). EuroQol Portable Valuation Technology (EQ‐PVT) which uses composite time trade‐off (cTTO) and discrete choice experiments (DCE) was modified for the EQ‐HWB‐S TM. Volunteer non‐ academic University of Sheffield staff were recruited. A mixed methods approach involving qualitative interviews and assessment of quantitative data was used to assess the applicability and feasibility of EQ‐PVT to EQ‐HWB‐S. Participants valued six states using cTTO (three EQ‐HWB‐S and three EQ‐5D‐ 5LTM) and four EQ‐HWB‐S states using DCE. Nineteen participants with mean (SD) age 48.2 (13.0) were interviewed. Mean TTO values were ordered as expected with higher mean values for the mild EQ‐HWB state compared to the moderate and severe states. Most participants found it fairly or very easy to understand cTTO questions for both EQ‐HWB‐S 94.7% (18/19) and EQ‐5D 89.5% (17/19). Pain, activities and depression were considered key drivers for respondents’ choices. Additional information in the EQ‐HWB‐S was useful in helping to imagine what life would be like but it could also be overwhelming and make the tasks difficult. ‘Coping’ was a problematic item as it was either used as an overall assessment of the state or ignored in favour of participants’ perceived ability to cope with the state. ‘Coping’ was replaced with ‘control’ which did not have the same problems. Participants generally preferred DCE to TTO. DCE presentations with overlap but with simple formatting were preferred. A modified standardised valuation has been successfully applied to health and wellbeing states defined by the EQ‐HWB‐S. A full feasibility study is now required.


Moore, s. et al (2020) Controlled observational study and economic evaluation of the effect of city-centre night-time alcohol intoxication management services on the emergency care system compared with usual care

Authors - Simon Moore, Tracey Young, Andy Irving, Steve Goodacre, Alan Brennan, Yvette Amos


Alcohol intoxication management services (AIMS) provide an alternative care pathway for alcohol-intoxicated adults otherwise requiring emergency department (ED) services and at times of high incidence. We estimate the effectiveness and cost-effectiveness of AIMS on ED attendance rates with ED and ambulance service performance indicators as secondary outcomes. A controlled longitudinal retrospective observational study in English and Welsh towns, six with AIMS and six without. Control and intervention cities were matched by sociodemographic characteristics. The primary outcome was ED attendance rate per night, secondary analyses explored hospital admission rates and ambulance response times. Interrupted time series analyses compared control and matched intervention sites pre-AIMS and post-AIMS. Cost-effectiveness analyses compared the component costs of AIMS to usual care before with results presented from the National Health Service and social care prospective. The number of diversions away from ED required for a service to be cost neutral was determined. Analyses found considerable variation across sites, only one service was associated with a significant reduction in ED attendances (−4.89, p<0.01). The services offered by AIMS varied. On average AIMS had 7.57 (mean minimum=1.33, SD=1.37 to mean maximum=24.66, SD=12.58) in attendance per session, below the 11.02 diversions away from ED at which services would be expected to be cost neutral. AIMSs have variable effects on the emergency care system, reflecting variable structures and processes, but may be associated with modest reductions in the burden on ED and ambulance services. The more expensive model, supported by the ED, was the only configuration likely to divert patients away from ED. AIMS should be regarded as fledgling services that require further work to realise benefit.

Foster, A. et al (2020 How do third sector organisations or charities providing health and well-being services in England implement patient-reported outcome measures (PROMs)? A qualitative interview study

Authors - Alexis Foster, Alicia O'Cathain, Janet Harris


To identify the facilitators and barriers to implementing patient-reported outcome measures (PROMs) in third sector organisations (TSOs) delivering health and well-being services. A qualitative interview study. Participants were recruited using purposive, opportunistic and snowballing methods. Framework analysis was used. TSOs including charities, community groups and not-for-profit organisations in England, UK. Thirty interviewees including service users, TSO front-line workers and managers, commissioners of TSOs and other stakeholders such as academic researchers. TSOs primarily used PROMs because of pressures arising from the external funding context. However, organisations often struggled to implement PROMs, rarely getting the process right first time. Facilitators for implementation included having an implementation lead committed to making it work, investing resources in data management systems and support staff and taking a collaborative approach to designing the PROMs process. The latter helped to ensure an appropriate PROMs process for the specific TSO including choosing a suitable measure and planning how data would be collected, processed and used. There was a dilemma about whether TSOs should use standardised well-being measures (e.g., the Warwick-Edinburgh Mental Well-being Scale) or design their own PROM. Not all TSOs sustained the collection and reporting of PROMs over time because this required a change in organisational culture to view PROMs as beneficial for the TSO and PROMs becoming part of front-line workers' job specifications. TSOs are trying to use PROMs because they feel they have no choice but often struggle with implementation. Having an implementation lead, designing an appropriate process, investing resources, training staff and taking mitigating action to address potential barriers can facilitate implementation. Some of the findings are consistent with the experiences of more clinical services so appear relevant to the implementation of PROMs irrespective of the specific context.

Candio, P. et al (2020) Modelling the impact of physical activity on public health: A review and critique

Authors - Paolo Candio, David Meads, Andrew J. Hill, Laura Bojke


While several reviews have assessed economic evaluations of physical activity in public health and, in most cases, found the interventions to be cost-effective, the validity of the conclusions reached depends on the appropriateness of the modelling methods used in the individual studies. To provide an overview and critique of modelling approaches and key structural assumptions used in applied studies to estimate the impact of physical activity on population health. Electronic databases were systematically searched for relevant model-based economic evaluations. A thematic approach was used to assess the modelling studies. The critique determined the appropriateness of the modelling frameworks and plausibility of key structural assumptions. Twenty-five models were identified. Cohort models were most frequently used. High variability in the modelling of downstream diseases was found across studies analysing similar populations. Structural assumptions regarding the dynamics of change of physical activity were unrealistic in most cases. Heterogeneity was addressed in only a few studies, while health equity concerns were, at best, acknowledged by authors. This literature is predominantly characterised by modelling approaches that may not adequately address the complexities associated with representing the physical activity behaviour- population health process. A consensus on how to model the impact of physical activity on public health and development of a reference model could help reduce these sources of uncertainty.

Candio, P. et al (2020) Cost-effectiveness of a proportionate universal offer of free exercise: Leeds Let’s Get Active

Authors - Paolo Candio, David Meads, Andrew J Hill, Laura Bojke


The purpose of this paper is to assess the cost-effectiveness of a proportionate universal programme to reduce physical inactivity (Leeds Let us Get Active (LLGA)) in adults. A continuous-time Markov chain model was developed to assess the cost implications and QALY gains associated with increases in physical activity levels across the adult population. A parametric survival analysis approach was applied to estimate the decay of intervention effect over time. Baseline model data were obtained from previous economic models, population-based surveys and other published literature. A cost-utility analysis was conducted from a health care sector perspective over the programme duration (39 months). Scenario and probabilistic sensitivity analyses were performed to test the robustness of cost-effectiveness results. In total, 51 874 adult residents registered to the programme and provided baseline data,19.5% of which were living in deprived areas. Under base case assumptions, LLGA was found to be likely to be cost-effective. However, variations in key structural assumptions showed sensitivity of the results. Results from this study suggest a non-negligible level of uncertainty regarding the effectiveness, and therefore, cost-effectiveness of a universal offer of free leisure centre-based exercise that targets hard to reach groups. Further data collection and a shift towards prospective evaluations are needed.

Taylor Buck, L. et al (2020) Use of a modified World Café process to discuss and set priorities for a Community of Practice supporting implementation of ReQoL a new mental health and quality of life Patient Reported Outcome Measure (PROM).

Authors – Elizabeth Taylor Buck, Christine M Smith, Amanda Lane, Anju Devianee Keetharuth, Tracey Young, Jo Cooke


Patient reported outcome measures (PROMs) are a means of assessing the quality and effectiveness of care from the patient’s perspective. However, the routine use of PROMs in clinical practice can be difficult to implement. New challenges arise at different stages of the implementation process and organisations need to invest time and financial resources into designing an appropriate strategy, information systems, providing technical support and preparing staff. Recovering Quality of Life (ReQoL) is a PROM that was specifically designed to measure mental health service users’ perspectives of recovery and quality of life. It is a co-produced, service user-centred outcome measure tested by over 6000 mental health service users. It is able to detect change across a broad spectrum of mild to severe mental health conditions. In May 2016 a licence to use the ReQoL measures became freely available to the NHS and publicly funded research. At the time of publication, 149 licences had been issued worldwide and eleven official translations had been made available. The team that developed ReQoL has continued to collaborate with mental health trusts across the country to support the implementation of ReQoL. The National Institute for Health Research and Applied Research Collaboration (NIHR ARC YHFootnote1), and the preceding NIHR CLAHRC-YH, along with the ReQoL development team have helped to guide and fund this work. The NIHR CLAHRC-YH supported two national events, the first of which was the launch of ReQoL at the Houses of Parliament in October 2016. More recently in November 2018, over 70 people, from 23 organisations, attended a second event focused on the development of a ReQoL Community of Practice. Community of Practices (CoPs) have existed in sectors such as education and business for over 30 years, operating as networks and support groups for people who share a common set of problems or interests. Members of CoPs maintain and build links with each other for the purposes of social interaction, knowledge sharing, knowledge creation and identity building.

Driscoll, A. et al (2020) Estimating the health loss due to poor engagement with cardiac rehabilitation in Australia

Authors – Andrea Driscoll, Seb Hinde, Laura Bojke, Patrick Doherty


Cardiac rehabilitation (CR) programs are effective in reducing cardiovascular mortality and readmissions. However, most patients are denied the benefits of CR due to low referral rates. Of those patients referred, commencement rates vary from 28.4% to 60%. This paper quantifies the scale of health loss in Australia due to poor engagement with the program, and estimates how much public funding can be justifiably reallocated to address the problem. Economic decision modelling was undertaken to estimate the expected lifetime health loss and costs to Medicare. Key parameters were derived from Australian databases, CR registries and meta-analyses. Population health gains associated with uptake rates of 60%, and 85% were calculated. CR was associated with a 99.9% probability of being cost-effective, even at a cost-effectiveness threshold lower than conventionally applied. Importantly, an average of 0.52 years of life expectancy are lost due to national uptake being below 60% achieved in some best performing programs in Australia, equivalent to 0.28 quality adjusted life years. The analysis indicates that $12.9 million/year could be justifiably reallocated from public funds to achieve a national uptake rate of 60%, while maintaining cost-effectiveness of CR due to the large health gains that would be expected. CR is a cost-effective service for patients with coronary heart disease. In Australia, less than a third of patients commence CR, potentially resulting in avoidable patient harm. Additional investment in CR is vital and should be a national priority as the health gains for patients far outweigh the costs.

Franklin. M. et al (2020) Conducting Value for Money Analyses for Non-randomised Interventional Studies Including Service Evaluations: An Educational Review with Recommendations

Authors - Matthew Franklin, James Lomas, Gerry Richardson


This article provides an educational review covering the consideration of conducting ‘value for money’ analyses as part of non-randomised study designs including service evaluations. These evaluations represent a vehicle for producing evidence such as value for money of a care intervention or service delivery model. Decision makers including charities and local and national governing bodies often rely on evidence from non-randomised data and service evaluations to inform their resource allocation decision-making. However, as randomised data obtained from randomised controlled trials are considered the ‘gold standard’ for assessing causation, the use of this alternative vehicle for producing an evidence base requires careful consideration. We refer to value for money analyses, but reflect on methods associated with economic evaluations as a form of analysis used to inform resource allocation decision-making alongside a finite budget. Not all forms of value for money analysis are considered a full economic evaluation with implications for the information provided to decision makers. The type of value for money analysis to be conducted requires considerations such as the outcome(s) of interest, study design, statistical methods to control for confounding and bias, and how to quantify and describe uncertainty and opportunity costs to decision makers in any resulting value for money estimates. Service evaluations as vehicles for producing evidence present different challenges to analysts than what is commonly associated with research, randomised controlled trials and health technology appraisals, requiring specific study design and analytic considerations. This educational review describes and discusses these considerations, as overlooking them could affect the information provided to decision makers who may make an ‘ill-informed’ decision based on ‘poor’ or ‘inaccurate’ information with long-term implications. We make direct comparisons between randomised controlled trials relative to non-randomised data as vehicles for assessing causation; given ‘gold standard’ randomised controlled trials have limitations. Although we use UK-based decision makers as examples, we reflect on the needs of decision makers internationally for evidence-based decision-making specific to resource allocation. We make recommendations based on the experiences of the authors in the UK, reflecting on the wide variety of methods available, used as documented in the empirical literature. These methods may not have been fully considered relevant to non-randomised study designs and/or service evaluations, but could improve and aid the analysis conducted to inform the relevant value for money decision problem.

Hinde, S. et al (2020) Quantifying the impact of delayed delivery of cardiac rehabilitation on patients’ health

Authors – Seb Hinde, Alexander Harrison, Laura Bojke, Patrick Doherty


Despite its role as an effective intervention to improve the long-term health of patients with cardiovascular disease and existence of national guidelines on timeliness, many health services still fail to offer cardiac rehabilitation in a timely manner after referral. The impact of this failure on patient health and the additional burden on healthcare providers in an English setting is quantified in this article. Two logistic regressions are conducted, using the British Heart Foundation National Audit of Cardiac Rehabilitation dataset, to estimate the impact of delayed cardiac rehabilitation initiation on the level of uptake and completion. The results of these regressions are applied to a decision model to estimate the long-term implications of these factors on patient health and National Health Service expenditure. We demonstrate that the failure of 43.6% of patients in England to start cardiac rehabilitation within the recommended timeframe results in a 15.3% reduction in uptake, and 7.4% in completion. These combine to cause an average lifetime loss of 0.08 years of life expectancy per person. Scaled up to an annual cohort this implies 10,753 patients not taking up cardiac rehabilitation due to the delay, equating to a loss of 3936 years of life expectancy. We estimate that an additional £12.3 million of National Health Service funding could be invested to alleviate the current delay. The current delay in many patients starting cardiac rehabilitation is causing quantifiable and avoidable harm to their long-term health; policy and research must now look at both supply and demand solutions in tackling this issue.

Fowler David, S. et al (2020) Complex programme evaluation of a ‘new care model’ vanguard: a shared commitment to quality improvement in an integrated health and care context

Authors – Sally Fowler Davies, Seb Hinde, Steven Ariss


NHS vanguards, under-pressure to perform, required better contracting and data management arrangements with evaluation teams, to ensure that integrated service outcomes could be reported effectively. This communication reflects the experience of evaluating an NHS vanguard and suggests how academic teams can improve capacity for complex programme evaluation of rapid improvements in integrated services. This should be based on a shared commitment to data collection and management. Also, robust knowledge exchange processes can enable systems change and sustainability. The identifying features of the particular site have been withheld.


Hinde, S. et al (2019) The relevant perspective of evaluations informing local decision makers: an exploration in weight loss services

Authors – Seb Hinde, Louise Horsfield, Laura Bojke, Gerry Richardson


Since 2013, obesity services in the UK National Health Service (NHS) have focused on a tiered structure, with tiers 3 (specialist weight management services) and 4 (primarily bariatric surgery) commissioned by Clinical Commissioning Groups (CCGs) and widely reported as cost effective and recommended by national guidelines. However, CCGs have been reluctant to fully conform to the guidance. We explore how the different evaluative perspective of those generating evidence from local decision makers has contributed to this failure of the CCGs to provide services considered cost effective. We explore four elements where the conventional economic evaluation framework, as applied by the National Institute for Health and Care Excellence (NICE), differ from the reality faced by local decision makers: the cost-effectiveness threshold, the implications of decision uncertainty and budgetary excess, the valuation of future costs and outcomes, and the scope of included costs. We argue that the failure of the conventional framework to reflect the reality faced by local decision makers is rendering much of the existing literature and guidance inappropriate to the key commissioners. Our analysis demonstrates that it is not reasonable to assume that the framework of economic evaluation used to inform national guidance applies to local decision makers, such as in the commissioning of weight loss services. This failure is likely to apply to the majority of cases where evidence is generated to inform national decision makers but commissioning is at a local level.

Hinde, S. et al (2019) Understanding and Addressing the Challenges of Conducting Quantitative Evaluation at a local Level: a worked example of the available approaches

Authors - Seb Hinde, Laura Bojke, Gerry Richardson


In the context of tightening fiscal budgets and increased commissioning responsibility, local decision-makers across the UK healthcare sector have found themselves in charge of the implementation and evaluation of a greater range of healthcare interventions and services. However, there is often little experience, guidance or funding available at a local level to ensure robust evaluations are conducted. In this paper, we evaluate the possible scenarios that could occur when seeking to conduct a quantitative evaluation of a new intervention, specifically with regards to the availability of evidence. We outline the full set of possible data scenarios that could occur if the decision-maker seeks to explore the impact of the launch of a new intervention on some relevant quantifiable outcomes. In each case we consider the implicit assumptions associated with conducting an evaluation, exploring possible situations where such scenarios may occur. We go on to apply the scenarios to a simulated dataset to explore how each scenario can result in different conclusions as to the effectiveness of the new intervention. We demonstrate that, across the full set of scenarios, differences in the scale of the estimated effectiveness of a new intervention and even the direction of effect are possible given different data availability and analytical approaches. When conducting quantitative evaluations of new interventions, the availability of data on the outcome of interest and the analytical approach can have profound effects on the conclusions of the evaluation. Although it will not always be possible to obtain a complete set of data and conduct extensive analysis, it is vital to understand the implications of the data used and consider the implicit assumptions made through its use.

Contact the team

Amanda Lane, Theme Administrator -