Skip to main content

Statisticians call for rigour and transparency in the evaluation of diagnostic tests

Recommendations – developed by a working group of statisticians – on reframing the evaluation of in vitro diagnostic tests have been published today by the Royal Statistical Society in its Series A journal. The group was co-chaired by Professor Jon Deeks, at BHP founder-member the University of Birmingham and former RSS President, Professor Deborah Ashby, at Imperial College London.

The report, which will be submitted to the UK Covid-19 Inquiry, is intended to help prevent future scenarios in which IVDs are marketed widely, but later attract serious concerns about the standards applied to their evaluation. Its co-authors include statisticians from the Universities of Oxford, Cambridge, Edinburgh, Birmingham and the London School of Hygiene and Tropical Medicine.

The research was prompted by concerns about the standards applied to the evaluation of diagnostic tests during the Covid-19 pandemic – particularly lateral flow tests – however the recommendations cover all new tests, especially those designed to detect infectious diseases.

It is published today in the RSS’s Series A journal and also presented at the Evidence Based Early Diagnosis conference at St Andrews.

The RSS Working Group on Diagnostic Tests set out 22 recommendations, designed to ensure that in vitro diagnostic (IVD) tests – which typically test samples of fluids such as blood, urine or saliva – are statistically robust and fit for purpose. The RSS Working Group identified Study-Design matters (10 recommendations); Regulation matters (6 recommendations); Transparency matters (6 recommendations).

Jon Deeks, Professor of Biostatistics at the University of Birmingham, said: “The Covid-19 pandemic provided a microcosmic insight into inadequacies in current processes to evaluate and regulate diagnostic tests. It’s important that we learn from these failures and establish robust processes that can be applied broadly across diagnostic tests.”

The report covers three areas of diagnostic testing: study-design of evaluations; regulation of tests; and transparency of test evaluations.

Key recommendations included:

  • Evaluation needs to take into account each specific intended use of the test, including the person being tested, the target condition and even the facilities where the testing will be done. Field or clinical evaluation studies should be carried out for each intended use.
  • Direct comparison of alternative IVDs and testing strategies should be available to inform clinical and public health decision-making.
  • The Medicines and Healthcare products Regulatory Agency (MHRA) should collaborate with independent experts to revise the national licensing process for IVDs. This will ensure public safety is protected. Protocols and reports for test evaluations should be publicly available to ensure transparency in all planning and decision-making.

The publication of the report is relevant for the opening of the ‘Test, Trace and Isolate’ module of the UK Covid-19 Inquiry. It also coincides with the MHRA’s recently-launched consultation on improved safety for high-risk diagnostic devices.

Professor Sheila Bird at the MRC Biostatistics Unit at the University of Cambridge, said: “Past Royal Statistical Society Working Party reports on matters which affect the public health have had enduring impact. Official Statistics – Counting with Confidence led to the UK Statistic Act of 2007; Statistics and Statisticians in Drug Regulation led to the appointment of professional statisticians by the UK, and later, European drug regulator; Statistical Issues in First-in-Man Studies led to safety-enhanced study-designs with open protocols. I hope that this month’s consultation by MHRA is indicative that Diagnostic Tests is making its mark already.”

Dr Andrew Garrett, President of the Royal Statistical Society, said: “The report provides a thorough evaluation of both diagnostic tests and diagnostic testing. It addresses how to develop, regulate, and use diagnostic tests in the future – a subject that is of increasing importance to individual and public health.”

You might also be interested in:

Launch of new international medical code for presymptomatic type 1 diabetes

Researchers at BHP founder-member the University of Birmingham have partnered with NHS England to produce a diagnostic code tailored for individuals in the early phases of type 1 diabetes, enhancing patient prospects for timely healthcare and access to cutting-edge treatments.

Today marks the introduction for a new SNOMED CT code specifically for presymptomatic type 1 diabetes, which will be integrated into the standardised and multilingual set of clinical healthcare terminology. SNOMED codes are crucial in electronic health records, being used to identify a person’s underlying medical conditions. This system acts as the most precise and extensive list in clinical health terminology globally.

Type 1 diabetes progresses gradually through three stages, with the initial two stages termed presymptomatic type 1 diabetes. Individuals in this phase exhibit biological markers, or autoantibodies, indicating the onset of the immune attack that targets insulin-producing beta cells. Given the absence of symptoms, detection relies heavily on screening initiatives such as the ELSA study, a trial led by Professor Parth Narendran at the University of Birmingham,  screening children for type 1 diabetes. Screening initiatives such as these will allow for early identification.

Lauren Quinn, who co-leads the ELSA study and assisted in the development of the of the SNOMED code, commented: “The introduction of this SNOMED code facilitates clinical care and follow-up for individuals with presymptomatic type 1 diabetes. It also allows researchers to identify people who could benefit from novel therapies to delay the onset of type 1 diabetes and recruit them to clinical trials of immunotherapies.”

“This will transform type 1 diabetes research by fast-tracking recruitment, unravelling how the condition develops and progresses, and bringing us closer to licensed disease-modifying treatments in type 1 diabetes.”

Dr. David Shukla, a GP and Clinical Research Fellow involved in code development, highlighted its practical implications: “The inclusion of a code for the diagnosis of presymptomatic type 1 diabetes will highlight to healthcare professionals involved in their care the individuals who are at high risk of developing type 1 diabetes. This will help ensure that when these people progress and develop symptomatic type 1 diabetes, it will be picked up and treated at a much earlier stage.”

“This reduces the risk of them presenting or being diagnosed late and developing diabetic ketoacidosis, an emergency complication of type 1 diabetes that can be fatal. This timely pick up and initiation of prompt treatment will lead to substantial improvements in their diabetes and future care.”

Hilary Nathan, Director of Policy and Communications at JDRF UK, added: “This recognition of presymptomatic type 1 diabetes with a SNOMED code is a crucial step towards the implementation of population screening programmes for early detection of type 1 diabetes. Early detection leads to short and long-term health benefits, improved quality of life and cost savings for healthcare providers.”

“The new code will unlock better monitoring, follow-up and education for people in the earliest stages of type 1. It will also help facilitate recruitment into clinical trials of emerging treatments, enabling people developing type 1 diabetes to access therapies that have the potential to claw back valuable time free from the burdens of type 1 diabetes management.”

The code for type 1 diabetes in SNOMED is ‘Diabetes mellitus type 1 – 46635009’. The new code presymptomatic type 1 diabetes, known as ‘‘Presymptomatic diabetes mellitus type 1 – 1290118005′, has now been introduced for inclusion in individuals’ electronic health records. 

These codes are now part of the ‘Health conditions’ category in the NHS app, allowing individuals and their families to access them as well. 

You might also be interested in:

New Inflammatory Bowel Disease testing protocol could speed up diagnosis

Patients with suspected inflammatory bowel disease (IBD) could benefit from improved testing protocols that could reduce the need (and lengthy wait) for potentially unnecessary colonoscopies, a new study has found.

In a paper published in Frontline Gastroenterology, researchers from the National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre (BRC) tested a new protocol to improve IBD diagnosis combining clinical history with multiple home stool tests.

In the two-year study involving 767 participants, patients were triaged and had repeated faecal calprotectin (FCP) tests. The research team found that the use of serial FCP tests were able to strongly predict possible IBD as well as Crohn’s Disease and Ulcerative Colitis, and observed that a second FCP test was a strong indicator of a potential need for further investigation including colonoscopy; although the researchers observed that only 20% of patients had two samples submitted prior to referral to secondary care.

Dr Peter Rimmer from the NIHR Birmingham Biomedical Research Centre and corresponding author of the study said: “Patients who experience symptoms associated with inflammatory bowel diseases often have a long wait until getting a diagnosis, and current testing is under immense strain.

“Using a comprehensive 13-point symptom checker and multiple FCP tests, we have been able to identify much more accurately patients who had IBD and other diseases. The rollout of this protocol could reduce the time taken to get a diagnosis and start treatment for IBDs as much more of the screening and testing can be done through primary care. The sensitivity of multiple FCP tests can be used to flag those patients who urgently need referral into secondary care.”

Dr Rachel Cooney, Consultant Gastroenterologist at University Hospitals Birmingham NHS Foundation Trust, researcher at the NIHR Birmingham BRC and co-author of the study, added: “In its simplest form, this study may help improve referral triage for IBD patients.

“But as we plan new care pathways, it could open up new exciting possibilities: with the growing availability of home FCP testing, these tests’ results combined with simple symptom questionnaires could feed into algorithms that allow patients to self-refer to secondary care services, reducing strain on primary care.

“This is something we’re going to explore in a large follow-up study we’re currently initiating.”

The NIHR Birmingham Biomedical Research Centre is hosted by BHP founder-member University Hospitals Birmingham NHS Foundation Trust in partnership with fellow BHP founder-member the University of Birmingham. Its roster of partner organisations also includes BHP members Birmingham Women’s and Children’s NHS Foundation Trust; Sandwell and West Birmingham NHS Trust; and Aston University. 

You might also be interested in:

Objective biomarker test could predict heart disease risk for patients with common arrhythmia

Patients with atrial fibrillation (AF) – a common heart arrhythmia – could have one blood test to assess their risk of cardiovascular events in the following five years, new research has found.

Published in Cardiovascular Research and presented at the Frontiers in CardioVascular BIology Congress 2024 conference in Amsterdam, the research suggests that a blood-based biomolecule test alone could assess the risk of having a cardiovascular event in the next five years.

The study of 1,586 AF patients found that a cluster of high-risk patients who recorded high levels of 13 biomolecules had five times more cardiovascular events than those in the low-risk cluster.

Professor Larissa Fabritz, from the Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg Eppendorf and an Honorary Professor at BHP founder-member the University of Birmingham said: “These validated findings show that one blood test could be used to help predict the risk of cardiovascular events for patients with atrial fibrillation, helping to differentiate healthcare where it’s most needed. Through a further validation study carried out in Birmingham, we are confident that the blood test can give a useful understanding of those in greatest need of interventions to avoid strokes, acute heart failure and death.”

The international team developed a profile of 13 specific biomarkers that were used to differentiate risk in atrial fibrillation. Using samples from AF patients, they analysed likely target biomarkers and through the trial and validation in the Birmingham BBC-AF registry found that a combination of elevated biomarkers corresponded with risk variation in patients.

The findings are taken from a subset of the EAST – AFNET 4 (Early Treatment of Atrial Fibrillation for Stroke Prevention) trial which demonstrated that early rhythm control – with antiarrhythmic drugs or atrial fibrillation ablation – delivered within one year after AF diagnosis improves outcomes in 2,789 patients with early AF and cardiovascular risk factors compared to usual care (UC) over a 5-year follow-up time.

You might also be interested in:

Birmingham scientists win funding to develop ‘lollipops’ for mouth cancer diagnosis

A ‘lollipop’ that can diagnose mouth cancer early could become a reality, thanks to a pioneering project funded by Cancer Research UK and the Engineering and Physical Sciences Research Council (EPSRC).

Scientists at BHP founder-member the University of Birmingham have been awarded £350,000 over the next three years to develop a prototype flavoured ‘lollipop’ from a material called a smart hydrogel.

Smart hydrogels – previously developed by the University’s Dr Ruchi Gupta and her team – work a bit like a fishing net: they absorb large quantities of water while ‘catching’ larger molecules, such as proteins. The ‘net’ can then be cut open to release the larger molecules for analysis in the lab. The idea is that patients suck on the lollipop, transferring a saliva sample into the hydrogel. Scientists can then release the ‘caught’ proteins by blasting the hydrogel with UV light and then analyse the liquid for saliva proteins which indicate the early stages of mouth cancer.

Around 12,400 people are diagnosed with cancers of the head and neck in the UK every year*. Currently, diagnosing mouth cancer can involve putting a flexible camera on the end of a tube through the nose or mouth and taking a biopsy for testing. This procedure is invasive, time-consuming and requires an endoscopist.

Mum of five, Rachel Parsons, needed a biopsy after being referred to Coventry University Hospital with a lump on her cheek in 2008. She admits she was unprepared for the procedure which, in her case, turned out to be painful.

“I had no idea what a biopsy really was,” said Rachel, from Coventry. “I had the kind of injection you get at the dentists and, when it wore off, it was really sore because I’d needed stitches.”

That was just the beginning of a 12-month nightmare for Rachel who ended up needing a nine-and-a-half-hour operation to remove a cancerous tumour from her cheek and replace the skin with tissue and veins from her forearm.

“The thought of putting a lollipop round your mouth instead of having a biopsy in the first instance is amazing,” said Rachel who has spent years as a patient ambassador, campaigning for more awareness of mouth cancer. “I wish something like that had existed when I was diagnosed.”

Dr Ruchi Gupta, Associate Professor of Biosensors at the University of Birmingham, said she was thrilled to receive funding to begin the next phase of the project: “Smart hydrogels have really exciting potential for diagnosing mouth cancer,” she said. “They can be easily moulded into shapes as a solid to ‘catch’ proteins in saliva.

“We’re really excited to start the next phase of this project. We’re hoping that we can be the first to make a device which is much kinder for diagnosing mouth cancer for patients and easier for GPs to use.”

Rachel, who still has numbness around her face and can’t open her mouth wide enough to eat a burger, added: “I’m so grateful for the research and treatment that saved my life. Things have improved immensely since then but what’s happening now could be absolutely brilliant for people diagnosed in future.”

Executive Director of Research and Innovation at Cancer Research UK, Dr Iain Foulkes, said: “Biopsies and nasoendoscopy are the gold standard for diagnosing mouth cancer, but it requires great skill to carry out and can feel unpleasant for patients. We want an accurate, faster and kinder alternative test which can help us diagnose cases of mouth cancer sooner.

“This project is an exciting first step towards an entirely new way to identify mouth cancers earlier. Research like this is guiding us towards a future where people can live longer, better lives, free from the fear of cancer.”

You might also be interested in:

Revolutionising diagnosis and management of cartilage tumours

The musculoskeletal radiology department at BHP member the Royal Orthopaedic Hospital (ROH) has developed a new website designed to enhance the diagnosis and management of cartilage tumours.

bactip.co.uk is a platform that equips healthcare professionals with the tools and knowledge to navigate the complexities of central cartilage tumours. By offering drawings and real case examples, it serves as a valuable resource for interpreting musculoskeletal radiology imaging findings related to these tumours, ensuring their accurate and consistent reporting. Developed collaboratively by experts in the field, bactip.co.uk offers an approach for assessing, diagnosing and monitoring these lesions.

A standout feature of bactip.co.uk is its integrated calculator, which streamlines the grading process based on tumour characteristics, like size and aggressiveness. This innovative tool aims to simplify decision making processes for healthcare professionals dealing with cartilage tumours. This advanced tool also standardises the reporting of central cartilage tumours, reducing subjective differences and improving patient care.

bactip.co.uk enables radiologists and clinicians to make informed decisions by offering an imaging follow up plan. Whether it involves suggesting a referral to an oncology specialist or safely discharging a patient from surveillance monitoring, the protocol provides a detailed framework customised for each unique case.

As a leading authority in orthopaedic excellence, the ROH musculoskeletal radiology department is proud to share its expertise through bactip.co.uk, an open-source free resource.

Dr A. Mark Davies, consultant radiologist at the ROH commented: “This initiative reflects our commitment to openness making sure that healthcare professionals worldwide can access our cutting-edge knowledge and best practices without any barriers. Our dedication to spreading knowledge and best practices aligns with our shared goal of enhancing patient outcomes on a global scale.”

You might also be interested in: