Respiratory Medicine CME
Volume 2, Issue 2 , Pages 63-67, 2009

Algorithms for assessing the probability of an Adverse Drug Reaction

Department of Respiratory Medicine, Dudley Group of Hospitals, Pensnett Road, Dudley, West Midlands, DY1 2HQ, UK

Article Outline

Abstract 

Adverse Drug Reactions (ADRs) are common and are associated with significant risk of morbidity, mortality and admission to hospital. Deciding if a clinical event is an Adverse Drug Reaction, or not, can be difficult. The decision is often based on clinical judgment alone, yet studies have shown that decisions based on clinical judgment often vary greatly between raters.

Therefore a number of decision aids or Algorithms have been developed to try and improve this variability. Studies have shown that the use of algorithms does improve the between and within rater agreement significantly, and gives a semi-quantitative measure of the likelihood of causality. There are variations between these algorithms but none of them can in themselves prove or disprove causality. These algorithms, their benefits and their problems will be discussed in this article.

Keywords: Adverse Drug Reactions, Algorithms, Causality, Rechallenge

 

Educational aims

•To discuss the epidemiology of Adverse Drug Reactions

•To illustrate the merits of Algorithms in decisions on causality of Adverse Drug Reactions

•To illustrate the problems of algorithms in decisions on causality of Adverse Drug Reactions

•To show the range of ADR algorithms that have been developed

Back to Article Outline

Introduction 

Adverse Drug Reactions are, according to the World Health Authority definition, “any response to a drug which is noxious and unintended and which occurs at doses normally used in man for prophylaxis, diagnosis or therapy of disease or for the modification of physiological function”.1 As such ADRs encompass side effects which are “a dose related and predictable reaction to a drug” and drug allergies.

ADRs account for 2–6% of all hospital admissions in the UK,2 these admissions are almost always medical rather than surgical and only a minority (9.7%) are due to non-prescribable medicines.3 They are a serious problem in terms of morbidity, mortality and the cost of patient care. In the USA it has been estimated that up to 20% of hospitalized patients suffer at least one ADR during an admission. Also in the United States medication related deaths (excluding illicit use) account for 120,000 deaths a year,4 another study suggests that about three of every 1000 hospital admissions die as a result of an ADR.5 The economic impact of ADRs is massive; they increase hospital stays by almost two days on average, with its intending costs. The total cost of ADRs in the USA has been estimated to exceed the cost of all diabetes treatment.

Back to Article Outline

Recognition of ADRs 

The ADRs produced by a certain new drug are often recognized when the medication is undergoing its phase three randomized controlled trials. Both in the USA and in the UK there is post marketing surveillance of ADRs. In the UK this involves reporting suspected ADRs to the Commission on Human Medicine using the yellow card system. In this system new or intensively monitored medicines should have all suspected ADRs reported and other medicines should have any suspected serious ADR reported. In spite of these mechanisms ADRs are vastly under reported6 and initial reports of adverse reactions to drugs have taken up to seven years for trends to begin to appear in the literature.

Under reporting of ADRs is likely to be due to a number of reasons. Reporting is not mandatory to clinicians in the UK and so is likely to be forgotten about amongst the many other work pressures. A clinician may have problems recognizing the scenario as an ADR, because of the background symptoms of the patient's original illness. Clinicians might also be wary of reporting an ADR, because of worries of inducing a complaint, even in this no blame culture NHS. It should be pointed out that the yellow card clearly states you do not need to be sure if it is or is not an ADR before you report it.

In recognizing an ADR there are a number of important factors. One is identifying those individuals in whom ADRs are most likely to occur. This includes the aged and the premature, those with liver and renal dysfunction, those on polypharmacy and patients with certain individual conditions, such as Human Immunodeficiency Virus infection (HIV).

Back to Article Outline

Assessment of causality 

It is often difficult to decide if an adverse clinical event is an ADR or due to deterioration in the primary condition. Furthermore, if it is an ADR, which medicine caused it, as many patients are on multiple new medications when ill, particularly if admitted to hospital.

In spite of these problems, the decision that a particular drug caused an ADR is usually based on clinical judgment alone. Studies have shown that there is a lot of variation in between rater and within rater decisions on causality of ADRs; this applies both to pharmacologists and physicians.7, 8 In one study two physicians and four pharmacists were asked to decide about 63 possible cases, they showed a between rater agreement of 38% to 63%.9 Another previously published study showed a 50% agreement between two raters with a Kw value of 0.3.10 These problems with using individual clinical decision making in allotting causality for a change in clinical condition to a medication led to the development of a number of decision aids.

Back to Article Outline

The benefits of algorithms 

Decision aids or algorithms were developed in the 70s and 80s, they tend to consist of yes/no questions, which can be used to categorise causality. For instance the Jones' algorithm11 will categorise the causality into remote, possible, probable or highly probable. Other algorithms consist of yes/no questions, the answers to which correspond to a set score. So by answering all the questions in the algorithm you end up with a total score, this numerical score then corresponds to a given category of causality.

There are benefits in the use of such algorithms; including standardization of methods. Algorithms being structured systems specifically designed for the identification of an ADR, should theoretically make a more objective decision on causality. As such algorithms should have a better between and within rater agreement than clinical judgment. Indeed this has been shown to be the case.9 In this study the between rater agreement of a panel of experts, using clinical judgment ranged from 41% to 57% (kappa=0.21–0.37, R(est)=0.49). When the same individuals used the Naranjo Algorithm it rose to between 83% and 92% (kappa=0.69–0.86, r=0.92), this rise being statistically significant. The within rater agreement was also high using the algorithm, being between 80% and 97% (kappa=0.64–0.95, r=0.95).

Back to Article Outline

Comparing the algorithms 

A number of algorithms or decision aids have been published including the Jones' algorithm,11 the Naranjo algorithm,9 the Yale algorithm,12 the Karch algorithm,13 the Begaud algorithm,14 the ADRAC,15 the WHO-UMC16 and a newer quantitative approach algorithm.17 Each of these algorithms has similarities and differences. An example of one of the more commonly used algorithms; the Naranjo algorithm (Fig. 1) is shown below. The consistency of three of these algorithms was directly compared in a study in 1986.18 In this study 28 ADRs were assessed using the Jones', the Yale and the Naranjo algorithms. There was 67% agreement between the Yale and the Naranjo algorithm (Kw=0.43), similarly there was 67% agreement (Kw=0.48) between the Yale and the Jones' algorithm. Agreement between the Naranjo and the Jones' algorithms was 64% but the Kw value was only 0.28.These levels of agreement are better than those that have previously been reported when two raters have compared the same ADRs using clinical judgment.

They concluded that the Naranjo algorithm compared well with the Yale in scoring ADRs but had the advantage of being less time consuming. The Yale algorithm containing 57 questions compared to the 10 questions in the Naranjo algorithm. They were less supportive of the use of the Jones' algorithm in view of its lesser agreement with the Naranjo algorithm. To reduce the ambiguity in the assessment of potential ADRs these algorithm have been introduced at pharmacovigilence centres in many countries.

Back to Article Outline

Problems with algorithms 

Although algorithms have better reproducibility than clinical judgment in rating ADRs, clinical judgment with its low inter- and intra-rater agreement still plays a big part in the identification and rating of potential ADRs by an algorithm. This is because the answers to some of the questions in the algorithm may be affected by clinical judgment. More importantly the first step in ADR identification depends on a clinical judgment, i.e. the decision that this might be an ADR and so deserves further assessment using an algorithm.

Further problems include that the questions in an algorithm are often weighted, these weights are arbitrarily assigned based on their perceived importance and vary between algorithms. This qualitative assigning of weights means that algorithms are unable to truly determine the probability of the ADR causality.

Even though algorithms have been shown to be more reproducible than clinical judgment alone, the validity of the measure must also be considered. The fact that the algorithms agree well with each other does not mean that they are right. Studies have looked at the validity of algorithms, by comparing the category of causality that they produce to the decision on causality decided by a group of experts in the field. This is not a true test of the validity of an assessment system, as this testing cannot work as for as the majority of ADRs, no true “Gold Standard” exists.

Further problems include the idea that most include questions on dechallenge/rechallenge, and the rechallenge often does not occur in the “real world” of clinical practice. This might not occur for a number of reasons, for many serious ADRs rechallenge might be considered unethical, since it may pose a considerable risk to the patient. Also for many lesser potential ADRs using a different drug rather than undergoing the rechallenge may well be deemed an easier and simpler option by the clinician. Even if the clinician is willing to consider rechallenge to strengthen the probability of causality for an ADR, the patients themselves will often refuse such a rechallenge. Without a rechallenge it is difficult with most of these algorithms for causality to be graded more than “possible”.

Algorithms depend on a YES/NO answer to individual questions, this is not always easy, sometimes a “maybe” might be more appropriate. So in a way algorithms may simply replace honesty with pragmatism. Lastly there are a great number of ADRs in a number of different body systems, so a single standardized assessment tool may not be ideal for such a diversity of possible presentations. More recent work has tried to develop assessment schemes for individual problems e.g. liver disease, interstitial lung disease, and renal failure.19

Back to Article Outline

Summary 

In summary algorithms are useful in assessing causality in possible ADRs, as they decrease the disagreement between assessors and can classify uncertainty in a semi-quantitative way. They are often used by journals and national pharmacovigilence organizations to mark individual case reports. They improve the scientific basis of causality assessment and are useful in education about causality assessment. However they cannot prove or disprove causality, nor give an accurate quantitative measurement of the likelihood of a relationship.

Back to Article Outline

CME questions 

This article has been accredited for CME learning by the European Board of Accreditation in Pneumology (EBAP). You can receive one CME credit by successfully answering these questions online.

a)Visit the journal CME site at http://www.resmedcme.com

b)Complete the answers online, and receive your final score upon completion of the test.

c)Should you successfully complete the test, you may download your accreditation certificate (subject to an administrative charge).

Back to Article Outline

Examples 

A 27 year old man presented to hospital with pleuritic chest pain, shortness of breath and cough. His chest X-ray showed right basal consolidation, he had a raised CRP and white cell count. His medication prior to this illness was Olanzapine 15mg once per day, which he has taking for a prior diagnosis of schizophrenia, he had been on this medication for 8 months in all.

He was originally treated as a case of community acquired pneumonia but on antibiotics his symptoms deteriorated so he went on to have a C.T. Pulmonary Angiogram, this showed multiple Pulmonary Emboli. Subsequent investigation showed that he also had nephrotic syndrome with proteinuria of 2.84g/litre and a serum albumen of 21g/litre. a Kidney biopsy showed he had membranous glomerulonephritis. Later 24hour urine collection showed a proteinuria of over 8g/litre.

He was started on diuretics, enalopril and 20mg of prednisolone, this failed to significanlty improve his proteinuria. The question of if his nephrotic syndrome could be due to his Olanzapine was then raised and his Olanzapine was stopped.

Question 1. 

Using the Naranjo Algorith how likley is it that the Olanzapine has caused the nephrotic syndrome?

A 29 year old man with severe uncontrolled asthma in spite of treatment with Uniphyline, montelukast, inhaled corticosteroids and a long acting beta agonist combination, nebulised bronchodilators and multiple coarses of systemic steroinds was found to have a raised Ig E level of 343IU/ml. As he had had several admissions to hospital with his asthma, he was assessed for treatment with Omalizumab (an anti Ig E antibody) as per NICE Guidelines. He was started on Omalizumab 300mg sub cutaneously each fortnight. This resulted in improvements in his lung function, symptoms and quality of life measurements.

Six weeks after starting the Omalizumab he presented to hospital with crushing central chest pain, which came on after a weekend absailing, a sport he had never previously been able to partake in.

Although his e.c.g. was normal, his troponin was raised at 0.23, a non ST elevation myocardial infarct was diagnosed. He went on to have coronary angiography with stenting of his LAD. He made a rapid recovery. At the patients bequest his treatment with Omalizumab was continued. Eighteen months latet he continues to objectively benefit from the Omalizumab and he has had no further symptoms of Ischaemic Heart Disease.

Question 2. 

Using the Naranjo Algorithm, how likely is it that the Omalizumab caused the myocardial infarction?

A 69 year old man, a welder by profession, was referred to the respiratory clinic for assessment of his COPD. His past medical history included Abdominal Aorta Repair, Coronary artery by-pass operation for Ischemic Heart Disease, Osteoarthritis of both knees and Dyslipidemia. His medication included Bisoprolol, salbutamol via a metred dose inhaler as required, Perindopril, Aspirin, Simvastatin and Omeprazole.

On assessment in the clinic he had dyspnoea for 10 years, with a decreased exercise tolerance and he had a number of exacerbations in the last year. He was an ex-smoker with a smoking history of 80 pack years. Spirometry performed in clinic revealed a forced expiratory volume (FEV1) of 0.97 liters increasing to 1.1 liters after 400mcg of Salbutamol. On Spirometry he had moderate COPD (as per NICE Guidelines) with FEV1 of 42% of predicted and an obstructive pattern with a FEV1/FVC ratio of 52%.

He was started on a maximal inhaled treatment of Salbutamol (2 puffs 4 times a day), Tiotropium -18mcg once a day, Seretide 500 Accuhaler twice a day (a combination of fluticasone 500mcg and Salmeterol 50 mcg). At follow up, he complained that he had had severe pain in both of his heels over the area of Achilles Tendons that started two weeks after starting the Seretide. Therefore he stopped the statin and the inhaled steroid (Seretide) and the symptoms resolved completely. On reintroduction of inhaled steroid the symptoms of tendonitis recurred again after two weeks. On examination there was some swelling and crepitus felt over the Achilles Tendons, after which, he was advised to stop Seretide for the second time, this again resulted in complete resolution of his symptoms.

Evidence of focal thickening and increased vascularity in the area of Achilles Tendons was observed on performing an ultrasound of his tendons; radiologically confirming the clinical diagnosis of Achilles tendonitis.

Question 3. 

Using the Naranjo Algorithm how likely is it that the inhaled corticosteroid caused the tendinitis?

Answers 

Question 1: Olanzapine and the Nephrotic Syndrome

Total score +2 so a possible ADR

No previous reports of Olanzapine and the nephrotic syndrome scores zero, but it did occur after the Olanzapine so scores+2.

It is not clear if it improves on withdrawal as this is only occuring now and no rechallenge has occurred.

No Olanzapine levels reported and no increased doses given, all score zero.

There are other causes of membranous glomerulonephritis minus one point.

there is objective evidence scores plus one point.

Question 2: Omalizumab and Myocardial Infarction
Total score zero, therefore doubtful ADR.

Scoring similar to before but, no recurrence on restarting the Omalizumab post MI scores minus 2, making total score zero.

Question 3: Inhaled Corticosteroids and Achilles Tenosynovitis
Total score probably an eight, therefore probable ADR.

There is subjectivity to the scoring in all cases but this case clearly scores higher.

Are there previous reports, not to inhaled but to other forms of steroids is this a zero or a+one.

It occured after the drug was given scores plus two.

It went when withdrawn scores plus one.

It came back on rechallenge scores plus two. He had a similar reaction first time he had the drug and there was objective evidence of the ADR

The total score is probably arguable but is between 6 and 8.

Back to Article Outline

Educational questions 

Answer true or false to the following statements: 


1.With regards to Adverse Drug Reactions (ADRs)
a)These are defined as a dose related and predictable reaction to a drug

b)These account for 2-6% of all hospital admissions in the UK

c)The majority of admissions due to ADRs are surgical, particularly peptic ulceration and pancreatitis

d)The total cost of ADRs in the USA has been estimated to exceed the costs of cardiac treatment

e)When they occur in hospital they tend to increase the length of stay by an average of almost two days


2.Recognition of Adverse Drug Reactions
a)Occurs predominantly during the phase three randomized controlled trials

b)Reporting of an ADR is mandatory in the UK

c)Occurs more commonly in the aged and premature

d)Is always easy

e)Reporting of an ADR should occur through the yellow card system in the USA


3.With regard to dechallenge/rechallenge
a)This should always occur, so as to prove causality

b)Is central to most causality algorithms

c)Is easy to perform

d)Should be performed with the patient blinded to what is occurring

e)Proves causality in an ADR


4.The problems with Algorithms in assessing ADRs include.
a)Better reproducibility

b)The removal of clinical judgment from the assessment of causality

c)There is no true gold standard

d)The questions in the algorithms do not always reflect real clinical practice

e)A standardized assessment tool may not be ideal for all disease processes


5.The benefits of algorithms in assessing causality include
a)Their ease of use

b)Increased between rater agreement

c)Education about causality assessment

d)Proof of causality

e)Classifying uncertainty semi-quantitatively


Back to Article Outline

References 

  1. Requirements for adverse drug monitoring. Geneva, Switzerland: World Health Organisation; 1975;
  2. Pirmohamed M, et al. Adverse drug reactions as a cause of admission to hospital: prospective analysis of 18820 patients. BMJ. 2004;329:15–19
  3. Proceedings of the British Pharmacological Society 75th Anniversary Meeting clinical pharmacology section (December 2006). Br J Pharmacol. 2007;63(6):776
  4. Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse reactions in hospitalized patients; a Meta analysis of prospective studies. J Am Med Assoc. 1998;279:1200–1205
  5. Manasse HR. Medication use in an imperfect world: drug misadventuring as an issue of public policy, part 1. Am J Hosp Pharm. 1989;46:929–944
  6. Laepe LL, Brennan TA, Laird N. The nature of adverse events in hospitalized patients. Results of the Harvard medical practice study II. New Engl. J Med. 1991;324:377–384
  7. Blanc S, Leuenberger P, Berger JP, Brooke M, Schelling JL. Judgements of trained observers on adverse drug reactions. Clin Pharmacol Ther. 1979;25:493–498
  8. Karsh FE, Smith CL, Kerzner B, Mazzullo JM, Weintraub M, Lasanga L. Adverse drug reactions. A matter of opinion. Clin Pharmacol Ther. 1976;19:489–492
  9. Naranjo CA, Busto U, Sellars EM, Sandor P, Ruiz I, Roberts EA, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981;30:239–245
  10. Naranjo CA, Busto U, Abel JG, et al. Empiric delineation of the probability spectrum of adverse drug reactions. Clin Pharmacol Ther. 1981;29:267–268
  11. Jones JK. Adverse drug reactions in the community health setting; approaches to recognizing, counseling and reporting. Fam Community Health. 1982;5(2):58–67
  12. Kramer MS, Hutchinson TA. The Yale algorithm. Special workshop-clinical. Drug Inf J. 1984;18:283–291
  13. Karch FE, Lasagna L. Towards the operational identification of adverse drug reaction. Clin Pharmacol Ther. 1977;21:247–254
  14. Begaud B, Evreux JC, Jouglard J, Lagier G. Imputation of the unexpected or toxic effects of drugs. Actualisation of the methods used in France. Therapie. 1985;40:115–118
  15. Mashford MI. The Australian method of drug event assessment. Drug Inf J. 1984;18:271–273
  16. WHO-UMC causality assessment system. Available from: <http://www.who-umc.org/pdfs/Causality.pdf>.
  17. Koh Y, Chu WY, Shu CL. A quantitative approach using genetic algorithm in designing a probability scoring system of an adverse drug reaction assessment system. Int J Med Inf. 2008;77:421–430
  18. Michel DJ, Knodel LC. Comparison of three algorithms used to evaluate adverse drug reactions. Am J Hosp Pharm. 1986;43:1709–1714
  19. Meyboom RH, Hekster YA, Egberts AC, Gribnau FW, Edwards IR. Causal or casual? The role of causality assessment in pharmacovigilence. Drug Saf. 1997;17(6):374–389

PII: S1755-0017(09)00005-0

doi:10.1016/j.rmedc.2009.01.004

Respiratory Medicine CME
Volume 2, Issue 2 , Pages 63-67, 2009

Article Tools

* Image Usage & Resolution