Finding and evaluating Complementary Medicine evidence
This module will provide an introduction to scientific evidence and how to find research-based studies about complementary medicine. Module Four will specifically look at how this information is used in decision making.
The amount of information that exists about complementary medicine can be overwhelming and can come from many different sources, including:
Media (television, radio, print and online)
It is important to consider the reliability of any information about complementary medicine you may have. How confident you are in applying that information to your situation? To answer this question, you need to know more about evidence.
What is evidence?
Evidence is a specific type of information used to support a conclusion.7
In the case of health care, evidence is obtained from research studies or expert opinion to determine if a treatment is effective.
Different sources of information have varying strengths and weaknesses and provide different levels of evidence.
Evidence is what is known about something that has been confirmed by scientific research. To make an evidence-informed choice about a complementary medicine therapy you are interested in, you should ask yourself these questions:
Will the therapy meet your goals?
What evidence is available about the benefits of the therapy?
Is there evidence that the therapy benefits people with your type of illness?
Is there evidence that the therapy complements your conventional medical treatment plan?
What is the risk of harm of using a therapy? Has the therapy been shown to be safe, or does it cause side effects or interact with other treatments?
What is the recommended dose? More is not always better!
When is the most appropriate time to use the therapy: before, during, or after your conventional treatment?
Different types of research studies are often described using an evidence pyramid, like the one shown here. The different layers are referred to as "levels of evidence". The strength of evidence from a study type increases as you approach the top of the pyramid.8
Interpreting research results can be difficult. Understanding if and how research results apply to your unique situation is a skill that takes time to learn. To learn about the basics of each level in the evidence pyramid, click on the headings below and an explanation of each one of the types of studies will appear.
Systematic reviews and meta-analyses
These studies are the strongest level of evidence for treatment decisions. Systematic reviews are thoroughly researched summaries of evidence on a particular topic. A meta-analysis is a systematic review that combines the results of all studies in a single statistical analysis.
Clinical trials (in vivo research)
These studies can provide a strong level of evidence. In a clinical trial, a specific therapy is tested under strict conditions to show whether or not a treatment works without the influence of other factors not being tested. These factors can include, for example, a person's gender, hope or belief that a treatment may work, or other medications they may be on. Usually these trials involve individuals being randomly assigned to receive either the treatment being studied or standard care. Sometimes, a placebo is used instead of standard care. A placebo is any medical treatment that is inert.
These studies may be biased (because many factors have not been controlled for) and the evidence is not as strong for decision making. Cohort studies are often conducted when a clinical trial is not feasible or is ethically not appropriate (for example, palliative care or end of life care). It involves watching a group of people (a "cohort") over time, and may involve giving the cohort a specific treatment or watching for their use of specific substances and seeing how their long-term health or specific disease outcomes change.
These studies are good for developing hypotheses, but the level of evidence is not very strong for decision making. Case-control studies compare two groups of individuals: those with a disease or a health condition of interest and a very similar group of people who do not have the disease or condition. Researchers then explore the medical history and lifestyle of people in each group to learn what factors may cause the disease or condition.
These studies are considered weak evidence because the outcomes seen may be very specific to the individual. Case series involve giving a small number of people a treatment, and documenting the results. The evidence is limited by a lack of comparison to people not receiving treatment. As result the study outcomes may only apply to the people in the study.
These studies are considered very weak evidence because the outcomes seen may be very specific to the individual.
A case report is a detailed report of the diagnosis, treatment, and follow-up of an individual. Care reports are usually only for sharing interesting or critical clinical experiences amongst health care providers, and in some cases to suggest possible future research.
These studies may be biased and are considered very weak evidence because the outcomes seen may be very specific to the individual.
Expert opinions are based on health care providers' experience and reading of research. These opinions, however, may be biased and not relevant to each individual's situation:
If the expert wants you to think a certain way (promoting a product or idea), or
If the expert has their biases affecting which research.
Pre-clinical animal model research (in vivo research)
These studies do not provide useful evidence for treatment because the outcomes are yet to be applied in human models or a clinical trial setting.
Research conducted in animal models is often used to test specific hypotheses about treatment before testing the treatments in humans. Animal models allow closer approximation to a human response. However, there can be limitations and significant differences between the animal model and application to a human model.
In vitro "test-tube" research
These studies do not provide useful evidence for treatment because the outcomes are yet to be applied in human models or a clinical trial setting. In vitro literally mean "in glass". This is a test tube research that helps us learn much about possible treatments and how things work in the body at a cellular level. However, applying treatments to cells in petri dishes or glass containers will not tell how a treatment will work in a person. As such, these studies do not provide useful evidence for treatment decisions.
Making sense of information on complementary medicine
Information on complementary medicines can be presented in different forms. Different sources have advantages and disadvantages. Click on the headings below to learn about some handy tips and advice.
Using complementary medicine evidence
As you look through the information you will find credible sources suggested in the next sections, keep in mind the following key points. These points will be re-visited in Module Four, as we explore how to apply what you find in your treatment decisions.9
Questions to ask:
Does complementary medicine therapy work for my goal and in my unique situation? Does it work?
What side effects are described (is it safe?)
Does it interact with any drugs, treatments or medical conditions?
When was the study conducted and/or the information last updated? Is it still relevant?
What the experts say.....
Use strong evidence from high-level phase 3 human research studies to make health decisions.9
Base your decisions about the use of complementary medicine therapies on strong levels of evidence like that from systematic reviews, meta-analysis or human clinical trials.
However, although meta-analyses and systematic reviews are the strongest level of evidence, there are very few of them. In addition, these types of summary articles may not include studies that are relevant to your unique situation. As such, the best evidence for your situation is often original research articles specific to your type of condition and the complementary medicine therapy you are considering.
The next sections will describe where to find this reliable evidence.
Research articles provide important information, including whether complementary medicine therapy works, its possible side effects, and the correct dosage. Research articles will give information about dosages, side effects and benefits.
When selecting a research article, it is important to look for one that has been published within the last five years and has been peer reviewed, this will help ensure the reliability, relevance and quality of the article.
Articles can be found using the following on-line research databases:
Does the study consider effectiveness (does it work) and safety (is it safe)?
2. Was the study conducted in humans with my type of condition?
Results can differ depending on medical conditions, so this is an important question to ask. In addition, studies not conducted on humans are in early research stages and the results may not translate well to humans.
3. Who funded the study, and was it published in a peer-review journal?
Is there a possible conflict of interest between those who conduct the study and the information that is being provided?
Although research articles are among the most credible sources of evidence, it is not always possible for people to access articles, or find ones that are relevant to your situation. Instead, many individuals turn towards the internet to find information about complementary medicine therapies.
Caution is needed in using the internet as the credibility of websites can vary widely.
Reliable internet sources
Reliable complementary medicine internet resources have the following characteristics.10
A clear purpose
Are from a reliable source (e.g. University, medical facility, or government)
Are backed up by scientific evidence from published research articles to support the claims
Have been recently updated
Is applicable to the Australian marketplace
Not sure about reliability?
Consider the source of the information:
Consider the goals, funding, and reputation of the organisation providing the information
Be careful when using information from marketing sources or magazines aimed at the general public. Do they have a particular bias or goal?
Look for information based on clinical trials or population studies
Be cautious of "pseudoscience", very broad cure claims, or conspiracy theory messages
Questions to ask about websites
Check to make sure you can find the following information:
Who is the creator of the site?
Any good health site makes learning who is responsible for the site and its information an easy task. You should not have to be a detective to figure it out. Reliable sources include universities, medical facilities, government agencies and scientific societies.
What is the purpose of the site?
Make sure it states their purpose clearly (e.g. for education, for sales and promotion). Look for sites that do not profit from your decisions. The mission of the site relates to the goals and values of the creator. The purpose of the site should be clearly stated in the "About this site" or in a mission statement. Usually a statement is made that the site provides "unbiased and accurate health information".
What is the source of the information?
Many sites recycle information from other websites or offline resources. If the website lists the sources of information, this is of value. Is the website backed up by evidence from published research articles to support the claims made?
What is the basis of the information?
What evidence exists to support the information? Are facts and figures from medical/health research given? Is the information referenced to a journal or consensus statement? Has an expert panel reviewed the evidence? This type of information should be set apart and be clearly ascertainable.
How is the information selected?
The best sites are peer reviewed: their development of content includes a review process by an outside board or independent agency.
How current is the information?
Check that the website has been recently updated.
Tutorial finding credible information
This video tutorial will show you how to find complementary medicine information. Please press the play icon to watch this video.
If you want more information you can also contact your local library for help.
This tutorial involves searching for information on valerian for sleeping problems using evidence-based websites, and finding the latest research in the PubMed database.
Below you will find the links to the websites referenced in this tutorial:
This module has provided an introduction to scientific evidence and how to find research-based studies about complementary medicine. The next module will specifically look at how this information is used in decision making.
7. Lomas J, Culyer AJ, McCutcheon C, McAuley L, Law S. Conceptualizing and Combining evidence for health system guidance. Canadian Health Services Research Foundation. 2005. Pp.38. 8. Evidence Pyramid Diagram developed by the Medical Research Library of Brooklyn, SUNY Downstate Medical Center. Accessed on 11 March 2016. 9. Adams KE, Cohen MH, Eisenberg D, Jonsen AR. Ethical considerations of complementary and alternative medical therapies in conventional medical settings. Ann Intern Med. 2002;137(8):660-664. 10. Guidelines for evaluating websites in Complementary therapies on the internet MacBeker W, Berman B, Published by Churchill Livingstone, 2003. Pp.186.