Your Biomarker doesn’t work (and a case for reproducible research)!

We have all hear about those “breakthrough biomarkers” that.. simply.. do not work!
There have been numerous reports in the literature that investigate why certain biomarkers don’t work after all.
But before discussing the reasons for failed biomarkers, let me give some examples of disease biomarkers that have been successfully used for diagnosis, prognosis or monitoring. For example:

  • The diagnosis of diabetes is based on the level of glucose in serum after 12 hours of fasting
  • Rubidium chloride is used in isotopic labeling to evaluate perfusion of heart muscle
  • The level of serum creatinine is a great indicator of renal function.
  • cholesterol values are a biomarker and risk indicator for coronary and vascular disease
  •  C-reactive protein (CRP) is a marker for inflammation.

A paper from (Eleftherios P. Diamandis (2010)) shows a few examples of “highly publicized biomarkers” that were never possible to validate. Among the many reasons for failed biomarkers are issues that have to do with experimental design (non-homogeneous groups, biases in selection of case patients and control subjects, for instance not matched by age, gender), methodological artifacts (problems with the analytical method), bias in sample collection and storage (e.g. in one case some cancer samples were much older than the samples from the controls).

In a biomarker study, the aim is to a set of measurable features that are predictive of a certain phenotype. Using microarray-, PCR- or RNA-Seq-based assessments of mRNA abundances, one can identify “transcriptomic biomarkers” – essentially a set of genes (sometimes called signatures) that can be used to for diagnosis, disease monitoring, to predict outcome, etc. However, in many cases, the uniqueness of the derived signature has been questioned, and it has noted that the signature may not be particularly informative of the underlying biological mechanisms. A paper by Venet et al., 2011 (“Most Random Gene Expression Signatures Are Significantly Associated with Breast Cancer Outcome“) points out some funny examples, for instance they show that a gene expression signatures—unrelated to cancer—of the effect of postprandial laughter, of mice social defeat and of skin fibroblast localization were all significantly associated with breast cancer outcome.

How data pre-processing can affect your biomarker results:

But, from all the reasons why a biomarker may not work, there is one that left me quite surprised. Paul C Boutros’s research has raised concerns about sensitivity of the generated biomarkers to relatively subtle changes in data pre-processing, demonstrated in lung cancer (shown by Starmans et al., 2009) and in tumour hypoxia biomarkers (Fox et al., 2014)!

Figure 2 from Fox et al., 2014

The figure shows the prognostic power of individual genes varies dramatically across different data pre-processing. The authors in Fox et al., 2014 propose an “ensemble” of pre-processing methodologies to identify biomarkers.

A case for reproducible research

I find both papers very interesting and will be using them as an example of the need to work towards the understanding of the unknown contributions of subtle pipeline changes and parameters on the accuracy of any result (not just biomarkers!). Many of the current challenges in ‘reproducible research’ are related to changes in individual pipeline components or software that make our results not possible to reproduce. That is why adopting a reproducible workflow, especially in computational biology, documenting and sharing your code and analysis decisions, makes it easier for others to reproduce and even extend your findings!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s