• March 20-22, San Francisco

    2013 Summit on Clinical Research Informatics

2013 CRI Panels

There are six CRI Panels scheduled for CRI 2013. Full descriptions, date and time are available in the Itinerary Planner.


Integrating Governance of Research Informatics and Healthcare IT Across an Enterprise: Experiences from the Trenches

Moderator: Peter J. Embi, MD, MS, The Ohio State University, Columbus, OH
Panelists: Umberto Tachinardi, MD, University of Wisconsin-Madison, Madison, WI; Yves Lussier, MD, University of Illinois at Chicago, Chicago, IL; Jonathan Silverstein, MD, MS, NorthShore University HealthSystem, Evanston, IL


Advances in health information technology and biomedical informatics have laid the groundwork for significant improvements in healthcare and biomedical research. For instance, Electronic Health Records can help improve the delivery of evidence-based care, enhance quality, and contribute to discoveries and evidence generation. Despite this promise, there are many challenges to achieving the vision and missions of our healthcare and research enterprises. Given the challenges inherent in doing so, institutions are increasingly moving to establish dedicated leadership and governance models charged with designing, deploying and leveraging various information resources to advance research and advanced care activities at AHCs. Some institutions have even created a new leadership position to oversee such activities, such as the Chief Research Information Officer. This panel will include research informatics leaders discussing their experiences from the proverbial trenches as they work to operationalize such cross-mission governance models. Panelists will start by providing an overview their respective positions and environments, discuss their experiences, and share lessons learned through their work at the intersection of clinical and translational research informatics and Health IT.


Increasingly, academic health centers are recognizing the importance of enabling advances in healthcare and research via the leveraging of information systems. New governance and process models are needed to drive the optimal design and use of the various information systems related to advancing practice and research, and academic health center have begun to recognize the need for leaders to drive such efforts. Unfortunately, the very nature of what is often needed to enable research activities and clinical activities across an organization differs and this makes it challenging integrate research functionality into the clinical systems. This is not only a significant impediment to current research, quality, and decision support efforts, it also limits the ability to obtain new grant funding and impacts on the ability to recruit new faculty. Traditional clinical IT groups are rarely structured or staffed to meet even a small fraction of this growing demand, and without effective leadership and changes to governance and prioritization structures, research endeavors are often delayed in favor of projects impacting day-to-day clinical and financial operations. Indeed, resource levels and prioritization are not the only issues. Development and deployment cycles needed for success are often different between research and clinical IT projects, with the clinical systems forcing longer release cycles while development and release cycles for research projects often require implantation and testing of new functionality in weeks to months. Because of the necessity to learn the intricacies of the local clinical systems and the long planning cycle of clinical systems, it is frequently difficult or impossible to transfer production clinical staff to a research project in a timely manner, even when funding is available. Yet, fully redundant workforces for each mission area are often not practical. Similarly, related activities that increasingly are complex and require dedicated attention include those related to data warehousing, clinical research information systems, research resource tracking, research compliance and regulatory systems, and myriad issues around data storage and sharing.

Given the challenges inherent in achieving the promise of leveraging information systems across academic health centers for activities that span clinical care and research, institutions are increasingly moving to establish dedicated leadership and governance models charged with designing, deploying and leveraging various information resources to advance research and advanced care activities at AHCs. Some institutions have even created a new leadership position to oversee such activities. One such position is that of the Chief Research Information Officer (CRIO), a role that is critical to managing the interface between clinical systems and research needs. Indeed, that emerging role was the subject of a very popular panel at last year’s AMIA Summits on Translational Science. While progress is being made, those in leadership positions such as this one are quickly gaining experience that should be of great interest and value to the attendees of this year’s Summits on Translational Science. Therefore, we propose to follow-up to last year’s popular panel on research informatics governance by convening a group of leaders who are functioning in such roles across a range of medical centers. Through this panel, they will address in detail some of their experiences from the proverbial trenches.

Presentations summary and objectives

Dr. Embi will serve as moderator, leading off with a brief survey of the types of research informatics governance models emerging and speak briefly to his experiences as a CRIO. The other four panelists (Drs. Tachinardi, Lussier, Starren, and Silverstein) will then follow sequentially, each with presentations of approximately 10-minute length. Panelists will provide an overview their respective positions and environments, discuss their experiences as leaders at the intersection of research and health informatics/IT within their respective organizations, and share lessons learned with the audience. The panel will conclude with a question and answer exchange with the audience.

At the conclusion of the session, attendees should be able to:

  • Provide a survey of the types of governance models that are emerging and how they are positioned within their organizations;
  • Review the challenges that must be overcome to successfully integrate research practices and methods into health information system environments;
  • Discuss prevailing examples of ongoing efforts to work across organizational boundaries to develop and operationalize comprehensive research and clinical information solutions for the enterprise;
  • Discuss how groups are successfully leveraging academic informatics groups to advance their research and clinical missions and realize “learning health systems”


A comprehensive framework for data quality assessment in CER

Moderator: Erin Holve, AcademyHealth, Washington, DC
Panelists: Michael Kahn, University of Colorado Anschutz Medical Campus, Denver, CO; Meredith Nahm, Duke University, Durham, NC; Patrick Ryan, Observational Medical Outcomes Partnership, Bethesda, MD; Nicole Weiskopf, Columbia University, New York, NY


The panel addresses the urgent need to ensure that comparative effectiveness research (CER) findings derived from diverse and distributed data sources are based on credible, high-quality data; and that the methods used to assess and report data quality are consistent, comprehensive, and available to data consumers. The panel consists of representatives from four teams leveraging electronic clinical data for CER, patient centered outcomes research (PCOR), and quality improvement (QI) and seeks to change the current paradigm where data quality assessment (DQA) is performed “behind the scenes” using one-off project specific methods. The panelists will present their process of harmonizing existing models for describing and measuring clinical data quality and will describe a comprehensive integrated framework for assessing and reporting DQA findings. The collaborative project is supported by the Electronic Data Methods (EDM) Forum, a three-year grant from the Agency for Healthcare Research and Quality (AHRQ) to facilitate learning and foster collaboration across a set of CER, PCOR, and QI projects designed to build infrastructure and methods for collecting and analyzing prospective data from electronic clinical data.


Detailed clinical data from disparate data sources, including electronic health records (EHRs), is the backbone of large-scale comparative effectiveness research (CER). Yet there exists no formal methods for assessing and reporting on the quality of data obtained from these sources. This proposal will develop a comprehensive data quality assessment framework and guidelines for the CER community.

This panel will present the collaboration between diverse research teams leveraging electronic clinical data for research and quality improvement (QI). The goal of this collaboration is to create draft recommendations and guidelines that can guide the development of new analytic and reporting methods specifically directed to data quality assessment and reporting for CER studies. The long-term vision is that all EHR-based clinical studies and all publically available data sets would be linked to data quality assessment results that allow for an independent assessment of the quality of the data used to generate the reported results. In addition, as NIH data sharing requirements become more stringent, the presence of uniform, standardized data quality assessment measures enables a potential data consumer to determine if a given data set is sufficient for their intended use.

The collaborative project is supported by the Electronic Data Methods (EDM) Forum, a three-year grant from the Agency for Healthcare Research and Quality (AHRQ) to facilitate learning and foster collaboration across a set of CER, patient centered outcomes research (PCOR), and QI projects designed to build infrastructure and methods for collecting and analyzing prospective data from electronic clinical data. The EDM Forum has commissioned collaborative projects that examine current challenges and opportunities for conducting CER, PCOR, and QI with electronic clinical data. Specific areas of focus include aspects of the data governance, clinical informatics, and analytic issues that are crucial to the design and use of electronic clinical data for CER, as well as lessons learned from quality improvement and other efforts to use electronic clinical data for health research and clinical care. The EDM Forum and the research projects connected to the Forum are funded by the American Recovery and Reinvestment Act (ARRA).

Panel Overview

The panel consists of representatives from four teams leveraging electronic clinical data for CER, PCOR, and QI. One representative from each of the projects will describe their pre-existing model for describing and measuring clinical data quality and will describe their role in constructing a harmonized model of data quality that captures the key elements of their individual models and any additional features described in the clinical data quality assessment literature.


Pains and Palliation in Distributed Research Networks: Lessons from the Field

Moderator: Michael G. Kahn MD, PhD, University of Colorado Denver, CO
Panelists: Jeffrey Brown PhD, Harvard Pilgrim Health Care Institute Boston, MA; Lisa Dahm, PhD, University of California Irvine, CA; Daniella Meeker PhD, Rand Corporation, Los Angeles, CA; Lisa Schilling MD, MSPH, University of Colorado Denver CO


Large-scale comparative effectiveness research studies require detailed clinical data collected across disparate clinical practice settings and institutions. Distributed research networks (DRNs) have been promoted as one approach to wide-scale data sharing that enables data sharing organizations to retain local data ownership and access control. Despite significant investments in distributed data sharing technologies, clinical research networks using distributed methods remain difficult to implement due to a broad range of organizational and technical barriers. The panelists represent four different research networks are in different stages of implementation maturity and are leveraging different informatics technologies. Challenges common to all DRNs include governance, semantic interoperability, and identity management. This panel will describe some of the critical challenges and experimental solutions to implementing, expanding, and sustaining DRNs. Each panelist will focus on a specific challenge that requires new informatics tools to reduce barriers to participation and data sharing.

Problem Description

With the growing deployment of electronic health records, the volume of detailed clinical observations collected during routine clinical care in standard clinical practices is increasing rapidly. Comparative effectiveness research (CER) focuses on evaluating the relative effectiveness of clinical interventions as observed in actual clinical practice rather than under artificial ideal conditions created in controlled clinical trials. Thus, access to detailed clinical data across a wide range of clinical practices is necessary for CER to compare clinical outcomes across differing clinical settings, workflows, constraints, and populations. A substantial investment has been made in creating large practice-based research networks. Many require technical staff to perform study-specific data extractions which are sent to a central coordinating center for analysis. DRNs have been promoted as an alternative approach than enables data contributing sites to retain control over data access and release. In a distributed network, data are extracted to a local data store whose content and access is controlled by local data administrators. Data requests are managed via a centralized data portal or coordinating center, are sent to all data sites who have agreed to participate in the data sharing activity. Local sites may deny access to their data at any time. Distributed data sharing models have held the promise of enabling a wider range of clinical sites to participate in national CER efforts. Yet, despite successes in non-medical fields, distributed data sharing methods have struggled to achieve wide spread adoption and sustainability. Significant technical and non-technical challenges remain. Existing informatics tools are too cumbersome to enable broad deployment in many clinical practices, even those with electronic health records. New tools and methods are required that substantially reduce barriers to data sharing for the vision of CER to be realized.

This panel brings together leaders of multiple DRN projects. These leaders bring real-world experiences with deploying and sustaining early-stage and more-mature networks. The diversity of technologies and approaches provides an opportunity to learn from these differing settings and experiences and provide an opportunity to challenge the Clinical Research Informatics community to develop the next-generation data sharing platforms and tools that enable smaller clinical practices to contribute to a national CER network.


Standard-based integration profiles for clinical research and patient safety

Panelists: Landen Bain, CDISC, USA; Christel Daniel, MD, PhD, INSERM UMRS 872eq20, Paris, France; 3AP-HP, Paris, France; Brendan C Delaney, BM, BCh, MD, Kings College of London, UK; Vasa Curcin, PhD, Imperial College London, UK; Gokce Banu Laleci Erturkmen, PhD, Software Research, Development and Consultancy, Ankara, Turkey


EHRs can now be adapted to integrate seamlessly with existing research platforms. However, key challenges need to be overcome in order to provide a platform that functions across many EHR systems.

The IHE Quality, Research and Public Health (QRPH) domain addresses the information exchange standards necessary to share information relevant to quality improvement in patient care and clinical research. In collaboration with CDISC’s Healthcare Link initiative, IHE QRPH has developed a set of integration profiles that specifically address EHR-enabled research.
The panel participants from three European projects will present how subsets of existing IHE QRPH profiles can be pulled together (and extended when necessary) to form a super profile which will standardize and automate the clinical trial process flow.

The EHR4CR project is providing adaptable, reusable and scalable tools and services for reusing data from hospital EHRs for Clinical Research. TRANSFoRm is developing an informatics infrastructure to support the learning healthcare system in European Primary Care. SALUS project is providing scalable, standard based interoperability framework for sustainable proactive post market safety studies. Overall, the panel will discuss the key steps towards realizing a joint EHR4CR/TRANSFoRm/SALUS European projectathon demonstrating EHR-enabled clinical research across Europe using standard-based integration and content profiles.

General description of the panel and issue(s) that will be examined
EHRs can now be adapted to integrate seamlessly with existing research platforms thus creating a unique opportunity for many stakeholders, including hospitals, clinical research promoters, pharmaceutical industry and policy makers. However, key challenges, including security and semantic interoperability issues, need to be overcome in order to provide a platform that functions across many EHR systems.

A set of 9 IHE QRPH profiles and standards that can be pulled together to form a super profile which will standardize and automate the clinical trial process flow of the EHR4CR platform from the patient recruitment to the submission of the clinical trial data to the sponsors will be presented. These profiles address the aspects of

  • i) representing and sharing a clinical research protocol for its execution (CRPC(Clinical Research Process Content), RPE(Retrieve Process for Execution)),
  • ii) representing and sharing clinical research documentation (eCRF, adverse event reporting form) to be pre-populated by existing clinical data in EHRs (RFD(Retrieve Form for Data Capture), CRD(Clinical Research Document), DSC(Drug Safety Content), RSP(Redaction Service Profile)),
  •  iii) addressing confidentiality and security aspects (CT(Consistent Time), XUA(Cross-Enterprise User Assertion), ATNA(Audit Trail Node Authentification))
  • iv) additional profiles, currently in the proposal stage, will further refine and extend these capabilities.

The panel participants will present their technical implementation approaches for integrating EHRs to clinical research and compare it to the specification of the IHE QRPH integration profiles.

The panel participants will propose a global use case for a projectathon that leverages the use cases of each of the profiles involved (from the ITI technical framework (CT, XUA, ATNA, RFD) and the QRPH technical framework (CRD, DSC, CRPC, RPE). This global use case reproducing a clinical trial timeline will be a first step towards global interoperability between EHR4CR, TRANSFoRM and SALUS platforms and towards a pan-EU capability for clinical research and patient safety.

The panel participants will especially focus on semantic interoperability issues. Integrated interoperability for clinics across countries in Europe, requires widespread access to published and maintained collections of coherent and quality-assured semantic resources, including models such as archetypes and templates (providing clinical context) that are based on reference information models (e.g HL7 RIM, openEHR), standardized data elements based on ISO data types and linked to well specified multi-lingual terminology value sets derived from high quality ontologies. The panel will discuss the specific role of the CDISC SHARE initiative and how semantic resources should be defined, validated, and disseminated. The panel will also discuss how users should be educated to improve the quality and consistency of EHR documentation and the re-use of primary health data.


Next-Generation Registries: Fusion of Data for Care and Research

Moderator (and panel organizer): Kenneth D. Mandl, MD, MPH, Harvard Medical School, Boston Children’s Hospital, Boston, Massachusetts
Panelists: Stephen Edge, MD, Chair, Health Services and Outcomes Research, Medical Director, Breast Center, Roswell Park Cancer Center; Chad Malone, Vice President of Medical Affairs Vice President of Business Development, Remedy Informatics; Keith Marsolo, PhD, University of Cincinnati College of Medicine, Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center; Marc D. Natter, MD, Boston Children’s Hospital, Harvard Medical School


Disease-based registries are a critical tool for electronic data capture of high-quality, gold standard data for clinical research as well as for population management in clinical care. Yet, a legacy of significant operational costs, resource requirements, and poor data liquidity have limited their more widespread use. Research registries have incurred more than $3 Billion in HHS investment over the past 17 years. Health delivery systems and Accountable Care Organizations are now investing heavily in registries to track care quality and manage the health of patient panels. Despite the investment, regulatory and financial models have often enforced a “single purpose” limitation on each registry, restricting the use of data to a pre-defined set of protocols and users. The need for cost effective, multi-sourced, and widely shareable registry data sets has never been greater, and requires next-generation platforms to robustly support multi-center studies, comparative effectiveness research, post-marketing surveillance and disease management. This panel explores diverse registry efforts, both academic and commercial, that have been implemented in leading edge clinical, research, and hybrid use cases. Panelists present their experience in these areas as well as lessons learned, challenges addressed, and near-term innovations and advances.

General Description of Program

The Institute of Medicine describes a Learning Healthcare System – a clinical medicine model in which outcomes are continuously monitored, the knowledgebase is continuously updated, and hypotheses may be rapidly tested. At its core, this represents a new paradigm for enlisting the health care enterprise as a creator and consumer of rich data for continuous improvement, innovation and discovery. A key tool in a learning health system is the clinical research registry, which historically has been well suited to the study of select populations and rare diseases. Results have been important in managing and advancing understanding of many conditions over the past decades, for example in cystic fibrosis, many forms of cancer, and drug and device safety surveillance. Classically, registries been manually populated by research staff interviewing patients and reviewing charts.

But now, electronic health records (EHRs) offer a complementary data source, large quantities of low-cost, ambient, and typically ad-hoc data gleaned from multiple health system sources, collected according to clinical workflows and requirements of regulators and payors. Such data sets are, however, often incomplete, rarely validated, and not typically accompanied by sufficiently detailed metadata or according to standardized vocabularies so as to permit meaningful data combination across sites.

Further, with consumer-facing IT, patients become a potential direct source of data for registries. Next-generation disease registry platforms must enable collection, aggregation and permissioned sharing of high quality data sets across the healthcare enterprise. We have assembled an expert panel to explore the benefits and challenges of using next-generation registry platforms for multi-sourced, multi-center studies, comparative effectiveness research, post-marketing surveillance and clinical management, focusing on how these systems may best contribute to accomplishing central infrastructure requirements of a Learning Healthcare System.


Tools for Identifying Reliable Evidence and Implementing it in Everyday Clinical Care

Panelists: Aaron M. Cohen, MD, MS, Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR; Dina Demner-Fushman, MD, PhD, National Library of Medicine, National Institutes of Health, Washington, DC; Alfonso Iorio, MD, Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada; Ida Sim, MD, PhD, Department of Medicine, University of California, San Francisco, CA; Neil R. Smalheiser, MD, PhD,Department of Psychiatry, University of Illinois at Chicago, Chicago, IL.


Just as translational medicine follows a long winding path from bench-to-bedside, so can Evidence-Based Medicine be envisioned as comprising a multi-step pipeline, from building evidence from raw data through synthesizing best practices and providing clinical decision support in a process described as the “evidence pyramid”.1 At one end, a heterogeneous mix of clinical and experimental studies including clinical trials, case reports, animal models and retrospective analyses are published as new knowledge. Then, experts collect and assess high-quality relevant evidence on specific issues and publish their conclusions (e.g., regarding efficacy and safety of treatments) as systematic reviews and meta-analyses. Finally, when an expert consensus has been reached, this must reach the attention of policy makers within the profession, the government and insurance companies, resulting in new practice guidelines and altered clinical practice within hospitals and clinics. At each stage, this process requires a large investment of time and effort from many individuals with a wide range of expertise.

Our panel will discuss the variety of innovative approaches that are being taken by different informatics research groups to improve each step within the evidence based medicine pipeline. These approaches are, in part, devoted to making existing data collection and synthesis practices faster and more efficient, but they also involve re-imagining and re-engineering the processes by which evidence is accumulated, evaluated and applied.

General Description

The practice of Evidence-Based Medicine (EBM) is critically dependent upon judicious use of the best and most up-to-date evidence to patient care decision making.2 Few clinical practitioners would disagree that this fundamental orientation is useful and appropriate. However, actually putting EBM into practice has uncovered many challenges. For example: How do we identify the medical topics for which evidence review and synthesis are most needed? Current groups such as the Cochrane Collaboration currently produce systematic reviews for only about 8,000 topics; this is not nearly enough to cover common and important medical conditions.3 For a given medical topic, how can all of the relevant literature be appropriately and efficiently collected, reviewed, and synthesized? What counts as evidence appropriate for a particular medical question? Do randomized controlled trials (RCTs) always provide the best quality evidence? What about medical questions for which RCTs do not exist or would be considered unethical? Finally, how can we best ensure that EBM reports impact clinical guidelines and electronic decision support tools to improve the care given by medical, dental, nursing and allied health practitioners?

These challenges can be roughly grouped according to the steps in the process of transforming individual units of evidence (such as publications) into a form that is effective for guiding and improving medical practice:

  1. Identifying topics requiring an analysis or re-analysis of the available evidence.
  2. Gathering a comprehensive set of evidence that is relevant to the topic.
  3. Assessing/synthesizing the available evidence in a timely and efficient manner.
  4. Making high quality evidence syntheses easily available.
  5. Translating evidence syntheses into action through practice guidelines and clinical decision support tools.

This process is cyclic; as medical research progresses and additional knowledge is discovered, new areas are identified that require evidence and analysis, and areas previously reviewed require updating. Therefore, the effort required to meet the EBM goal of “judicious use of the best available evidence” is large, ongoing, and exponentially increasing.4 Currently the steps in this process are implemented in a manner that is incomplete, cumbersome, loosely organized, and largely manual. The entire process could be improved by appropriate informatics tools that support the individual activities as well as the transitions or “hand offs” of the output of one step to the next.

This panel will discuss current and potential future work in applying advanced informatics tools and platforms to facilitate the EBM pipeline. Speakers will summarize the current state of work in their focused areas of research, and will lead a discussion on the opportunities and challenges for creating an integrated, public, collaborative platform of tools that will speed up the process of creating, disseminating, and applying the best available evidence across all areas of medicine and realize the full potential of EBM.