Evaluation of a Training Program to Improve Organizational Capacity for Health Systems Analytics

 

 Permissions and Reprints

Abstract

Objective The Leadership in Analytics and Data Science (LEADS) course was evaluated for effectiveness. LEADS was a 6-month program for working biomedical and health informatics (BMHI) professionals designed to improve analytics skills, knowledge of enterprise applications, data stewardship, and to foster an analytics community of practice through lectures, hands-on skill building workshops, networking events, and small group projects.

Methods The effectiveness of the LEADS course was evaluated using the Kirkpatrick Model by assessing pre- and postcourse knowledge, analytics capabilities, goals, practice, class lecture reaction, and change in the size of participant professional networks. Differences in pre- and postcourse responses were analyzed with a Wilcoxon signed rank test to determine significance, and effect sizes were computed using a z-statistic.

Results Twenty-nine students completed the course with 96% of respondents reporting that they were “very” or “extremely” likely to recommend the course. Participants reported improvement in several analytics capabilities including Epic data warehousing (p = 0.017), institutional review board policy (p = 0.005), and data stewardship (p = 0.007). Changes in practice patterns mirrored those in self-reported capability. On average, the participant professional network doubled.

Conclusion LEADS was the first course targeted to working BMHI professional at a large academic medical center to have a formal effectiveness evaluation be published in the literature. The course achieved the goals of expansion of BMHI knowledge, skills, and professional networks. The LEADS course provides a template for continuing education of working BMHI professionals.

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the JHM Institutional Review Board.

• References

• 1 Davenport TH, Harris JG. Competing on Analytics: The New Science of Winning. Boston, MA: Harvard Business Review Press; 2017
  Google Scholar
• 2 Wachter RM, Howell MD. Resolving the productivity paradox of health information technology: a time for optimism. JAMA 2018; 320 (01) 25-26
  CrossrefPubMedGoogle Scholar
• 3 Burns WA. Healthcare data's perfect storm: Why healthcare organizations are drowning in the data they are creating and why they need even more data to weather the storm. Updated 2013. Available at: https://s3.amazonaws.com/rdcms-himss/files/production/public/HIMSSorg/Content/files/HitachiDataSystems_BR-458_perfectStorm.pdf . Accessed February 2, 2019
  PubMedGoogle Scholar
• 4 Violino B. The secrets of highly successful data analytics teams. Updated 2017. Available at: https://www.cio.com/article/3234353/the-secrets-of-highly-successful-data-analytics-teams.html?upd=1552589464590 . Accessed March 27, 2019
  PubMedGoogle Scholar
• 5 Mantas J, Ammenwerth E, Demiris G. , et al; IMIA Recommendations on Education Task Force. Recommendations of the international medical informatics association (IMIA) on education in biomedical and health informatics. First revision. Methods Inf Med 2010; 49 (02) 105-120
  Article in Thieme ConnectPubMedGoogle Scholar
• 6 Hersh W. The health information technology workforce: estimations of demands and a framework for requirements. Appl Clin Inform 2010; 1 (02) 197-212
  Article in Thieme ConnectPubMedGoogle Scholar
• 7 Berner ES, Dorsey AD, Garrie RL, Qu H. Assessment-based health informatics curriculum improvement. J Am Med Inform Assoc 2016; 23 (04) 813-818
  CrossrefPubMedGoogle Scholar
• 8 Breeden EA, Clauson KA. Development and implementation of a multitiered health informatics curriculum in a college of pharmacy. J Am Med Inform Assoc 2016; 23 (04) 844-847
  CrossrefPubMedGoogle Scholar
• 9 Tremblay MC, Deckard GJ, Klein R. Health informatics and analytics - building a program to integrate business analytics across clinical and administrative disciplines. J Am Med Inform Assoc 2016; 23 (04) 824-828
  CrossrefPubMedGoogle Scholar
• 10 Jaspers MW, Mantas J, Borycki E, Hasman A. IMIA accreditation of biomedical and health informatics education: current state and future directions. Yearb Med Inform 2017; 26 (01) 252-256
  Article in Thieme ConnectPubMedGoogle Scholar
• 11 Singer JS, Cheng EM, Baldwin K, Pfeffer MA. ; UCLA Health Physician Informaticist Committee. The UCLA health resident informaticist program - a novel clinical informatics training program. J Am Med Inform Assoc 2017; 24 (04) 832-840
  CrossrefPubMedGoogle Scholar
• 12 Hübner U, Shaw T, Thye J, Egbert N, Marin H, Ball M. Towards an international framework for recommendations of core competencies in nursing and inter-professional informatics: the TIGER competency synthesis project. Stud Health Technol Inform 2016; 228: 655-659
  PubMedGoogle Scholar
• 13 Dorsey AD, Clements K, Garrie RL, Houser SH, Berner ES. Bridging the gap: a collaborative approach to health information management and informatics education. Appl Clin Inform 2015; 6 (02) 211-223
  Article in Thieme ConnectPubMedGoogle Scholar
• 14 Fridsma DB. Developing the health informatics workforce of the future: academic and industry partners. J Am Med Inform Assoc 2017; 24 (03) 677-678
  CrossrefPubMedGoogle Scholar
• 15 Khairat S, Sandefer R, Marc D, Pyles L. A review of biomedical and health informatics education: a workforce training framework. J Hosp Adm 2016; 5 (05) 10-20
  PubMedGoogle Scholar
• 16 Palis AG, Quiros PA. Adult learning principles and presentation pearls. Middle East Afr J Ophthalmol 2014; 21 (02) 114-122
  CrossrefPubMedGoogle Scholar
• 17 White SK. The 13 most in-demand tech jobs for 2019 - and how to hire for them. Updated 2018. Available at: https://www.cio.com/article/3235944/hiring-the-most-in-demand-tech-jobs-for-2018.html . Accessed March 27, 2019
  PubMedGoogle Scholar
• 18 Hersh W, Margolis A, Quirós F, Otero P. Building a health informatics workforce in developing countries. Health Aff (Millwood) 2010; 29 (02) 274-277
  CrossrefPubMedGoogle Scholar
• 19 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385
  CrossrefPubMedGoogle Scholar
• 20 Furukawa MF, Vibbert D, Swain M. HITECH and health IT jobs: Evidence from online job postings. Updated 2012. Available at: https://www.healthit.gov/sites/default/files/pdf/0512_ONCDataBrief2_JobPostings.pdf . Accessed March 13, 2019
  PubMedGoogle Scholar
• 21 Anderson JG. Evaluation in health informatics: social network analysis. Comput Biol Med 2002; 32 (03) 179-193
  CrossrefPubMedGoogle Scholar
• 22 Park JY, Im I, Chang-Soo S. Is social networking a waste of time? The impact of social network and knowledge characteristics on job performance. Knowl Manag Res Pract 2017; 15 (04) 560-571
  PubMedGoogle Scholar
• 23 Bennett J, Owers M, Pitt M, Tucker M. Workplace impact of social networking. Property Manage 2009; 28 (03) 138
  PubMedGoogle Scholar
• 24 Gaggioli A, Mazzoni E, Milani L, Riva G. The creative link: investigating the relationship between social network indices, creative performance and flow in blended teams. Comput Human Behav 2015; 42: 157
  CrossrefPubMedGoogle Scholar
• 25 Dinter B, Moslein KM, Kollwitz C. , et al. Combining open innovation and knowledge management for a community of practice - an analytics driven approach. Twenty-second Americas Conference on Information Systems; 2016
  PubMedGoogle Scholar
• 26 Wenger E, McDermott RA, Snyder W. Cultivating Community of Practice: A Guide to Managing Knowledge. Boston, MA: Harvard Business School Press; 2002
  Google Scholar
• 27 Nagy P, Kahn Jr CE, Boonn W. , et al. Building virtual communities of practice. J Am Coll Radiol 2006; 3 (09) 716-720
  CrossrefPubMedGoogle Scholar
• 28 Wenger-Trayner E, Wenger-Trayner B. . Updated 2015. Available at: Https://Wenger-trayner.com/wp-content/uploads/2015/04/07-brief-introduction-to-communities-of-practice.pdf https://wenger-trayner.com/introduction-to-communities-of-practice/ . Accessed March 27, 2019
  PubMed
• 29 Rouse DN. Employing Kirkpatrick's evaluation framework to determine the effectiveness of health information management courses and programs. Perspect Health Inf Manag 2011; 8: 1c
  PubMedGoogle Scholar
• 30 Moldovan L. Training outcome evaluation model. Procedia Technol 2016; 22: 1184-1190
  PubMedGoogle Scholar
• 31 Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco: Berett-Koehler; 1994
  Google Scholar
• 32 Fast facts: Johns Hopkins Medicine. Updated 2018. Available at: https://www.hopkinsmedicine.org/about/downloads/JHM-Fast-Facts.pdf . Accessed March 27, 2019
  PubMedGoogle Scholar
• 33 The Data Trust. Updated 2018. Available at: https://intranet.insidehopkinsmedicine.org/data_trust/index.html . Accessed March 27, 2019
  PubMedGoogle Scholar
• 34 Phillips JJ. Handbook of Training Evaluation and Measurement Methods. 3rd ed. Houston, TX: Gulf Publishing; 1997
  Google Scholar
• 35 Kerby DS. The simple difference formula: an approach to teaching nonparametric correlation. Compr Psychol 2014; 3 (01) 1-10
  CrossrefPubMedGoogle Scholar

• Supplementary Material