The Laura P. and Leland K. Whittier Virtual PICU

Learn more about the VPICU

The Laura P. and Leland K. Whittier Virtual Pediatric Intensive Care Unit (VPICU)

L.K. Whittier Foundation

Supported by the incredible generosity and foresight of the L.K. Whittier Foundation since 1998, the Laura P. and Leland K. Whittier Virtual Pediatric Intensive Care Unit (VPICU) has brought the benefits of information science technology to the care of critically ill children across the country. In partnership with the L.K. Whittier Foundation, the VPICU has become the central resource for pediatric critical care quality, research and education in the United States. Information derived from the VPICU data resources continues to improve the quality of care at Children’s Hospital Los Angeles (CHLA) and throughout the nation—letting the actual evidence from the care of thousands of critically ill children inform the care of the next child.

The VPICU mission

The VPICU mission is to create a common information space for the providers of critical care. This vision preceded Facebook, the social web, Listservs and big data and pioneered the application of machine learning and artificial intelligence to this task. We have created this common information space through web sites, telemedicine, internet-based communication, collaborative research and critical-care quality improvement.

Our successes have decreased the time to discovery and helped improve the quality of care for critically ill children by identifying and sharing best practices and benchmarking excellence. The VPICU has grown to national prominence, expanding the information space using crowd sourcing, collaboration, knowledge dissemination, dynamic content, cloud computing, rich user experiences and, most critically, user-generated content and design. Applications not even imagined when the VPICU was founded are reaching the bedside in the nation’s PICUs.

The VPICU is the pre-eminent communication highway in pediatric critical care and has supported research and quality improvement. It has contributed to saving the lives of tens of thousands of children. However, there is much more that we need do to bring the full benefits of the VPICU to the bedside of critically ill children. In the modern information landscape, big data is changing health care practice. The VPICU was founded on the concept of big data and has been exploiting big data for over ten years with the help of the L.K. Whittier Foundation. The VPICU has created a vast data lake of critical care data from across the United States detailing how critical illness happens to children. The data we have collected is so massive and nearly beyond human comprehension, that it now needs to be managed and made relevant and available in the right place at the right time.


A Common Information Space

What is a Common Information Space?

Fifteen years ago, the common information space seemed a strange concept. The space, as imagined then, represented a combination of communication technologies that would bring pediatric critical care together as one Virtual ICU. Since then, this virtual space has been used to care for children geographically distant from academic ICUs, educate caregivers, conduct research and support quality improvement.

Today, the common information space is readily understood as a space where knowledge is shared and is bounded only by the internet and defined by myriad communication technologies developed by the VPICU.

pedsccm screenshot


A collaborative, independent, information resource for the PICU community.

piculist screenshot


Serving caregivers for over two decades, the PICULIST is now its own Slack channel.



A crowdsourced knowledge base for pediatric critical care medicine.


LEADING THE WAY: Artifical Intelligence in the PICU


Nowhere is rapid decision making more urgent, or more complex, than in managing critically ill children. Important to that decision making is the ability to identify similar patients. Patient similarity has long been a challenging thorn to many researchers because of its subjective nature.

The Laura P. and Leland K. Whittier Virtual Pediatric Intensive Care Unit Data Science team works closely with physicians at Children’s Hospital Los Angeles to develop a framework to address this challenge.

The clinical status of an individual patient can be defined by multiple contexts. To understand a patient’s clinical status, the VPICU Patient Similarity Framework combines these contexts as needed by corresponding modules. A module generates a representation for each individual patient that enables the computation of a mathematical distance between any two patients within a given clinical context (e.g. mortality, diagnosis, volatility, etc). The summation of the distances from various contexts of interest enables the retrieval of similar patients.

Listen in to Learn More

Dynamically Evolving Severity of Illness Score

Severity of illness (SOI) scores have been developed since the early 1980s to aid clinicians assess their critically ill patients. Examples of clinically-used systems are the Pediatric Index of Mortality (PIM) and Pediatric Risk of Mortality (PRISM). The VPICU developed a Recurrent Neural Network (RNN) that continuously generates individual SOI scores by predicting an individual child’s risk of ICU mortality. By incorporating many more variables than traditional models and integrating measurements of those variables in a dynamic manner, the RNN model demonstrated significantly higher discrimination, measured by the Area Under the Receiver Operating Characteristic (AUROC), than other models. The VPICU’s work demonstrates the RNN’s potential to provide accurate, continuous, and real-time assessment of a child’s condition in the ICU.

Area Under the Receiver Operating Characteristic (AUROC) curve of RNN mortality model predictions at different times after ICU admission. Shown for reference are the AUROCs of PRISM III (12th hour version), PIM2, and PELOD Day 1 scores in the same cohort.

Responsive image

Predicting Cardiac Death Within 1 Hour After Terminal Extubation

In US pediatric ICUs, limitation or withdrawal of life sustaining treatment is the most common mode of death. Intensivists are often called upon to counsel families about the dying process and to estimate time to death after treatment withdrawal. Some patients that have planned treatment withdrawal may be eligible for organ donations after cardiac death (DCD). DCD presents significant challenges; these include identifying patients who are anticipated to die within a set timeframe after treatment withdrawal, managing family expectations in case the donor does not die within the timeframe for donation, and utilizing operating rooms and organ procurement teams.

Using historical data from CHLA’s PICU, the VPICU team developed a LSTM-based RNN and a Cox Proportional Hazard (CPH) model to predict death within an hour of terminal extubation. The team also implemented the Dallas Predictor Tool (DPT) developed by Children’s Medical Center Dallas for comparison. The RNN and CPH models showed higher discrimination (as measured by AUROC) than the DPT model, both in the overall cohort and in the subcohort of DCD candidates. At any fixed sensitivity, the VPICU’s RNN and CPH models also achieved a lower number of needed to alert than the DPT model, showing the potential for the VPICU’s models to help identify potential candidates for DCD with minimal institutional waste in terms of preparing the operating room. The models developed by VPICU were able to identify 93% of potential organ donors with a number needed to alert of 1.08, meaning that 13 of 14 prepared operating rooms would have yielded a viable organ. The VPICU is now collaborating with several institutions to improve and validate the CPH model.

Responsive image

Interpreting a Recurrent Neural Network Model for ICU Mortality

Deep learning algorithms have demonstrated success in a wide range of applications; however the lack of transparency into how they generate predictions has impeded their acceptance in healthcare, where decisions may be lifesaving. The ability to determine how input features contribute to a model’s predictions is important for several reasons. First, it may provide useful information for clinical intervention or new avenues for investigation. Second, it facilitates an environment in which users can interact with the model and learn its strengths and weaknesses. Third, the information can be used to improve the model itself.

The VPICU developed Learned Binary Masks (LBM) to identify which input features were used by a Recurrent Neural Network (RNN) model to predict an individual child’s risk of mortality (ROM). In essence, the LBM finds which observations must remain unchanged between consecutive measurements to drive the ROM predictions to zero. The team also modified KernelSHAP to provide additional insights into the model predictions. In either method, feature contributions evolve over time within one individual; they also vary across different individuals. The VPICU aggregated and analyzed feature contributions in different ways to facilitate different levels of analysis of the RNN model and its predictions: (1) over a volatile time period within an individual patient’s predictions; (2) over populations of ICU patients sharing specific diagnoses; (3) across the general population of critically ill children.

The figure on the right illustrates the evolving ROM predictions of the RNN model for an individual diagnosed with pneumonia and acute respiratory distress syndrome (top panel), feature contributions to the evolving predictions (middle 2 panels, where the horizontal axis represents time), and the top contributing features to the RNN’s predictions during the patient’s volatile period (bottom panel). Despite the RNN model not using diagnoses as inputs, features related to the respiratory system (EtCO2, Pulse Oximetry, Respiratory Rate), or associated with infections (Temperature, Heart Rate, Systolic/Diastolic/Mean Arterial Blood Pressure, and blood gas measurements (ABG pH, ABG PO2, ABG PCO2, ABG HCO3, ABG TCO2) were identified.

Analyzing the Impact of Extraneous Features on Recurrent Neural Networks

Electronic Medical Records (EMR) are a rich source of patient information that can be used to generate predictions on a wide array of clinical tasks. Identifying which features are relevant in predicting different clinical outcomes can be challenging. The VPICU conducted a series of experiments, wherein an RNN’s predictive performance on various clinical modeling tasks was measured with and without the presence of extraneous features in the inputs. The extraneous features were randomly drawn from theoretical and empirical distributions.

The figure below displays that inclusion of extraneous features as inputs to the RNN had negligible effect on its ability to predict both mortality and ICU-free days. The results support the notion that RNNs can be trained without meticulous efforts spent on feature selection.

Responsive image


PICU Data Collaborative


The adoption of electronic medical record (EMR) systems has enabled researchers to apply a wide range of emergent machine and deep learning methodologies to the collected data with the goal of discovering and applying knowledge for better diagnosis, prognosis and patient treatment. At the recent AIMED conference in Laguna Nigel CA it was abundantly clear, as it has been to many investigators in health care big data research for years, that one of the major impediments to achieving the ‘big data’ promise of improved care for children, is the lack of sufficiently large datasets. This is particularly true in pediatric critical care. Although we capture and record large amounts of clinical data it is not widely available for researchers. This problem stems from: the lack of suitably large amounts of high quality data; and from the siloed nature of datasets used by various groups and studies, which make it difficult to make “apples-to-apples” comparisons. Furthermore the nature of pediatric critical care, wherein one unit may see a very limited number of certain diagnoses, makes this collaboration even more essential.

To address the problem, we propose a Pediatric Data Collaborative where:
  • Members contribute and share critical care EMR data
  • Data conform to a common schema to facilitate algorithm development, benchmarks and valid standard datasets for development
  • Members gather to share, discuss, and prioritize data aggregation, algorithm development, and clinical deployment


The PICU Data Collaborative will be comprised of institutions who contribute anonymized pediatric critical care EMR data to a shared data platform which resides in a private cloud-computing environment. Responsive image

Collaborative Technical Overview

Each Collaborative member will be responsible for collecting and aggregating data from its various sources based upon an agreed data schema and element list. The anonymization algorithm will be shared across members to ensure compliance and usability. Standard EMR data (including demographics, diagnoses, bedside and laboratory measurements, medications and procedures for each patient) would be aggregated during the initial phase with the subsequent potential to add additional data including notes, waveform data, imaging data and even genomic data at a later stage as the Collaborative matures. To facilitate new research and development, the data platform will enable automated or semi-automated methods converting the disparate data sets into a format that is easily digestible by data scientists. The platform will also provide artificial intelligence workflows and access to data science workspaces, empowering members to develop the science instead of wrangling with the technicalities of the data.

Common Schema and Benchmarks

Benchmark data sets historically have led to rapid gains in their associated fields. The most noteworthy example of this is the ImageNet database which is freely available and easily accessible to all researchers. Since 2010, annual challenges using this standardized database have led to rapid advances in computer vision capabilities. Well-curated medical datasets that are openly available for benchmarks are scarce, and this dearth makes the tracking of real progress in EMR algorithm development nearly impossible. The MIMIC-III database, which includes vital measurements, laboratory results, notes, fluid balance, procedure codes, diagnoses, imaging reports, among other data, is the most comprehensive and only freely accessible dataset of its kind. It is primarily an adult (16 years or above) critical care database with more than 38,000 patients, but it also contains almost 8,000 neonates. This database has enabled significant progress, but its single-institution nature places uncertainty on the transferability of algorithm developments. Discoveries made from the MIMIC database are rarely validated using databases from other institutions.

Our proposed rich collection of curated and standardized data can be used for benchmarking algorithm development geared towards improving pediatric critical care. The Collaborative could define tasks or problems akin to the ImageNet challenges. Regardless of the task or challenge, a common test set would be set aside to compare algorithms and track progress. Access to the data set will be controlled by the collaborative.

Responsive image
The initial dataset will consist of EMR data from PICU admissions restricted to vitals, lab values, vent settings, drugs, and MAR, of up to 175 individual data points mostly numeric structured data. Waveform, image and unstructured notes will not be included initially. For sites that also participate in VPS linkage to the clinical VPS data is also possible and will provide richer diagnostics, SOI and procedural data. A single data export encompassing as many years as possible retrospectively is initially sought from each collaborative participating member. It is hoped that this endeavor will provide over 100,000 PICU admissions in a static database for research purposes and to demonstrate the value of enhancing and enlarging the dataset.

Symposia for Collaborative Members

An important activity of the Collaborative would be an annual symposium, where researchers and clinicians gather to:

  • Discuss new exciting problems
  • Present results of new approaches on benchmarks
  • hare success stories of implementation
  • Talk about obstacles and how they can be overcome
  • Provide community of researchers working on common problems with common goals


    Interpreting a Recurrent Neural Network's Predictions of ICU Mortality Risk

    Long Van Ho, Aczon M., Ledbetter D., Gunny A. & Wetzel R.

    Continuous Prediction of Mortality in the PICU: A Recurrent Neural Network Model in a Single Center Dataset

    Melissa Aczon, Ledbetter D., Laksana E. & Wetzel R.

    Machine Learning to Predict Cardiac Death Within 1 Hour After Terminal Extubation

    Meredith Winter, Day T., Ledbetter D., Acson M., Newth C. & Ross P.

    Improving Recurrent Neural Network Responsiveness to Acute Clinical Events

    David Ledbetter, Laksana E., Aczon M. & Wetzel R.

    The Impact of Extraneous Features on the Performance of Recurrent Neural Network Models in Clinical Tasks

    Eugene Laksana, Aczon M., Ho L.H., Carlin C., Ledbetter D. & Wetzel R.

    Predicting Individual Responses to Vasoactive Medications in Children with Septic Shock

    Nicole Fronda, Asencio J., Carlin C., Ledbetter D., Aczon M., Wetzel R. & Markovitz B.

    Predicting Individual Physiologically Acceptable States for Discharge from a Pediatric Intensive Care Unit

    Cameron Carlin, Ho L.H., Ledbetter D., Aczon M. & Wetzel R.

    Applying Machine Learning to Pediatric Critical Care Data

    Jon B. Williams, Ghosh D. & Wetzel R.

    The Dependence of Machine Learning on Electronic Medical Record Quality

    Long Van Ho, Ledbetter D., Aczon M. & Wetzel R.

    Early Prediction Of Patient Deterioration Using Machine Learning Techniques With Time Series Data

    Sareen Shah, Ledbetter D., AczonM., Flynn A. & Rubin S.

    Deep Learning Recommendation of Treatment from Electronic Data

    Melissa Aczon, Ledbetter D., Ho L.V., Gunny A. & Wetzel R.

    Estimating Data Requirements to Detect Pediatric Critical Decompensation

    Melissa Aczon, Ledbetter D. & Shah S.

    Development of a deep learning model that predicts Bi-level positive airway pressure failure.

    Im DD, Laksana E, Ledbetter DR, Aczon MD, Khemani RG, Wetzel RC.


    Predicting High-Flow Nasal Cannula Failure in an ICU Using a Recurrent Neural Network with Transfer Learning and Input Data Perseveration: A Retrospective Analysis

    Pappy G, Aczon M, Wetzel R, Ledbetter D



    Responsive image

    Randall Wetzel, MBBS, MRCP, LRCS, MSB, FAAP, FCCM

    Dr. Randall Wetzel is the founder and Director of The Laura P. and Leland K. Whittier VPICU at Children’s Hospital Los Angeles. He is a tenured Professor of Pediatrics and Anesthesiology at the University of Southern California and has served as an attending physician in pediatric critical care for over 40 years both at Johns Hopkins Hospital and at Children's Hospital Los Angeles. Dr. Wetzel completed his undergraduate and medical degrees in the UK. He then trained in pediatrics at Case Western’s Rainbow Babies and Children’s Hospital, Critical Care Medicine at Johns Hopkins Hospital and subsequently Anesthesiology at Hopkins. He is board certified in Pediatrics, Pediatric Critical Care, Anesthesiology and Pediatric Anesthesiology. Additionally he received an MBA with a focus on information technology from Hopkins. Recently his research has involved data science applied to critical care medicine. He has served as the Director of Critical Care Medicine and as the Chairman of Anesthesiology Critical Care Medicine at CHLA. He has trained hundreds of residents and over 100 critical care fellows. He also founded Virtual Pediatric Systems which collects information of all admissions to over 160 pediatric ICUs in the United States.


    Dr. Melissa Aczon is a Principal Data Scientist at CHLA, where she works closely with pediatric ICU physicians to formulate clinically meaningful problems into data science problems, and develops algorithms to solve them using advanced computational techniques. She leverages her deep understanding of mathematics to solve signal processing, detection, classification and estimation problems from a wide array of applications. Prior to joining CHLA, Dr. Aczon was a Principal Scientist at Arete Associates, where she led a team of scientists to develop algorithms that both improve the detection capability and reduce false alarms of a very complex sensor system. She has worked with data coming from many different types of sensors including radar, optical and acoustic systems. She has a Bachelor of Science in Mathematics from Harvey Mudd College and a Ph.D. in Scientific Computing and Computational Mathematics from Stanford University.
    Dr. Flynn is a software development engineer with a Ph.D. in Computer Science and Informatics from Cardiff University. A member of the data systems team, she designs and develops data integration and manipulation processes to support the team’s applications.
    Ruiqi (Tina) Huang is a Data Scientist at CHLA. Together with an interdisciplinary team, she focuses on solving clinically meaningful problems by developing clinical support tools using state-of-the-art computational methods, computer science, and large scale EHR data. Her current works include developing a screening tool to identify children with pediatric sleep disorder using wearable technology, and a bedside evaluation tool of a child's condition in the ICU using real-time multimodal EHR data. Prior to joining CHLA, she was a graduate researcher under UCLA Medical Imaging Informatics striving to aid translational research by applying natural language processing (NLP) to summarize key findings in radiology reports and clinical texts. She holds a Master of Science and Bachelor of Science in Bioengineering (specialized in Biomedical Data Science), and Bachelor of Arts in Computer Science and Linguistics from University of California, Los Angeles.
    Eugene Laksana works as a Data Scientist at Children’s Hospital Los Angeles (CHLA) where he leverages deep learning techniques on 10 years of electronic health records (EHR) to develop algorithms for state-of-the-art clinical decision support. He is experienced with handling multimodal data and has worked on projects ranging from investigating temporal decay in EHR data value at CHLA to detecting suicide ideation using facial tracking software and machine learning at Carnegie Mellon University. Eugene holds. a B.Sc in Computer Science from the University of Southern California. He really likes capybaras.
    Ishmael Obeso is a Data Scientist currently working at Children's Hospital Los Angeles in the Virtual Pediatric ICU. His background is as a neuroscience researcher at UCLA, where he focused on computational neuroscience and brain-machine interfacing. He hopes to use his domain knowledge in neuroscience to further his work as a Data Scientist.
    Mike Reilly currently manages the cloud infrastructure and software supporting the data team of The Laura P. and Leland K. Whittier Virtual PICU at CHLA. Mike has an extensive background planning, deploying, and managing public and private cloud environments covering servers, storage, and application virtualization. His past work includes managing projects and creating solutions across a diverse background of industries including banking, entertainment, healthcare, and logistics. Mike holds a Bachelor’s degree in Business Administration with specialties in both Business Management and Information Systems.
    Boris Rubel is currently a Junior Data Scientist. He holds a background in Applied Mathematics from the University of California, San Diego and previously served as a sports-biomechanics researcher at CHLA’s Motion and Sports Analysis Laboratory where he analyzed lower extremity kinematic data using 3D motion capture systems. Having previously collaborated with CHLA’s physical therapists, Boris now works with pediatric critical care clinicians to quantify and support research questions through mathematical frameworks.
    Paul has been with the team since 2002. In that span he was the technical lead on two telemedicine grants and developed several web-based applications used in research studies. Currently, Paul serves as the VPICU’s Senior Program Manager. Paul’s main focus is managing the completion of all key objectives ensuring the success of The Laura P. and Leland K. Whittier VPICU.
    Alice Zhou is a Data Scientist at Children's Hospital Los Angeles (CHLA). She specializes in visualization, providing clinicians and data scientists alike with the tools for developing clinical decision support. Alice has worked on various projects at CHLA, such as researching acceptable cerebral perfusion pressures in kids and evaluating how different performance assessments methodologies estimate out-of-sample performance. She is a recent graduate of Wellesley College and holds a Bachelor's Degree in Media Arts and Sciences.

    Need more information?

    If you have any questions regarding The Laura P. and Leland K. Whittier Virtual PICU, please send us an email.