Suchergebnisse

ObjectiveTo evaluate the performance of a natural language processing (NLP) method to develop an automated crosswalk between the 9th and 10th revisions of the International Classification of Diseases (ICD) for diagnosis codes in the Charlson comorbidity index (CCI).
ApproachSBERT, an advanced NLP transformer-based model, was used to produce sentence embeddings, numeric vectors that represent the semantic meaning of text, for the labels (i.e., descriptors) of 932 ICD-10-CA (Canadian Adaptation) codes in the CCI (up to six digits). Sentence embeddings were also produced for all ICD-9-CM (Clinical Modification) code labels (15,145). Cosine similarity scores (CSS) were calculated for all possible pairs of ICD-10-CA and ICD-9-CM code labels. CSSs were classified as equivalent (CSS = 1), high (0.8 ≤ CSS < 1), and low (CSS < 0.8). CSS categories for CCI diagnosis codes were compared to an ICD-9-CM to ICD-10-CA crosswalk file manually created by the Canadian Institute of Health Information.
ResultsOf the 932 CSSs for ICD-10-CA codes in CCI, 84 (9%) were classified as equivalent, 284 (30.5%) were high, and 564 (60.5%) were low. For ICD-10-CA codes with low CSSs, the median was 0.67 (interquartile range 0.14).
Conclusions and ImplicationsAn ICD-10-CA to ICD-9-CM crosswalk based on NLP had low accuracy for identifying semantically similar diagnosis code labels. The accuracy of this method might be improved by fine-tuning and training on task-specific data. Evaluation of different text analysis-based models would provide guidance for research involving ICD code labels.

ABSTRACT
ObjectivesThe Public Health Agency of Canada (PHAC) established the Canadian Chronic Disease Surveillance System (CCDSS) in 2009 to facilitate national estimates of chronic disease prevalence, incidence, and health outcomes. The CCDSS uses population-based linked health administrative databases from all provinces/territories (P/Ts) and a distributed analytic protocol to produce standardized disease estimates. Our purpose is to describe the process, benefits, and challenges of implementing a distributed model for disease surveillance across thirteen jurisdictions with unique healthcare databases.
ApproachThe CCDSS is founded on deterministic linkage of three administrative health databases in each Canadian P/T: health insurance registration files, physician billing claims, and hospital discharge abstracts. Disease case definitions are developed by expert Working Groups after literature reviews are completed and validation studies are undertaken. Feasibility studies are initiated in selected P/Ts to identify challenges when implementing the disease case definitions. Analytic code developed by PHAC is then distributed to all P/Ts. Data quality surveys are routinely conducted to identify database characteristics that may bias disease estimates over time or across P/Ts or affect implementation of the analytic code. The summary data produced in each P/T are approved by Scientific Committee and Technical Committee members and then submitted to PHAC for further analysis and reporting.
ResultsNational surveillance or feasibility studies are currently ongoing for diabetes, hypertension, selected mental illnesses, chronic respiratory diseases, heart disease, neurological conditions, musculoskeletal conditions, and stroke. The advantages of the distributed analytic protocol are: (a) changes in methodology can be easily made, and (b) technical expertise to implement the methodology is not required in each P/T. Challenges in the use of the distributed analytic protocol are: (a) heterogeneity in healthcare databases across P/Ts and over time, (b) the requirement that each P/T use the minimum set of data elements common to all jurisdictions when producing disease estimates, and (c) balancing disclosure guidelines to ensure data confidentiality with comprehensive reporting. Additional challenges, which include incomplete data capture for some databases and poor measurement validity of disease diagnosis codes for some chronic conditions, must be continually addressed to ensure the scientific rigor of the CCDSS methodology.
ConclusionsThe CCDSS distributed analytic protocol offers one model for national chronic disease surveillance that has been successfully implemented and sustained by PHAC and its P/T partners. Many lessons have been learned about national chronic disease surveillance involving jurisdictions that are heterogeneous with respect to healthcare databases, expertise, and population characteristics.

The objective of this project is to implement a harmonized artificial intelligence (AI)-based de-identification of free-text medical data across multiple Canadian jurisdictions. This federated learning approach will allow these jurisdictions to leverage each other's data and resources while no individual-level data leaves the jurisdiction.
Federated Learning enables health data centers in different jurisdictions to collaborate in training machine learning models without sharing individual-level data. This approach will significantly reduce privacy and cybersecurity risks and barriers that are involved in sharing and moving data across different jurisdictions.
In a federated learning environment, machine learning models are trained on multiple data sources available in local data centers; local data are not shared to a central computing/analysis environment. Instead, parameters (such as model weights) are shared between these local data centers to generate a global model that will be shared and used by all participating data centers.
In this case study, four health research data centers in different Canadian provinces will take part in deployment of an AI-based application for de-identification of free-text data. The data centers are members of Health Data Research Network (HDRN) Canada. The deployment will include:

harmonized annotation and labeling of local data,
local training of entity recognition algorithms,
integrating model weights from each data centers to create a global model
development of license agreements between the participating data centers to allow sharing model weights

This is an ongoing project. The talk will demonstrate learning experiences, advantages, and challenges in a federated learning environment and explore the feasibility of transporting this approach to other multi-jurisdiction research networks.

IntroductionNotwithstanding Canada's exceptional longitudinal health data and research centres with extensive experience transforming data into knowledge, many Canadian studies based on linked administrative data have focused on a single province or territory. Health Data Research Network Canada (HDRN Canada), a new not-for-profit corporation, will bring together major national, provincial and territorial health data stewards from across Canada. HDRN Canada's first initiative is the $81 million SPOR Canadian Data Platform funded under the Canadian Institutes of Health Research Strategy for Patient-Oriented Research (SPOR).
Objectives and ApproachHDRN Canada is a distributed network through which individual data-holding centres work together to (i) create a single portal and support system for researchers requesting multi-jurisdictional data, (ii) harmonize and validate case definitions and key analytic variables across jurisdictions, (iii) expand the sources and types of data linkages, (iv) develop technological infrastructure to improve data access and collection, (v) create supports for advanced analytics and (vi) establish strong partnerships with patients, the public and with Indigenous communities. We will share our experiences and gather international feedback on our network and its goals from symposium participants.
ResultsIn January 2020, HDRN Canada launched its Data Access Support Hub (DASH) which includes an inventory listing over 380 datasets, information about more than 120 algorithms and a repository of requirements and processes for accessing data. HDRN Canada is receiving requests for multi-province research studies that would be challenging to conduct without HDRN Canada.
Conclusion / ImplicationsThus far, HDRN Canada services and tools have been developed primarily for Canadian researchers but HDRN Canada can also serve as a prompt for an international discussion about what has/has not worked in terms of multi-jurisdictional research data infrastructure. It can also present an opportunity for the development of metadata, standards and common approaches that support more multi-country research.

Introduction
Unstructured text data (UTD) are increasingly found in many databases that were never intended to be used for research, including electronic medical record (EMR) databases. Data quality can impact the usefulness of UTD for research. UTD are typically prepared for analysis (i.e., preprocessed) and analyzed using natural language processing (NLP) techniques. Different NLP methods are used to preprocess UTD and may affect data quality.

Objective
Our objective was to systematically document current research and practices about NLP preprocessing methods to describe or improve the quality of UTD, including UTD found in EMR databases.

Methods
A scoping review was undertaken of peer-reviewed studies published between December 2002 and January 2021. Scopus, Web of Science, ProQuest, and EBSCOhost were searched for literature relevant to the study objective. Information extracted from the studies included article characteristics (i.e., year of publication, journal discipline), data characteristics, types of preprocessing methods, and data quality topics. Study data were presented using a narrative synthesis.

Results
A total of 41 articles were included in the scoping review; over 50% were published between 2016 and 2021. Almost 20% of the articles were published in health science journals. Common preprocessing methods included removal of extraneous text elements such as stop words, punctuation, and numbers, word tokenization, and parts of speech tagging. Data quality topics for articles about EMR data included misspelled words, security (i.e., de-identification), word variability, sources of noise, quality of annotations, and ambiguity of abbreviations.

Conclusions
Multiple NLP techniques have been proposed to preprocess UTD, with some differences in techniques applied to EMR data. There are similarities in the data quality dimensions used to characterize structured data and UTD. While a few general-purpose measures of data quality that do not require external data; most of these focus on the measurement of noise.

IntroductionCapture of obesity using administrative health data is poor, with many cases being under coded within the data. Linking multiple health data sources may improve case ascertainment and facilitate the use of administrative health data for obesity research and surveillance.
Objectives and ApproachThis research aims to determine if using individual-level linked data from multiple sources can improve case ascertainment for obesity in administrative health data. Data from between April 1, 2001 and March 31, 2015 were obtained from the Manitoba Population Data Repository. Eighteen obesity case definitions were developed with different observation times and combinations of diagnosis, procedure, and prescription codes from physician billing claims, hospitalization abstracts, and prescription drug records. Body mass index (BMI) records from primary care data and the Bone Mineral Density (BMD) registry were used for validation. Sensitivity, specificity, and Cohen's kappa were calculated.
ResultsIndividuals with a higher BMI class had more physician visits and were more likely to have comorbidities and obese codes in the administrative health data. A higher BMI class was associated with being in a lower income quintile and the age group 40-59. Overall, the case definitions for obesity had high specificity (0.98-0.99) and low sensitivity (0.005-0.19) when validated using primary care data. Case definitions with obesity codes from multiple databases 3 year prior to and including the index date had the highest sensitivity (0.06-0.19) and kappa (0.04-0.23). Results with the BMD data were similar (specificity: 0.97-0.99; sensitivity: 0.007-0.21). Stratified analyses found agreement measures improved slightly for females, those who had chronic conditions and a later index year, and the age group 40-59.
Conclusion / ImplicationsWhen using multiple databases to build a case definition for obesity, sensitivity improves but remains low. Individuals with other chronic conditions and a higher BMI class were more likely to be accurately classified as an obese case.

BackgroundThe International Classification of Diseases (ICD) is revised over time and there are region-specific versions, including ICD-10-CA (Canada) and ICD-9-CM (USA). Studies spanning multiple ICD versions require crosswalks to translate diagnosis codes across versions, but manual crosswalk development is costly and requires clinical expertise.
ObjectiveTo evaluate the accuracy of a pre-trained large language model (LLM) to automatically translate chronic disease diagnosis codes from ICD-10-CA to ICD-9-CM.
ApproachEight prompts were developed to instruct the OpenAI Generative Pre-trained Transformer 4 (GPT-4) LLM to translate 1,272 ICD-10-CA codes for the Elixhauser Comorbidity Index to ICD-9-CM. Prompt accuracy (%) was measured against a crosswalk developed by the Canadian Institute of Health Information. Variability was assessed by replicating each prompt three times. Mean accuracy ± standard deviation was reported for each prompt across replications, for both five-digit and truncated three-digit codes.
ResultsThe highest prompt performance was observed when assigning a persona of a medical coding specialist (40.8% ± 0.9%), requesting justification for the selected code (41.4% ± 1.1%), and providing diagnosis code labels (47.5% ± 0.7%). For truncated three-digit codes, these prompts achieved accuracy of 82.0% ± 0.5%, 80.8% ± 0.9%, and 82.7% ± 0.1%, respectively. Combining these three prompting techniques marginally improved accuracy to 48.6% ± 0.7% for five-digit codes and 84.3% ± 0.2% for truncated three-digit codes.
ConclusionGeneral-purpose LLMs are currently not sufficiently accurate at automating ICD code translation for chronic diseases.
ImplicationsAdditional experiments with fine-tuning, task-specific training, and prompt engineering are needed to improve accuracy and reduce variability.

ABSTRACT
ObjectivesValidation studies compare diagnostic information in linked administrative and reference (i.e., gold standard) data; they are an essential tool to develop accurate case definitions, the rules used to identify individuals in administrative data with a specific health condition. Validation studies often estimate the accuracy of multiple case definitions, in order to identify the data features (e.g., diagnosis codes, type of data source) that influence accuracy estimates. Descriptive analyses are commonly used to select a case definition(s) with the greatest accuracy estimates, but fail to account for uncertainty in accuracy estimates. The objectives were to: (1) compare the performance of regression-based approaches to test for differences in diagnostic accuracy estimates, and (2) demonstrate how to apply and use these models.
ApproachComputer simulation was used to compare three regression models: (a) univariate fixed-effects models applied to estimates of sensitivity and specificity; (b) univariate fixed-effects model for Youden's index, the average of sensitivity and the complement of specificity; and (c) bivariate random-effects joint model of sensitivity and specificity. The simulations varied the means and variances of sensitivity and specificity, the correlation between these parameters, and the number of case definitions. Performance was compared using: (a) bias (i.e., difference between estimated and observed mean), (b) mean squared error (MSE), the sum of the estimated variance and bias squared, and (c) 95% confidence interval (CI) coverage, the proportion of times the population mean is contained in the 95% CI. For objective 2, we applied the models to estimates of diagnostic accuracy from a published rheumatoid arthritis (RA) validation study with 61 case definitions.
ResultsUnivariate models of sensitivity and specificity had lower bias than the bivariate model (e.g., univariate=1.8%, bivariate=2.2%). The bivariate model had a smaller MSE than the univariate models when sample size was large and there was a small correlation between sensitivity and specificity (e.g., univariate=3.4%, bivariate=2.6%). Across all scenarios, the univariate model for Youden's index showed small bias (average=2.4%) and MSE (average=2.1%). For objective 2, the univariate models of sensitivity, specificity, and Youden's index revealed multiple case definition features that were associated with estimates of RA diagnostic accuracy: 1+ diagnosis in hospital records, >1 diagnosis in physician claims, and 1+ diagnoses by a specialist physician.
ConclusionsWe recommend the bivariate model when a validation study contains a large number of case definitions. When the data contain a small number of case definitions, univariate models are recommended.

Introduction: When designing prospective and retrospective longitudinal cohort studies, investigators must make decisions about study duration (i.e. length of follow-up) and frequency of outcome measurement. The impact of these decisions have been previously investigated in the prospective setting, but have not been described for retrospective cohort studies.

Objectives: To examine the impact and potential challenges of longitudinal design decisions in retrospective cohort studies and illustrate the effects of varying study duration and frequency of outcome measurement in the retrospective setting using a numeric example.

Methods: Linked administrative data from Manitoba was used. The cohort included all mothers who experienced the death of an infant between April 1, 1999 and March 31, 2012 and a matched (3:1) group of mothers who did not experience death. A generalized linear mixed model was used to model differences in the trend in the number of healthcare contacts for the two groups. Holding sample size constant, the model was fit to the data for combinations of duration and frequency. Estimated standard errors and regression coefficients were compared.

Results: A total of 2576 mothers were included; 644 experienced death of an infant and 1932 were matched to this group. Thirteen combinations of frequency (1, 2, 3, 4 periods/year) and duration (1, 2, 3, 4 years) were compared. As frequency increased from 1 to 4 periods/year, the standard error of the group-time-time interaction decreased up to 98.9%. As duration increased from 1 to 4 years, the standard error of the interaction decreased up to 96.9%. As frequency and duration increased, the coefficients trended toward zero.

Conclusions: Retrospective designs using administrative data offer greater flexibility to select time-related design elements than prospective designs, but present potential new challenges. Recommendations about how to select and report time-related design decisions in retrospective cohort studies should be included in reporting guidelines.

IntroductionPublic housing is a form of subsidized housing that is owned and/or managed by government. Previous research suggests that public housing has a positive impact on personal finances and education outcomes, but less is known about if/how it impacts health and healthcare use.
Objectives and ApproachUsing linked administrative health and social data, we tested for changes in healthcare use among a cohort who moved into public housing in 2012 and 2013 in Manitoba, Canada, and compared utilization to a matched general population cohort who did not move into public housing. Generalized linear models with generalized estimating equations tested for differences in numbers of healthcare contacts in the years before and after the move-in date, adjusted for economic, residential mobility, and health characteristics. The data were modeled using a Poisson (rate ratio, RR), negative binomial (incident rate ratio, IRR), or a binomial (odds ratio, OR) distribution.
ResultsThere were 2619 residents in the public housing cohort; 99.7% were matched to the general population. The cohort by time interaction was statistically significant for inpatient days (p
Conclusion/ImplicationsPublic housing residents were more likely to use healthcare services than the matched population, but changes in use were similar in the two cohorts. There is little evidence that public housing impacts healthcare use, but it serves an important function of meeting basic needs for a vulnerable population group.

IntroductionAlthough RCTs remain the gold standard for generating clinical evidence, follow up of participants to study long-term effects is limited by cost and other logistical considerations. Linkage of participant information to routinely collected data potentially offers a cost-effective solution to achieving long-term follow-up of treatment effects
Objectives and ApproachThis scoping review aimed to identify RCTs that had been extended by record linkage, and characterize these in terms of nationality, numbers of trials, disease areas and outcomes, types of data, linkage modes and duration of follow-up. We followed published guidelines for the conduct of scoping reviews, with a registered protocol and comprehensive literature search. Criterion-based selection of studies and extraction of date were performed in duplicate. Descriptive statistics were used to summarise the characteristics of eligible studies.
ResultsOne hundred thirteen RCTs had been extended by record linkage. Fifty-six were conducted in Nordic countries, 26 in the USA and 24 in the UK. Types of linkage data used were: vital statistics 36, adimistrative data 31, cancer registry 28, special registries 13 and others 11. The literature spanned 45 years, but 66 (58%) were published between 2010 and 2016. Linkage methods were reported as: deterministic 33, probabilistic 16 and unspecified 64. In 44 studies researchers reported ethics approval for linkage; this was not obtained in 39 cases and was absent in 30 reports. The overall follow up times achieved by record linkage were: 1-4 y (6 studies), 5-9y (34), 10-19y (48), 20-29y (21), 30-39y(4) and over 50y (1).
Conclusion/ImplicationsAlthough we uncovered over 100 RCTs that were extended by record linkage this is tiny compared with the number of trials that have been undertaken. Linkage to routinely collected data seems to be a feasible but under-used approach to extending the follow-up of clinical trial participants for very long periods.

ObjectiveAdministrative data often lacks complete family linkages, particularly to fathers, hindering familial health research. We compared methods to address missing paternal linkages in research investigating the familial transfer of mental disorders.
ApproachA population-based cohort study of Manitoba (Canada) born adults, +18 years old between 1977 and 2017 with maternal linkages. Three methods were used to address missing paternal linkages: indicator category, complete case, and multiple imputation, which identified three candidate fathers for each individual within -4 to +7 years of mother's age with same postal code. For each method, the association of maternal and paternal history with mental disorder risk during follow-up was tested using multivariable logistic regression models adjusted for demographics and comorbidities.
ResultsThe cohort included 142,549 individuals; 22.6% lacked paternal linkages. Using indicator category, maternal and paternal histories were associated with mental disorder risk, with odds ratio (OR) 1.49, 95% confidence interval (CI): 1.45-1.52 and OR 1.34, 95% CI: 1.30-1.37, respectively. Similar results were obtained in the complete case analysis (OR 1.49, 95% CI: 1.45-1.53 and OR 1.33, 95% CI: 1.30-1.37, for maternal and paternal history, respectively). With multiple imputation, maternal history's association with mental disorder risk remained consistent with the other methods (pooled OR 1.52, 95% CI: 1.49-1.56); paternal history was associated with a smaller risk (pooled OR 1.25 95% CI: 1.22-1.29).
Conclusions and ImplicationsThe three methods for addressing missing paternal linkages produced similar findings about the familial transfer of mental disorders. Future familial studies could incorporate multiple methods to demonstrate robustness of findings.

BackgroundThe Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) is a standardized structure for observational data that enables collaborative health research. Health Data Research Network (HDRN Canada) is conducting a pilot study to implement this standard at partner data linkage centers.
ObjectiveOur objective is to describe the process used to implement the OMOP CDM, methodological and technical challenges, and the approach to evaluate CDM implementation at multiple sites.
ApproachCDM working group members include technical staff from data linkage centers in four provinces and a national data coordinating center, HDRN Canada operational and executive leads, and OMOP CDM consultants. The evaluation aims to collect quantitative and qualitative information on Extract, Transform, and Load (ETL) processes, performance of the CDM in a multi-site observational case study, and data quality.
ResultsInitial steps have included collaborative learning about the CDM structure, importing open-source tools and licensed coding standards (i.e., Systematized Nomenclature of Medicine-Clinical Terms [SNOMED-CT], RxNorm) at data linkage centres, and exploring the technical requirements for developing ETL processes and hosting OMOP data. Creation of select OMOP tables has been accompanied by discussions about models for mapping clinical and prescription drug coding standards. A scoping review of evaluation strategies is underway.
Conclusions and ImplicationsThe benefits of standardizing health data for multi-site research and supporting the production of findable, accessible, interoperable and reusable (FAIR) national data underscores the importance of HDRN Canada embarking on this OMOP CDM implementation project. The lessons learned will benefit other multi-site data standardization initiatives.

Objectives We developed a machine-learning model-based algorithm (MBA) for smoking in Administrative Health Data (AHD). The validity of this MBA was compared to a rule-based algorithm (RBA).
Approaches The study included adults (≥18 years) from a clinical registry containing self-reported current smoking from 2017 to 2020 in Manitoba, Canada. Clinical data were linked to up to five years of hospitalization, physician billing claims, and prescription medication records. The RBA was based on diagnosis codes for tobacco use and nicotine dependence medication. MBAs, constructed using random forest (RF) models, included these indicators in addition to comorbid condition diagnoses and sociodemographic factors. Sensitivity, specificity, positive and negative predictive values (PPV, NPV), and 95% confidence intervals (CIs) were estimated.
Results The cohort comprised 24,718 individuals (88.6% female); prevalence of current smokers was 10.0%. The RBA had sensitivity of 27.3% (95% CI: 24.2-30.7), specificity of 96.6% (95% CI: 96.1-97.0), and PPV of 47.2% (95% CI: 42.9-51.5). The MBA had sensitivity of 68.6% (95% CI: 65.1-71.9), specificity of 76.3% (95% CI: 75.2-77.3), and PPV of 24.3% (95% CI: 23.2-25.6). NPV was high irrespective of algorithms. Stratified analyses revealed similar estimates for males and females, and the number of years of AHD did not affect the MBA results.
ConclusionsAn RF-based MBA for smoking ascertainment in linked AHD sources improved sensitivity compared to the RBA. However, the RBA excelled in specificity and PPV.
ImplicationBalancing accurate smoker identification with the risk of false positives is crucial when choosing an algorithm to ascertain current smokers using AHD.

IntroductionThe validity of chronic disease case definitions for administrative health data may change over time due to changes in data quality. Trend control charts to identify out-of-control (OOC; i.e., unexpected) observations in a time series may indicate where disease estimates are influenced by changes in data quality.
ObjectiveApply and compare trend control charts methods for multiple sclerosis (MS) incidence and prevalence estimates using previously-validated case definitions for Manitoba, Canada.
MethodsEight case definitions were identified from published literature and applied to Manitoba administrative health data from January 1, 1972 to December 31, 2018. Incidence and prevalence trends were modeled using negative binomial and generalized estimating equation models, respectively. Trend control charts were used to plot predicted case counts against observed case counts. Control limits to identify OOC observations were calculated using two methods: predicted case count ±0.8*standard deviation (0.8*SD) and predicted case count ±2*standard deviation (2*SD). Differences in proportion of OOC observations across case definitions was assessed using McNemar's test.
ResultsThe proportion of OOC observations ranged from 0.71 to 0.90 for incidence and 0.72 to 0.98 for prevalence when using the 0.8*SD control limits. A lower proportion of OOC observations (0.46 to 0.74 for incidence; 0.30 to 0.74 for prevalence) was observed for the 2*SD control limits. Neither method resulted in significant differences in OOC observations across case definitions.
ConclusionsThe proportion of OOC observations in trend control charts varied with the control limit method adopted, but statistical significance did not. Trend control charts are a potentially useful tool for developing surveillance methods, but may benefit from disease-specific calibrated control limits.