Suchergebnisse

Background/Objective: The Padua Inventory-Revised (PI-R) is a widely applied instrument to measure obsessive-compulsive symptoms in clinical and nonclinical samples. We conducted a reliability generalization meta-analysis on the PI-R. Method: An exhaustive literature search yielded 118 empirical studies that had applied the PI-R, from which 30 studies (33 samples) reported an original reliability estimate. Results: Assuming a random-effects model, the average internal consistency reliability (Cronbach's alpha) was .92 (95% CI [.91, .93]) for the total scores, and ranged from .74 to .89 for the subscales. Assuming mixed-effects models, moderator analyses showed a positive statistically significant association between the standard deviation of the total scores and the reliability coefficients (p = .002; R2 = .38). Conclusions: In terms of reliability, the PI-R scale was found to be adequate for both research and clinical purposes, although exhibiting large heterogeneity across studies. Future empirical studies using the PI-R should be required to provide at least one reliability estimate based on their own data. ; Antecedentes/Objetivo: El Padua Inventory-Revised (PI-R) es un instrumento ampliamente utilizado para medir los síntomas obsesivo-compulsivos en muestras clínicas y no clínicas. Llevamos a cabo un meta-análisis de generalización de la fiabilidad del PI-R. Método: Una búsqueda exhaustiva de la literatura arrojó 118 estudios empíricos que habían aplicado el PI-R, de los cuales 30 estudios (33 muestras) reportaron una estimación propia de la fiabilidad. Resultados: Asumiendo un modelo de efectos aleatorios, la fiabilidad en términos de consistencia interna promedio (alfa de Cronbach) fue de 0,92 (IC del 95% [0,91, 0,93]) para las puntuaciones totales, y osciló entre 0,74 y 0,89 para las subescalas. Asumiendo modelos de efectos mixtos, los análisis de moderadores mostraron una relación positiva estadísticamente significativa entre la desviación típica de las puntuaciones totales y los coeficientes de fiabilidad (p = 0,002; R2 = 0,38). Conclusiones: En términos de fiabilidad, se encontró que el PI-R es adecuado tanto para fines clínicos como de investigación, aunque con una alta heterogeneidad entre los estudios. Es necesario que los estudios empíricos futuros que apliquen el PI-R proporcionen al menos una estimación de la fiabilidad basada en sus propios datos. ; This research was funded by Agencia Estatal de Investigación (Spanish Government) and by FEDER funds, AEI/10.13039/501100011033, projects n° PID2019-104033GA-I00 and PID2019-104080GB-I00.

Reliability generalization (RG) is a meta‐analytic approach that aims to characterize how reliability estimates from the same test vary across different applications of the instrument. With this purpose RG meta‐analyses typically focus on a particular test and intend to obtain an overall reliability of test scores and to investigate how the composition and variability of the samples affect reliability. Although several guidelines have been proposed in the meta‐analytic literature to help authors improve the reporting quality of meta‐analyses, none of them were devised for RG meta‐analyses. The purpose of this investigation was to develop REGEMA (REliability GEneralization Meta‐Analysis), a 30‐item checklist (plus a flow chart) adapted to the specific issues that the reporting of an RG meta‐analysis must take into account. Based on previous checklists and guidelines proposed in the meta‐analytic arena, a first version was elaborated by applying the nominal group methodology. The resulting instrument was submitted to a list of independent meta‐analysis experts and, after discussion, the final version of the REGEMA checklist was reached. In a pilot study, four pairs of coders applied REGEMA to a random sample of 40 RG meta‐analyses in Psychology, and results showed satisfactory inter‐coder reliability. REGEMA can be used by: (a) meta‐analysts conducting or reporting an RG meta‐analysis and aiming to improve its reporting quality; (b) consumers of RG meta‐analyses who want to make informed critical appraisals of their reporting quality, and (c) reviewers and editors of journals who are considering submissions where an RG meta‐analysis was reported for potential publication. ; Funding for this study was provided by the Ministerio de Economía y Competitividad of the Spanish Government and by funds from the Fondo Europeo de Desarrollo Regional (FEDER; Project No. PSI2016- 77676-P).

1. Background A reliability generalization (RG) meta-analysis is a special type of psychometric meta-analysis that aims to integrate reliability coefficients obtained when a given test is applied to different primary studies, to examine how reliability of a test scores varies from an application to the next. Its purpose is to estimate the average reliability of a test scores, to investigate whether reliability can be generalized to different contexts, situations, and target populations, and in case of heterogeneity, to identify study characteristics that might be statistically associated to reliability coefficients (e.g., mean and SD of tests cores, target population, test version, etc.). Current checklists to guide the reporting of meta-analyses are not adequate for RG meta-analyses: PRISMA checklist (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; Moher et al., 2009) AMSTAR checklist for reporting meta-analyses on intervention efficacy (Grimshaw, Wells, et al., 2007) MOOSE checklist for reporting meta-analyses of observational studies (Stroup, Berlin, Morton et al., 2000) MARS guidelines for reporting meta-analyses on intervention efficacy (APA Publications and Communications Board Working Group on Journal Article Reporting Standards, 2008) Due to the absence in the literature of guidelines specifically devised to inform researchers and consumers of RG meta-analyses on how to conduct and report an RG meta-analysis, we developed the REGEMA checklist, a tool to guide researchers to report RG meta-analyses fulfilling the transparency and reproducibility principles of the Open Science. 1.1 Objectives The main objectives of this research were: (a) To examine the inter-coder reliability of the REGEMA checklist; (b) to investigate the capacity of REGEMA checklist to assess the reporting quality of RG meta-analyses, and (c) to assess the degree of compliance of REGEMA checklist by RG meta-analyses. 2. Method 2.1 Selection criteria of the studies To be included in this systematic review the meta-analyses had to fulfil the following selection criteria: (a) RG meta-analysis reporting reliability estimates of one or several tests to assess psychological constructs; (b) it had to be carried out between 1998 and 2019; (c) it had to include all primary studies that applied the test/s of interest (meta-analyses of only psychometric studies were excluded); (d) in case of focusing on more than one test, it had to report separate reliability estimates of each one of them, and (e) to be written in English or Spanish. 2.2 Searching for the studies The following databases were consulted: PsycINFO, Web of Science, and Google Scholar, using the key words in the title: 'reliability generalization', 'meta-analysis AND reliability', 'meta-analysis AND internal consistency', 'meta-analysis AND test-retest', 'meta-analysis AND (interrater OR intrarater)', 'meta-analysis AND (alpha coefficient OR coefficient alpha)', 'meta-analysis AND intraclass'. A total of 150 RG meta-analyses (19 unpublished and 131 published RG meta-analyses) were identified and included in this systematic review. 2.3 Data extraction REGEMA checklist was applied to each RG meta-analysis. Other characteristics registered were the publication source, the year of the meta-analysis, and the idiom. Inter-coder reliability was assessed with two coders that independently extracted the data from the RG meta-analyses and applied the REGEMA checklist. Inconsistencies were resolved by consensus. 2.4 Statistical analysis To estimate the inter-coder reliability of REGEMA checklist, Cohen's kappa coefficient and inter-coder agreement percentage were calculated for each item and subitem of the checklist. To assess the degree of compliance of each item of the REGEMA checklist, the compliance percentage from all the RG meta-analyses analyzed was calculated. 3. Results REGEMA checklist exhibited a very satisfactory degree of inter-coder reliability. The degree of compliance of REGEMA by RG meta-analyses is deficient in many of their items (< 50% of compliance), especially in: (a) Background: defining the psychological construct assessed; (b) determining the selection criteria of the studies; (c) the reliability analysis of data extraction data from the studies, and (d) how heterogeneity among reliability coefficients was assessed. 4. Conclusions and implications REGEMA checklist is easy to be applied and enables to assess the reporting quality of RG meta-analyses. REGEMA checklist can be useful for: (a) Researchers interested in conducting an RG meta-analysis; (b) Potential readers of RG meta-analyses, as a tool to make a critical reading of them, and (c) Editors of scientific journals that publish RG meta-analyses, as a guide for an adequate reporting of this type of meta-analyses. * Funding This research was funded by a grant from the Ministerio de Ciencia e Innovación of the Spanish Government and by FEDER funds (Project nº PID2019-104080GB-I00). ; unknown ; unknown