Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future. Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.© 2025. The Author(s).

Aberystwyth University
Eastern Illinois University
Austral University of Chile
Vrije Universiteit Amsterdam
Max Planck Inst Biol Intelligence
CABI
Nature Conservancy
Inst Avian Res Vogelwarte Helgoland
University of Hong Kong
Southern Illinois University Carbondale
University of Tasmania
International Institute for Applied Systems Analysis
North Carolina State University
University of Alberta
University of Plymouth
University of Glasgow
University of Edinburgh
Iowa State University
University of Bath
University of Stirling
University of Western Australia
Brigham Young University
University of Oxford
McMaster University
University of Gothenburg
University of Cambridge
Universite de Montpellier
Universität Wien
University of Toronto
University of Aberdeen
Aarhus university
KU Leuven
LLC Catbird Stats
Rhodes Coll
Universite de Pau et des Pays de l'Adour
Elon University
University of Brighton
Univ Sorbonne Paris Nord
European Forest Inst
Western Norway University of Applied Sciences
University of Regina
Off Natl Stat
Rochester Institute of Technology
Inst Ecol & Biodiversidad
Institut für Interdisziplinäre Gebirgsforschung
St Norbert Coll
Indian Institute of Science (IISC) - Bangalore
Whitebark Inst
Whitman College
Universidade Federal da Fronteira Sul
Lake Superior State University
Drexel University
CHU de Nancy
Wesleyan University
Wilfrid Laurier University
Universidad Rey Juan Carlos
Universidade Federal de Goias
Publ Hlth Scotland
Ordu University
Lansstyrelsen Ostergotland
University of Southern Mississippi
Wofford Coll
Nomad Ecol
Oyo State Coll Agr & Technol
University of South Wales
Vive Crop Protect
Univ South Carolina
University of Rome Tor Vergata
Dept Primary Ind Fisheries
Fram Project AS
Arthur Rylah Institute for Environmental Research (ARI)
Bath Spa University
Saint Olaf College
Boise State University
Missouri State University
Memorial University Newfoundland
Federation University Australia
Ministry of Energy & Natural Resources - Turkey
Grand Bay Natl Estuarine Res Reserve
Bar Ilan University
Royal Society for Protection of Birds
University of Maine System
University of Bristol
University of Connecticut
Technical University of Darmstadt
University Antananarivo
Ben-Gurion University of the Negev
Universidad de la Republica, Uruguay
King Abdulaziz University
Czech University of Life Sciences Prague
BirdLife International
Georg-August-Universität Göttingen
Deakin University
University of Southern Queensland
Virginia Commonwealth University
Universidade Federal de Santa Catarina (UFSC)
Macquarie University
University of New South Wales Sydney
Universidad de Valladolid
University of Waterloo
Universidad de los Andes (Colombia)
Virginia Polytechnic Institute & State University
University of Hull
University of Arkansas System
IRTA
State University of New York (SUNY) System
Linkoping University
Swedish University of Agricultural Sciences
University of Arizona
McGill University
VOLCANI INSTITUTE OF AGRICULTURAL RESEARCH
Universidade Federal do Rio de Janeiro
Universite Paris-Est-Creteil-Val-de-Marne (UPEC)
Trent University
University of Iceland
Montana State University System
University of Quebec
University of Groningen
Rutgers University System
University of Rhode Island
Purdue University System
University of Wyoming
Monash University
Northern Arizona University
California State University System
Clemson University
Wake Forest University
University of Konstanz
Yale University
Liverpool John Moores University
University of Michigan System
University of Queensland
Tufts University
University of Bergen
University System of Maryland
University of Massachusetts System
University of Wisconsin System
Jagiellonian University
Ulm University
British Trust for Ornithology
University of Exeter
Utah System of Higher Education
University of Antwerp
United States Department of Agriculture (USDA)
Indiana University System
University of Texas System
University of Auckland
Bangor University
MRC Biostatistics Unit
Agence Nationale de Securite Sanitaire de l'Alimentation, de l'Environnement du Travail (ANSES)
University College Dublin
Consejo Superior de Investigaciones Cientificas (CSIC)
Rhodes University
Technische Universitat Dresden
University of Munster
Oregon State University
City University of New York (CUNY) System
Universite Catholique Louvain
Universidade de Lisboa
University of Cape Town
University of Wurzburg
Lund University
Norwegian University of Science & Technology (NTNU)
Complutense University of Madrid
University of Zurich
Leibniz Association
University of Turku
Universite de Montreal
Umea University
University of Jyvaskyla
University of Bielefeld
Centre National de la Recherche Scientifique (CNRS)
Imperial College London
Colorado State University System
CIRAD
University of Minnesota System
University of Melbourne
University of Bayreuth
Michigan State University
University of California System
University of Colorado System
Karolinska Institutet
National University of La Plata
University Hohenheim
University of Birmingham
Helmholtz Association
Duke University
University of St Andrews
Consiglio Nazionale delle Ricerche (CNR)
University of Saskatchewan
University of Neuchatel
State University System of Florida
University System of Ohio
University of Bern
University of Tennessee System
University of Helsinki
University of London
Cornell University
University of Sydney
Harvard University
University of Guelph