Interpreting meta-analysis output
aree meta-analyze pools effect sizes across studies for the same standardized feature, phenotype, and feature type. This page explains each output column and, critically, when a pooled number is a warning sign rather than a summary you should trust.
How pooling works
src/meta_analysis/run.py groups the evidence table by (feature_id_standardized, phenotype, feature_type, simulated, species_taxid, contrast_type) — never across feature types, demo/real origins, species or contrast types, since those are not comparable pools. Records with mapping_confidence == "unresolved" are excluded from pooling (no stable identity to group on) but remain visible in the raw evidence table. Within each group, effect sizes and standard errors are pooled with a random-effects (DerSimonian–Laird) estimator implemented directly in Python (src/meta_analysis/pooling.py) — see methods.md for the statistical detail.
Only records with a reported standard error, or enough information to derive one from an unadjusted p-value, enter the pool. All replication counts and summary context are calculated from that same poolable subset. When multiple source identifiers collapse to the same standardized identifier within one comparison, only a uniquely higher-confidence mapping is retained. If one study contributes multiple comparisons to the same group, AREE stops with an actionable error: without covariance information, those contrasts cannot be treated as independent observations. The curator’s way out is to set meta_analysis_primary: true on one comparison in the study YAML, chosen before looking at results. Only flagged comparisons from that study are then pooled; the others stay in the evidence table and on evidence cards, and are counted in n_excluded_non_primary. The choice is made within each contrast_type: comparisons of different contrast types never share a group, so a primary heat response does not exclude the same study’s baseline contrast (contrast_types.md).
A primary is itself pooled, so it cannot keep all of a study’s comparisons out of the pool. For a comparison that should never pool, such as a field contrast confounded with genetics, site or husbandry, set exclude_from_meta_analysis with a one-sentence reason instead:
exclude_from_meta_analysis:
reason: "Confounded field contrast: farm vs wild mixes disease state with genetics, site and husbandry."Opted-out comparisons stay in the evidence table and on evidence cards, are counted in n_excluded_opted_out, and never reach the within-study guard. A comparison cannot be both opted out and meta_analysis_primary: true; the schema rejects that. Like the primary flag, decide it before looking at results and record why in provenance.curation_notes.
Multiple testing
Every pooled p-value is adjusted with the Benjamini–Hochberg procedure within its test family: all features pooled for the same (phenotype, feature_type, simulated, species_taxid, contrast_type). A genome-wide reanalysis contributes one test per gene to that family — the first real OsHV-1 pool (CALLA2026_OSHV + DELISLE2020_OSHV_TEMP) has 23,094 — and at that scale 4,687 genes were nominally significant at p < 0.05 by chance and signal together. adjusted_p_value is the number to read; p_value is kept for transparency.
Both CLI filters (--phenotype, --feature-type) are family keys, so they select whole families, and a feature’s adjusted_p_value is identical whether you ran one phenotype or all of them. Filtering never shrinks the denominator.
The adjustment is per family, not across families: a 12-gene simulated demo pool and a 23,000-gene real pool are corrected against their own sizes.
Output columns
| column | meaning |
|---|---|
k_studies |
number of distinct study_id values contributing |
studies |
pipe-separated list of contributing study IDs |
n_evidence_records |
number of poolable evidence rows; equals k_studies under the current one-effect-per-study policy |
n_available_records |
all mapped records in the group, including records that lack a usable SE |
n_excluded_unpoolable / excluded_studies |
records/studies visible as evidence but absent from the numerical pool |
n_excluded_duplicate_mappings |
lower-confidence aliases removed when multiple source IDs resolve to one feature within a comparison |
n_excluded_non_primary |
records from comparisons their study did not mark meta_analysis_primary, left out of the pool |
n_excluded_opted_out |
records from comparisons marked exclude_from_meta_analysis, left out of every pool |
total_sample_size |
sum of sample_size across the poolable study-level effects |
pooled_effect |
random-effects pooled estimate of the effect size |
pooled_se |
standard error of the pooled effect |
ci_lower / ci_upper |
95% CI on the pooled effect (pooled_effect ± 1.96 × pooled_se) |
z, p_value |
test statistic and two-sided p-value for the pooled effect (computed with the normal survival function to avoid cancellation; sufficiently extreme tails can still underflow in floating-point arithmetic) |
adjusted_p_value |
Benjamini–Hochberg false-discovery-rate adjusted p_value, computed within the test family described below |
n_tests_in_family |
number of features tested in that family — the denominator the adjustment was made against |
q_statistic |
Cochran’s Q, the weighted sum of squared deviations from the fixed-effect estimate |
i_squared |
percentage of total variation attributable to between-study heterogeneity rather than sampling error |
tau_squared |
estimated between-study variance component |
direction_consistency |
fraction of contributing records agreeing with the majority effect-direction sign |
distinct_tissues, distinct_life_stages |
breadth of biological contexts represented |
distinct_stressors |
pipe-separated list of standardized stressors contributing — a list of names, not a count, unlike the two columns above it |
mapping_confidences |
union of mapping-confidence levels among contributing records |
quality_flags_union |
union of quality flags among contributing records |
Reading pooled_effect and ci_lower/ci_upper correctly
A pooled effect with a CI that excludes zero, few studies, and low heterogeneity is reasonable evidence of a consistent effect. A pooled effect alone, without checking i_squared and direction_consistency, can be actively misleading — averaging together a strong positive effect and a strong negative effect can produce a pooled estimate near zero that looks “unremarkable” instead of flagging a real conflict.
Worked example: a genuine conflicting-evidence case
Running the demo meta-analysis for larval_viability genes (reports/meta_analysis/larval_viability_gene_meta_analysis.tsv) produces this row for gene LOC105331241 (sod1, superoxide dismutase 1):
| field | value |
|---|---|
k_studies |
2 |
studies |
GIGAS_LARV05 | GIGAS_OA02 |
pooled_effect |
-0.191 |
pooled_se |
0.800 |
ci_lower / ci_upper |
-1.759 / 1.377 |
p_value |
0.811 |
adjusted_p_value |
0.811 (12 tests in the larval_viability/gene/simulated family) |
i_squared |
93.6% |
tau_squared |
1.198 |
direction_consistency |
0.5 |
mapping_confidences |
exact | inferred |
quality_flags_union |
identifier_mapping_uncertain | low_replication |
This is not “no effect.” It is two studies reporting opposite-direction effects for the same gene under the same phenotype (larval_viability): GIGAS_OA02 finds sod1 (as LOC105331241) strongly down-regulated under ocean acidification (log2FC ≈ -1.00, adjusted p = 0.03), while GIGAS_LARV05 finds it up-regulated under nutritional limitation (log2FC ≈ 0.60). This is intentional in the demo data — the curation note in registry/demo_studies/GIGAS_OA02.yaml states the study “deliberately includes an antioxidant-pathway gene (sod1) that is down-regulated here but up-regulated in GIGAS_LARV05 under the same larval_viability phenotype, to demonstrate how AREE surfaces conflicting same-phenotype cross-study evidence in meta-analysis (low direction_consistency) rather than averaging it away silently.”
The signature to recognize:
direction_consistency = 0.5withk_studies = 2means an exact 50/50 split — the weakest possible directional agreement above “no majority.”i_squared ≈ 93.6%means almost all of the variability across these two estimates is between-study heterogeneity, not sampling noise — the two studies are not measuring compatible effects.- The wide CI (
-1.76to1.38) spanning zero and both original study-level effect signs, combined with a non-significant pooledp_value(0.81), is the direct numerical consequence: pooling two opposite-direction effects returns “no discernible average effect,” which is a true statement about the pooled number but a misleading summary of the underlying evidence if read in isolation.
How to read this correctly: treat high i_squared (a common informal threshold is >75%) together with direction_consistency well below 1.0 as a signal that the two studies may be capturing genuinely different biology — different stressors (ocean_acidification vs. nutritional_limitation), different underlying mechanisms, or context-dependent regulation — not as noise to be smoothed over by the pooled estimate. This is exactly why the candidate-scoring framework does not let a low pooled p-value substitute for directional agreement: direction_consistency_score is weighted as heavily as n_studies_score (both 20 of 100 points), and heterogeneity is subtracted as an explicit penalty — see interpreting_candidate_scores.md. This candidate would score low on both counts and, separately, fails the high_priority_cross_study hard gate (direction_consistency >= 0.7) regardless of its overall score.
Cross-species results (--level orthogroup)
Gene-level pooling never crosses species. aree meta-analyze --level orthogroup does, through orthogroups, in two stages:
- Within species. Each species’ estimate for an orthogroup is its gene-level pooled estimate, unchanged, so every gene-level rule (primary comparisons, real standard errors, duplicate mappings) still holds.
- Across species. Those estimates are pooled with the same DerSimonian-Laird random-effects model. Its heterogeneity is now between species:
q_between_species,i_squared_between_species,tau_squared_between_species.
It writes <label>_orthogroup_meta_analysis.tsv (one row per orthogroup, phenotype and feature type) and <label>_orthogroup_species_estimates.tsv (the per-species rows behind each one).
What may pool:
- Single-copy orthogroups only, by default (
single_copy_all,single_copy). In a multi-copy orthogroup one species’ gene is compared with another species’ different paralog; those are counted and set aside.--include-multi-copypools them only where each species has evidence for exactly one gene, and the row keeps itsmulti_copylabel. - Orthogroups from the active run only. Evidence harmonized under an earlier run is set aside until it is re-harmonized.
- Simulated and real evidence never pool together, as at gene level. The demo’s illustrative orthogroups are classed
demo_crosswalk; their IDs collide with real OrthoFinder IDs, so they are never looked up in the run.
How to read a row:
k_studies_by_speciesandspecies_with_k2say how much each species contributes.meets_replication_gateis true only when at least two species each have k >= 2 studies. A row below the gate is a pairing of single-study estimates, not replicated cross-species evidence.direction_consistency_speciesis the share of species agreeing with the majority sign. With two species, 0.5 means they disagree.- A high
i_squared_between_speciesmeans the species respond differently; the pooled effect then describes no species well. adjusted_p_valueis Benjamini-Hochberg within each phenotype, feature type and origin family of orthogroup-level tests.
The first real cross-species rows came on 2026-10-16, once the first real clam study reached main: oyster and Manila clam thermal-tolerance evidence pooled through 5,681 single-copy orthogroups, none past the gate with one clam study. With the second clam study (2026-10-17) it pools through 5,975 (5,152 multi-copy orthogroups set aside), and 5,633 meet the replication gate. The demo adds oyster and mussel larval rows sharing demo orthogroups, also below the gate.
aree build-cross-species-cards tiers these rows and writes a card for each one that shows every species side by side; see interpreting_candidate_scores.md.