Interpreting candidate biomarker scores
AREE assigns every candidate (one row per pooled feature/phenotype/feature-type group) a numeric score and a tier. This page explains both — and, more importantly, explains that the tier is decided by hard gates checked before the score, not by the score itself. A high score cannot promote a weakly-supported candidate into a strong tier.
Candidate identity also includes simulated and species_taxid. Assay diversity, mapping confidence, quality flags, forest data, and card filenames are calculated within that partition, so a demo record or another species can never promote or overwrite a real single-species candidate.
The score: a transparent weighted sum
src/prioritize/scoring.py computes a 0-1 value for each named component and combines them as a plain weighted sum, minus a heterogeneity penalty — there is no hidden model or machine-learning step. Weights sum to 100 across the positive components:
| component | weight | what it measures | source |
|---|---|---|---|
n_studies_score |
20 | independent replication (k / (k + 5)) |
count of distinct study_id |
direction_consistency_score |
20 | agreement in effect direction across studies | meta-analysis direction_consistency |
sample_size_score |
10 | total biological replication (n / (n + 100)) |
sum of sample_size |
effect_magnitude_score |
10 | absolute pooled effect (\|e\| / (\|e\| + 2)) |
pooled effect_size |
significance_score |
10 | adjusted-significance strength (s / (s + 5) where s = -log10(q)) |
pooled adjusted_p_value |
phenotype_relevance_score |
10 | resilience (1.0) > disease (0.8) > stress_response (0.4) > exposure_only (0.1) | phenotype’s resilience_relevance |
assay_diversity_score |
10 | molecular layers with a significant record for this feature under this phenotype (n_supporting_layers / 3, capped) |
feature_type → molecular_layer, adjusted_p_value |
context_breadth_score |
5 | spread across tissues × life stages (c / (c + 3)) |
distinct tissue × life_stage |
mapping_confidence_score |
3 | trust in identifier harmonization (worst mapping among contributing records) | mapping_confidence |
quality_score |
2 | study/data quality (1 - n_quality_flags/5, floored at 0) |
quality_flags |
Every quality flag counts the same in quality_score: batch_confounded (treatment fully confounded with a sequencing run, MS acquisition batch, or extraction batch) removes the same 0.2 as low_replication, and counts toward the quality_score >= 0.4 high-priority gate the same way. Adding the term (2026-10-16) did not change any existing score, because no registered study declared it yet. Once a study declares it, every candidate with a record from that study loses 0.2 quality_score (0.4 points of the 0-100 score) unless the flag was already in its union.
Why the unbounded components saturate instead of clipping
Every component whose input has no natural upper bound uses v / (v + half), which is strictly increasing everywhere, equals 0.5 at half, and approaches but never reaches 1.0. Each half above is the value that used to be a hard cap, so the ordering is unchanged and only the ceiling is gone.
This replaced min(1, v / cap) on 2026-09-05, because clipping broke the ranking exactly where it matters. On the first genome-wide real pool, |pooled_effect| clipped at 2 and adjusted p clipped at 1e-5; 126 of the 1,994 high-priority candidates therefore scored an identical 68.57 despite pooled effects spanning 2 to 9.7 log2FC and adjusted p spanning 1e-5 to 1e-75. A reader choosing validation targets from the top of that list was choosing between ties. After the change the largest tie group is 10, and those are rounding collisions at two decimal places, not saturation.
The components with genuinely bounded inputs still clip: direction_consistency and quality_score are 0-1 by construction, and assay_diversity_score is capped by the number of molecular layers in the feature_types vocabulary.
Absolute scores are lower than before this change — nothing reaches a component value of 1.0 any more. Compare candidates within one run, not across this change.
Separately, a heterogeneity penalty of up to 15 points is subtracted, scaled by i_squared / 100 — high cross-study heterogeneity directly reduces the final score rather than being ignored.
score = sum(weight[c] * component[c] for c in components) - 15 * (i_squared / 100)
clipped to [0, 100]. The function (candidate_score in scoring.py) is pure — identical inputs always produce the identical score — which is what makes candidate-score reproducibility directly testable.
Notice mapping_confidence_score and quality_score are weighted quite lightly (3 and 2 points respectively) relative to n_studies_score and direction_consistency_score (20 each). The score alone will not sink a candidate just for imperfect identifier mapping or a couple of quality flags — which is exactly why those two properties are instead enforced as hard gates, not left to the weighted sum, for the top tier (see below).
The tiers, and the gates that decide them
src/prioritize/rank.py computes tier membership from hard boolean conditions, evaluated independently of the numeric score:
is_high_priority = (
k_studies >= 2
and phenotype_relevance_score > 0.1
and direction_consistency >= 0.7
and quality_score >= 0.4
and adjusted_p_value <= 0.05
)
is_multi_omics = own_layer in supporting_layers and n_supporting_layers >= 2Tier assignment, in order:
high_priority_cross_study— requires all of:k_studies >= 2(at least two independent studies)phenotype_relevance_score > 0.1(notexposure_only— see resilience_vs_exposure.md)direction_consistency >= 0.7quality_score >= 0.4(at most 3 distinct quality flags under the current scoring formula)adjusted_p_value <= 0.05— the pooled effect survives Benjamini–Hochberg control within its phenotype / feature-type family (see interpreting_meta_analysis.md). Without this gate, two genome-wide studies promote every gene whose effects happen to share a sign: on the first real OsHV-1 pool that was 12,658 of 23,094 genes, 8,533 of them with pooledp > 0.05.
multi_omics_convergence— at least two molecular layers (transcriptomics,dna_methylation,proteomics,metabolomics, as declared per feature type inregistry/controlled_vocabularies/feature_types.yaml) each carry a significant record (study-level adjusted p ≤ 0.05) for the same standardized feature under the same phenotype, and the candidate’s own layer is one of them. Checked only if the candidate did not already qualify forhigh_priority_cross_study.Three things this deliberately does not count: a DMR and a single-CpG DML for one gene (two views of one methylation layer); a gene expressed under heat and methylated under pathogen challenge (two questions, not convergent evidence for either); and two other layers converging on a gene the candidate itself shows no signal for. Before 2026-09-05 the gate was “≥ 2 distinct feature types anywhere for this gene”, which promoted 27 of the 48 simulated demo candidates; under the current gate none of them qualifies, because every cross-layer overlap in the demo is cross-phenotype. The card still lists that adjacent evidence and marks each row with whether it counts.
metabolite_linked_convergence— the candidate’s own layer is significant, and another layer carries a significant same-phenotype record for a feature linked to it biochemically rather than by a shared ID: a metabolite and a gene whose product KEGG annotates (by orthology) to catalyse a reaction involving that metabolite (enzyme), or a transporter (KEGG BRITE ko02000) curated as carrying it (transporter, alwaysinferred). Checked only if neither tier above applies. This is a deliberately weaker, separately named kind of convergence: it does not add ton_supporting_layersor to the score; currency metabolites (ATP, NAD⁺, …) and hub compounds reaching more than 25 genes are never credited; and every link isinferredorone_to_many_ortholog, neverexact. The card lists each link’s compound, reaction, KO, confidence and source checksum, plus the exclusions. Candidates carryconvergence_basis(shared_identifier/metabolite_gene_link/none),linked_layers,linked_features,link_confidencesandlink_table_status. See metabolite_gene_links.md.emerging— everything else: single-study support, or support that fails one or more of the high-priority gates. Always labeled as requiring replication.
A high score cannot override these gates. A candidate with k_studies = 1 that happens to have a huge effect size, tiny p-value, and perfect direction consistency (trivially 1.0 with only one study) can still score numerically high on the weighted sum — but it is placed in emerging, not high_priority_cross_study, because it fails the k_studies >= 2 gate. Likewise, the conflicting-evidence example in interpreting_meta_analysis.md (LOC105331241/sod1, direction_consistency = 0.5) fails the direction_consistency >= 0.7 gate regardless of its score and cannot reach high_priority_cross_study.
This is the concrete mechanism (see design.md section 6) that prevents “statistically significant in one study” from reading as “validated”: the score is informative for ranking within a tier, but tier membership itself is gated on replication, directional agreement, phenotype relevance, quality, and family-wise significance — properties a single striking p-value cannot substitute for. significance_score itself is also computed from adjusted_p_value, not the raw pooled p.
Evidence class: the tier answers a different question
The tier says how well replicated a signal is. It does not say what the signal is evidence of — and a reader who sees high_priority_cross_study will reasonably assume “resilience biomarker”. For the first real pool that assumption is wrong for every candidate.
Each candidate therefore carries two more columns, computed in src/prioritize/rank.py and printed at the top of its evidence card:
| column | values | meaning |
|---|---|---|
evidence_class |
resilience_associated, disease_associated, stress_response, exposure_only |
the most conservative comparison-level resilience_classification among contributing records; ontology default only when absent |
context_replication |
single_study, single_context, multi_context |
whether the contributing studies vary in stressor, tissue, or life stage — replication of the biology, as distinct from replication of the study |
In the initial two-study OsHV-1 pool (September 2026), the selected contrasts measured disease_susceptibility: animals were challenged with the virus, without an individual survival outcome used in those contrasts. Its 1,994 top-tier candidates were therefore disease_associated. Subsequent CALLA2026 viral read counts are QC measurements, not a survival phenotype. These historical counts do not describe the current expanded pool; see the dashboard. The selected contrasts provide evidence of an infection response. Whether any of them distinguishes an animal that survives from one that does not is a question these data cannot answer, and the card says so rather than leaving the tier to imply otherwise.
context_replication is multi_context for that pool, but only because the two studies used different life stages (spat vs. juvenile); the stressor and tissue are identical. The card names the dimensions that vary and the ones that do not, instead of asserting a generality the pool has not established.
Because the phenotype gap outranks the replication gap, a card whose evidence_class is not resilience_associated leads its recommended next step with closing that gap — measuring the feature in animals whose outcome was recorded — and only then gives the tier-appropriate validation advice. Confirming an expression difference that was never tied to an outcome does not turn it into a biomarker.
Contrast type: predictive baseline candidates
Evidence class says what was measured; contrast_type says between whom and when. A resilience_associated candidate can rest on a difference between resilient and susceptible animals under unstressed conditions (baseline_phenotype_difference) or on how animals respond to the stress. Only the first could be read before any stress, which is what a breeder needs, so those candidates carry is_predictive_baseline = True and get their own section at the top of aree top-candidates. Baseline and response evidence never pool together. The flag changes neither tier nor score; for a baseline candidate, replication means independent resistant-vs-susceptible group pairs. See contrast_types.md.
Reading the output
aree build-evidence-cards ranks every candidate and writes two things under reports/evidence_cards/:
candidates.tsv— every ranked candidate, all tiers, with the fullcomponent_*breakdown, the meta-analysis fields, and acard_filecolumn. This is the complete, auditable ranking;emergingcandidates are not hidden, they are labeled.- One markdown card per candidate that carries a significant signal, plus an
index.jsonlisting those cards. A candidate qualifies when its pooledadjusted_p_valueis at or below--max-adjusted-p(default 0.05), or when at least one contributing study reported an adjusted p at or below it. The second clause is what keeps conflicting-direction candidates on a card:LOC105331241/sod1pools top = 0.81because two studies disagree, but one of them reportedpadj = 0.03, and that conflict is exactly what a reader needs to see.
The threshold exists because a genome-wide pool yields one candidate per gene. Rendering ~30,000 cards for genes with no signal anywhere took tens of minutes and buried the reader; the first real OsHV-1 pool renders a few thousand instead. Pass --all-cards to render every candidate regardless.
Each card shows the tier, the contributing studies, the pooled and BH-adjusted p-values with the family size, and a forest-style table of per-study effect estimates with each study’s own adjusted p, so a reviewer can see exactly which inputs drove the score rather than trusting a single number.
Unpooled candidates (pooling_status = unpooled)
Pooling needs a standard error, either reported or backed out of an unadjusted p-value. Some sources publish neither: typically an author-curated, significance-filtered DEG list with only log2FC and padj (HESSER2024_VCOR). AREE does not back a standard error out of padj, and it does not pool such records. A significance-selected subset overstates effects and inflates apparent replication. Those records still never disappear:
- When a feature also has poolable evidence for the same phenotype, its card names the excluded records (
Records left out of the pool…) and lists them in the forest table withpooled = no. They carry no weight. - When a feature’s evidence for a phenotype is entirely unpoolable, it gets its own
candidates.tsvrow withtier = emerging,pooling_status = unpooled,k_studies = 0and no score. These rows sort after every scored candidate, and their card says the effect is a reported, selection-biased value rather than an estimate.
The fix is upstream: obtain the full result table from the authors, or reanalyze the raw reads, so the study can be pooled like any other.
Cross-species candidates
aree build-cross-species-cards tiers the orthogroup-level meta-analysis (see interpreting_meta_analysis.md): one row per orthogroup, phenotype and feature type, pooling each species’ own gene-level estimate. The rule is the same as for single-species candidates: hard gates first, score second. It writes, under reports/evidence_cards/cross_species/, a candidates.tsv with every row and its component breakdown, an index.json, and a card for each row whose pooled adjusted p, or any one species’ gene-level adjusted p, is at or below --max-adjusted-p.
The five convergence gates, each named in the gates_failed column when it fails:
| Gate | Passes when |
|---|---|
replication |
at least two species each have k ≥ 2 studies of their own |
direction |
every species’ estimate has the same sign |
significance |
the pooled cross-species BH-adjusted p ≤ 0.05 |
phenotype_relevance |
the weakest evidence class across all species’ contributing records is more than exposure-only |
quality |
each species’ own quality flags are within the gene-level limit (the weakest species decides) |
The tiers:
| Tier | Meaning |
|---|---|
cross_species_convergence |
every gate passes |
cross_species_emerging |
species agree in sign and the pooled estimate is significant, but another gate fails, usually replication |
cross_species_discordant |
species that are each significant at gene level disagree in sign; shown, never dropped |
cross_species_unsupported |
the pooled estimate is not significant, or species disagree in sign only because one of them is noise |
The direction gate is judged on point estimates, so any sign disagreement fails it. Only a disagreement between species that are each significant at gene level is called discordant. Two null estimates disagree in sign about half the time: on the first real oyster-clam pool, 2,880 of 2,886 sign disagreements involved a non-significant species, and only 6 were reversals between significant ones.
The cross-species score uses its own weights, which sum to 100:
| Component | Weight | What it measures |
|---|---|---|
per_species_replication_score |
20 | the smallest per-species study count, saturating at 2 = 0.5 |
n_species_score |
15 | number of species, saturating at 2 = 0.5 |
direction_consistency_score |
15 | share of species agreeing with the majority sign |
effect_magnitude_score |
10 | absolute pooled cross-species effect |
significance_score |
10 | pooled BH-adjusted p, on the −log10 scale |
phenotype_relevance_score |
10 | weakest evidence class across all species’ records |
orthology_score |
8 | copy class: single_copy_all 1.0, single_copy and demo_crosswalk 0.9, multi_copy 0.3 |
sample_size_score |
5 | total biological replicates across species |
mapping_confidence_score |
4 | worst identifier mapping behind any species’ estimate |
quality_score |
3 | the weakest species’ own gene-level quality (1 - n_flags/5) |
Between-species I² is subtracted as a penalty of up to 15 points. The smallest per-species study count carries the most weight because a cross-species claim is only as strong as its weakest species.
A cross-species card shows each species’ gene, original identifiers, studies, tissue, life stage, stressor, sample size, effect with 95% CI and gene-level adjusted p side by side, then the pooled estimate and between-species heterogeneity. It names the orthogroup run, or says the orthogroup is the demo crosswalk’s illustrative one. It also warns when the species’ studies differ in stressor, tissue or life stage, because species and study design are then confounded. Agreement across species shows an association recurs in orthologous genes. It does not show the genes do the same thing in each species.
On the demo, oyster and mussel larval studies give five emerging rows and three unsupported rows. None converges, because the demo mussel has one study per gene. The demo’s one sign disagreement involves a non-significant mussel estimate, so it is unsupported, not discordant.
On real data (2026-10-17), the three harmonized oyster thermal-tolerance studies and the two Manila clam heat studies (TUCCI2025_HEAT_PRIMING, PERUZZA2023_HEATWAVE) pool through 5,975 single-copy orthogroups: 133 emerging, 5,842 unsupported, none discordant or convergent. The replication gate is met in 5,633 rows. 131 emerging rows pass replication, direction and significance and fail only phenotype relevance (both clam comparisons are exposure-only) and quality (each species’ own flags exceed the gene-level limit, as they do for its single-species candidates). With one clam study (2026-10-16) the same pool had 5,681 rows, 64 emerging and 6 discordant; the discordant rows did not survive the second clam study.
On the dashboard. The home page shows cross-species tiers, rows and cards only for a phenotype past the WP6 gate: at least two species each with a within-species gene pool of k >= 2. A row for a gated phenotype is counted as withheld, not ranked. Every top single-species candidate also lists its orthologs in other species, the phenotypes they were tested under and how many of their records were significant. That lookup pools nothing, so it is shown whether or not the gate is open.