Each number below is documented exactly as the pipeline implements it,
not simplified for the reader. The weights and thresholds are pulled
live from analysis/scoring.py and
analysis/nlp_extraction.py when this page renders, so it
cannot quietly drift out of sync with what the code actually computes.
Every claim is labelled asserted or uncertain by looking for one specific list of hedging words.
A claim is one sentence containing a subject and a verb. Each one is tagged asserted or uncertain by a single check: does the sentence contain any of these words or phrases, matched whole-word and case-insensitively?
reportedly, stored as this claim's
certainty_reason
→ uncertain
The full list, live from the code:
certainty_reason,
which means every label traces back to the exact phrase that caused it
instead of being a black-box judgment.
One formula produces every "X% uncertain" figure on the site. Only the set of claims being counted changes.
Count the claims in one article and you get that article's figure. Count them across a topic, a story or a publisher and you get that one instead. It appears as marker colour on the maps, as density-over-time on topic pages, and as one of the three inputs to the publisher trust score below.
Three observable red-flag rates, each normalised to 0 to 1, combined by weight and subtracted from 100.
This is not a truth or fact-check score. It measures how often a publisher shows three specific warning signs, nothing more.
Only reporting sources are scored. Each of the three signals below is a claim about a newsroom's record with its own reporting, so they are not computed for sources that publish commentary. A newsletter or a signed column is meant to argue a position, and counting how often it hedges a claim measures the form of writing rather than the care taken over a fact. Every source is marked as reporting or commentary in the registry, by hand, and the publisher pages for commentary sources say so in place of a score.
| Signal | Weight | What it measures |
|---|---|---|
| correction_rate | 0.4 | How often this publisher self-corrects, as correction_count over article_count. A 15% rate counts as fully maxed out. |
| uncertainty_density | 0.25 | The hedged-claim rate above, across all of this publisher's claims. A 50% rate counts as fully maxed out. |
| assertive_mismatch_rate | 0.35 | Of the articles that did get corrected, how many had been framed assertively, meaning stated with confidence and then walked back. |
| Signal | Rate | ÷ maxes out at | Normalised | × weight | Penalty |
|---|---|---|---|---|---|
| correction_rate | 0.03 | 0.15 | 0.2 | 0.4 | 0.08 |
| uncertainty_density | 0.12 | 0.5 | 0.24 | 0.25 | 0.06 |
| assertive_mismatch_rate | 0.5 | 1.0 | 0.5 | 0.35 | 0.175 |
| total penalty | 0.315 | ||||
python scoring.py trust <publisher_id> checks
a single one directly.
An article counts as a correction if its opening contains one of these keywords.
correction_count
The title plus the first 500 characters are searched for any of the following, matched case-insensitively as substrings:
One local LLM call per article, rating how the piece is written rather than what it is about.
The model scores word choice, framing and story selection. It is explicitly not scoring the political leaning of the topic or of the people the article covers.
Most articles get no score at all. The first thing the model is asked is whether the article takes a side on a contested political question. A match report, a transfer, a rescue, an obituary or a product launch does not, however warmly or critically it is written, so it is recorded as not politically coded and no number is produced. Approval of a person is not right-wing and sympathy for a victim is not left-wing.
The model also has to name the contested question — the actual political argument the article comes down on one side of. If it cannot finish the sentence "this article takes the side of ___ in the argument about ___", the verdict is thrown out and the article is recorded as not politically coded. A subject being serious, sad, or merely involving the government is not a political question.
Where there is a score, the model must first quote the article's own words that carry the lean, say which way each one points, and only then give the number. Every quote is checked against the article text before it is stored, and any that cannot be found there is discarded. Words spoken by someone the article quotes are that person's words, not the article's, and do not count as evidence on their own.
A score that cannot survive its own evidence is not published. If none of the quotes can be found in the article, or if every quote that can be found points the opposite way from the number, the article is left without a leaning rather than shown one that its own working contradicts. What the model said is kept in the audit trail either way, so the rate is something you can go and look at.
/admin/reasoning, and the article's own page shows it.
If Ollama cannot be reached the score is left unset rather than
guessed, so it fails open. That is a different state from not
politically coded, and the two are stored apart.
How much one source's leaning swings from article to article, across its whole output.
Shown on the Story Map sidebar's second chart. It is computed over every article the source has published, not only the ones belonging to the story you clicked.
+0.30 +0.34 +0.28 +0.32 +0.31
-0.55 +0.62 +0.10 -0.30 +0.75
An article is labelled speculative, mixed or assertive purely by what fraction of its claims are uncertain.
The inputs are the certainty labels from section 01, so this adds no new judgment of its own. It only summarises the ones already made.
A separate measure of loaded or persuasive language, independent of whether the underlying claim is true.
Every sentence is scanned for terms belonging to five weighted categories:
| Category | Weight | Example terms |
|---|---|---|
| certainty_booster | 1.0 | clearly, obviously, undeniably, undoubtedly, … |
| absolutist | 0.6 | always, never, every, everyone, … |
| emotional_charge | 1.4 | shocking, outrageous, alarming, devastating, … |
| credibility_loaded_verb | 1.2 | claims, claimed, insists, insisted, … |
| attribution_verb | 0.15 | alleges, alleged, denies, denied, … |
clearly, weight 1.0)
and one absolutist term
(always, weight 0.6):
reporter_intensity_score. The reason is that loaded
language inside an accurately quoted subject's own words reflects what
that person said, not how the reporter framed it. Standard attribution
verbs such as "alleged", "denied" and "accused" carry a near-zero
weight, because they are required vocabulary for reporting contested
claims responsibly rather than signs of editorialising.
A coarse structural check that only catches the genres a source marks itself.
Whether an article needs a content note, its picture held back, or an age confirmation. Judged on what the article describes, not on whether its subject is upsetting.
Articles are grouped only when an LLM judges them to describe the same specific incident, not merely a related one.
same_event_links, so re-running the
pipeline only pays the LLM cost for genuinely new pairs. Every article
ends up in exactly one story: an article with no confirmed match
becomes its own single-article story, so nothing is silently dropped.