Compare commits
7
Commits
72b32dc69b
...
7ce8c65939
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7ce8c65939 | ||
|
|
41c144bf20 | ||
|
|
8365b7093e | ||
|
|
b66a8eefa5 | ||
|
|
a52f4123dd | ||
|
|
20991fedfc | ||
|
|
0212e407e5 |
No files matched your search
@@ -497,7 +497,7 @@ Computed in `src/curate/signals.rs` after embeddings. Every signal is `Option<f6
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `interest` | z-scored standing-interest match (§9.1) | no embedding, or fewer than 30 eligible articles have embeddings (then use raw top-1 cosine and log it) |
|
| `interest` | z-scored standing-interest match (§9.1) | no embedding, or fewer than 30 eligible articles have embeddings (then use raw top-1 cosine and log it) |
|
||||||
| `knn` | signed rated-neighbour preference (§9.2) | no embedding, or gate closed |
|
| `knn` | signed rated-neighbour preference (§9.2) | no embedding, or gate closed |
|
||||||
| `feed` | mean Beta-smoothed rating rate over the article's distinct direct feeds (§9.3) | no direct feed with any rating, or gate closed |
|
| `feed` | mean Beta-smoothed rating rate over the article's distinct direct feeds and author (§9.3) | no direct feed or author with any rating, or gate closed |
|
||||||
| `social` | existing `composite_social_score` | no `social` rows |
|
| `social` | existing `composite_social_score` | no `social` rows |
|
||||||
| `heuristic` | `longform_points(word_count)` − excerpt-only penalty − roundup penalty, from `prefilter.rs` with the social, Scour/HN, multi-source, and feed-prior terms **removed** | never |
|
| `heuristic` | `longform_points(word_count)` − excerpt-only penalty − roundup penalty, from `prefilter.rs` with the social, Scour/HN, multi-source, and feed-prior terms **removed** | never |
|
||||||
|
|
||||||
@@ -531,7 +531,9 @@ preference: 14 rated articles with embeddings → knn gate 0.35; feed gate 0.0 (
|
|||||||
|
|
||||||
### 9.3 Feed affinity
|
### 9.3 Feed affinity
|
||||||
|
|
||||||
From the same rating set. Credit each rating's `value` to the article's distinct direct feeds (`SourceKind::Feed`), split evenly; if there are none, to `best_entry_id`'s feed. Per feed: `rate = (up + 1) / (up + down + 2)` where `up = Σ max(value, 0)` and `down = Σ max(−value, 0)`, decayed as in §9.2. A candidate's `feed` signal is the **mean** over its distinct direct feeds that have any rating (never the max). Gate: `feed_floor = 15`, `feed_full = 40` attributable ratings.
|
From the same rating set. Credit each rating's decayed `value` to the article's distinct direct feeds (`SourceKind::Feed`), split evenly; if there are none, use `best_entry_id`'s feed. For an aggregator-only article, that fallback aggregator feed receives only `0.25 ×` the feed credit. When an article has an author, its normalized author key (trimmed, internal whitespace collapsed, lowercased) separately receives the full credit, so future articles by that author carry the history across any feed.
|
||||||
|
|
||||||
|
Per feed and author: `rate = (up + 1) / (up + down + 2)` where `up = Σ max(value × decay, 0)` and `down = Σ max(−value × decay, 0)`. A candidate's `feed` signal is the **mean** over its distinct direct feeds and author that have any rating (never the max). A rating is attributable if it credits at least one feed or author. Gate: `feed_floor = 15`, `feed_full = 40` attributable ratings.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -987,7 +989,7 @@ No test touches the network. Mock backends for all three providers, following th
|
|||||||
- **Rating events**: latest explicit event wins; `cleared` removes an article from the learned set; migration copies old rows with the right labels and values; the CLI `set`/`clear` append rows with `source = 'cli'`.
|
- **Rating events**: latest explicit event wins; `cleared` removes an article from the learned set; migration copies old rows with the right labels and values; the CLI `set`/`clear` append rows with `source = 'cli'`.
|
||||||
- **Embeddings**: BLOB round trip; wrong length and non-finite rejected; cache hit on same hash, miss on changed text/model/dimension; response mapped by index and length-checked; a failed batch does not abort the others; embedded text contains no feed name or author.
|
- **Embeddings**: BLOB round trip; wrong length and non-finite rejected; cache hit on same hash, miss on changed text/model/dimension; response mapped by index and length-checked; a failed batch does not abort the others; embedded text contains no feed name or author.
|
||||||
- **Interest z-scores**: a broad interest with uniformly high cosine does not dominate; a specific interest with one strong match does; raw fallback under 30 articles.
|
- **Interest z-scores**: a broad interest with uniformly high cosine does not dominate; a specific interest with one strong match does; raw fallback under 30 articles.
|
||||||
- **Preference**: one loved article gives a positive `knn` to a near neighbour; two unrelated loved clusters both score high (the anti-centroid test); `good` moves the signal 0.35× as much as `loved`; decay halves at the half-life; gate is 0 below `knn_floor`, 1 at `knn_full`, linear between; feed credit sums to 1 across direct feeds; feed affinity uses the mean.
|
- **Preference**: one loved article gives a positive `knn` to a near neighbour; two unrelated loved clusters both score high (the anti-centroid test); `good` moves the signal 0.35× as much as `loved`; decay halves at the half-life; gate is 0 below `knn_floor`, 1 at `knn_full`, linear between; ordinary feed credit sums to 1 across direct feeds; aggregator-only credit is 0.25× to its feed and 1× to its author; feed affinity uses the mean of rated feeds and author.
|
||||||
- **Normalization**: a constant signal normalizes to 0.5 for everyone; ties get equal percentiles (400 identical zeros → all 0.5, no id ramp); absent values do not shift others; effective weights sum to 1; a candidate missing a signal is scored on the rest.
|
- **Normalization**: a constant signal normalizes to 0.5 for everyone; ties get equal percentiles (400 identical zeros → all 0.5, no id ramp); absent values do not shift others; effective weights sum to 1; a candidate missing a signal is scored on the rest.
|
||||||
- **Triage and deep parsing**: realistic fixtures; malformed items do not sink a batch; unknown facet tokens degrade to `None`; every enum token in both prompts round-trips; cached assessments are reused within `assessment_reuse_days` and ignored with `--rescore`.
|
- **Triage and deep parsing**: realistic fixtures; malformed items do not sink a batch; unknown facet tokens degrade to `None`; every enum token in both prompts round-trips; cached assessments are reused within `assessment_reuse_days` and ignored with `--rescore`.
|
||||||
- **Admission**: a strong-interest, weak-heuristic, no-social article reaches the deep set; a 60-word stub with high interest similarity is not admitted by `interest`/`knn`; quotas honoured; inactive retrievers release quota; exploration deterministic per date; auto-includes always admitted; excluded articles get thin rows with the right reason.
|
- **Admission**: a strong-interest, weak-heuristic, no-social article reaches the deep set; a 60-word stub with high interest similarity is not admitted by `interest`/`knn`; quotas honoured; inactive retrievers release quota; exploration deterministic per date; auto-includes always admitted; excluded articles get thin rows with the right reason.
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
-- Persist page-extracted authors independently of the best Miniflux entry.
|
||||||
|
|
||||||
|
ALTER TABLE articles ADD COLUMN author TEXT;
|
||||||
+139
-20
@@ -23,6 +23,9 @@ pub const INTEREST_ZSCORE_MIN_ARTICLES: usize = 30;
|
|||||||
const ZSCORE_STD_FLOOR: f64 = 1e-3;
|
const ZSCORE_STD_FLOOR: f64 = 1e-3;
|
||||||
/// How many interests and rated neighbours `signals_json` records (§7.5).
|
/// How many interests and rated neighbours `signals_json` records (§7.5).
|
||||||
const RECORDED_TOP: usize = 3;
|
const RECORDED_TOP: usize = 3;
|
||||||
|
/// Aggregators carried the link rather than authored the article, so they get
|
||||||
|
/// only a small share of an aggregator-only article's feed-affinity credit.
|
||||||
|
pub const AGGREGATOR_FEED_SHARE: f64 = 0.25;
|
||||||
|
|
||||||
/// The signal names that go through the percentile normalizer, in the order
|
/// The signal names that go through the percentile normalizer, in the order
|
||||||
/// they are rendered (§12.2). LLM scores (`triage`, `quality`, `fit`) are
|
/// they are rendered (§12.2). LLM scores (`triage`, `quality`, `fit`) are
|
||||||
@@ -126,6 +129,10 @@ pub struct RatedExample {
|
|||||||
pub embedding: Vec<f32>,
|
pub embedding: Vec<f32>,
|
||||||
/// Distinct direct feeds that carried the rated article (§9.3).
|
/// Distinct direct feeds that carried the rated article (§9.3).
|
||||||
pub feeds: Vec<FeedId>,
|
pub feeds: Vec<FeedId>,
|
||||||
|
/// Whitespace-normalized, lowercase author key (§9.3).
|
||||||
|
pub author: Option<String>,
|
||||||
|
/// Whether the article arrived only through link aggregators (§9.3).
|
||||||
|
pub aggregator_only: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl RatedExample {
|
impl RatedExample {
|
||||||
@@ -153,6 +160,7 @@ impl FeedRate {
|
|||||||
pub struct PreferenceState {
|
pub struct PreferenceState {
|
||||||
pub examples: Vec<RatedExample>,
|
pub examples: Vec<RatedExample>,
|
||||||
feed_rates: HashMap<FeedId, FeedRate>,
|
feed_rates: HashMap<FeedId, FeedRate>,
|
||||||
|
author_rates: HashMap<String, FeedRate>,
|
||||||
pub attributable_feed_ratings: usize,
|
pub attributable_feed_ratings: usize,
|
||||||
pub knn_gate: f64,
|
pub knn_gate: f64,
|
||||||
pub feed_gate: f64,
|
pub feed_gate: f64,
|
||||||
@@ -160,8 +168,11 @@ pub struct PreferenceState {
|
|||||||
|
|
||||||
impl PreferenceState {
|
impl PreferenceState {
|
||||||
/// Build the state from already-loaded examples (pure; tests use this).
|
/// Build the state from already-loaded examples (pure; tests use this).
|
||||||
pub fn build(examples: Vec<RatedExample>, ranking: &RankingConfig) -> Self {
|
pub fn build(mut examples: Vec<RatedExample>, ranking: &RankingConfig) -> Self {
|
||||||
let (feed_rates, attributable_feed_ratings) = feed_rates(&examples);
|
for example in &mut examples {
|
||||||
|
example.author = normalize_author(example.author.as_deref());
|
||||||
|
}
|
||||||
|
let (feed_rates, author_rates, attributable_feed_ratings) = feed_rates(&examples);
|
||||||
Self {
|
Self {
|
||||||
knn_gate: gate(examples.len(), ranking.knn_floor, ranking.knn_full),
|
knn_gate: gate(examples.len(), ranking.knn_floor, ranking.knn_full),
|
||||||
feed_gate: gate(
|
feed_gate: gate(
|
||||||
@@ -171,6 +182,7 @@ impl PreferenceState {
|
|||||||
),
|
),
|
||||||
examples,
|
examples,
|
||||||
feed_rates,
|
feed_rates,
|
||||||
|
author_rates,
|
||||||
attributable_feed_ratings,
|
attributable_feed_ratings,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -194,12 +206,14 @@ impl PreferenceState {
|
|||||||
let Some(embedding) = embeddings.get(&rating.article_id).cloned() else {
|
let Some(embedding) = embeddings.get(&rating.article_id).cloned() else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
let feeds = db
|
let article = db.get_article(rating.article_id).await?;
|
||||||
.get_article(rating.article_id)
|
let feeds = article.as_ref().map(direct_feeds).unwrap_or_default();
|
||||||
.await?
|
let author = article
|
||||||
.as_ref()
|
.as_ref()
|
||||||
.map(direct_feeds)
|
.and_then(|article| normalize_author(article.author.as_deref()));
|
||||||
.unwrap_or_default();
|
let aggregator_only = article
|
||||||
|
.as_ref()
|
||||||
|
.is_some_and(crate::discovery::aggregator_only);
|
||||||
let age_days = (now.as_second() - rating.event_at.as_second()).max(0) as f64 / 86_400.0;
|
let age_days = (now.as_second() - rating.event_at.as_second()).max(0) as f64 / 86_400.0;
|
||||||
examples.push(RatedExample {
|
examples.push(RatedExample {
|
||||||
article_id: rating.article_id,
|
article_id: rating.article_id,
|
||||||
@@ -209,6 +223,8 @@ impl PreferenceState {
|
|||||||
decay: decay(age_days, ranking.rating_half_life_days),
|
decay: decay(age_days, ranking.rating_half_life_days),
|
||||||
embedding,
|
embedding,
|
||||||
feeds,
|
feeds,
|
||||||
|
author,
|
||||||
|
aggregator_only,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
Ok(Self::build(examples, ranking))
|
Ok(Self::build(examples, ranking))
|
||||||
@@ -298,16 +314,23 @@ impl PreferenceState {
|
|||||||
(knn, neighbours)
|
(knn, neighbours)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Mean Beta-smoothed rate over the article's rated direct feeds (§9.3).
|
/// Mean Beta-smoothed rate over the article's rated direct feeds and author
|
||||||
|
/// (§9.3).
|
||||||
pub fn feed(&self, article: &Article) -> Option<f64> {
|
pub fn feed(&self, article: &Article) -> Option<f64> {
|
||||||
if self.feed_gate <= 0.0 {
|
if self.feed_gate <= 0.0 {
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
let rates = direct_feeds(article)
|
let mut rates = direct_feeds(article)
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter_map(|feed| self.feed_rates.get(&feed))
|
.filter_map(|feed| self.feed_rates.get(&feed))
|
||||||
.map(|rate| rate.rate())
|
.map(|rate| rate.rate())
|
||||||
.collect::<Vec<_>>();
|
.collect::<Vec<_>>();
|
||||||
|
if let Some(rate) = normalize_author(article.author.as_deref())
|
||||||
|
.as_ref()
|
||||||
|
.and_then(|author| self.author_rates.get(author))
|
||||||
|
{
|
||||||
|
rates.push(rate.rate());
|
||||||
|
}
|
||||||
(!rates.is_empty()).then(|| rates.iter().sum::<f64>() / rates.len() as f64)
|
(!rates.is_empty()).then(|| rates.iter().sum::<f64>() / rates.len() as f64)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -315,6 +338,13 @@ impl PreferenceState {
|
|||||||
pub fn feed_credit(&self, feed: FeedId) -> Option<(f64, f64)> {
|
pub fn feed_credit(&self, feed: FeedId) -> Option<(f64, f64)> {
|
||||||
self.feed_rates.get(&feed).map(|rate| (rate.up, rate.down))
|
self.feed_rates.get(&feed).map(|rate| (rate.up, rate.down))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Per-author `(up, down)` credit, exposed for tests of §9.3.
|
||||||
|
pub fn author_credit(&self, author: &str) -> Option<(f64, f64)> {
|
||||||
|
normalize_author(Some(author))
|
||||||
|
.and_then(|author| self.author_rates.get(&author))
|
||||||
|
.map(|rate| (rate.up, rate.down))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `0.5 ^ (age_days / half_life_days)` (§9.2).
|
/// `0.5 ^ (age_days / half_life_days)` (§9.2).
|
||||||
@@ -354,22 +384,46 @@ pub fn direct_feeds(article: &Article) -> Vec<FeedId> {
|
|||||||
feeds
|
feeds
|
||||||
}
|
}
|
||||||
|
|
||||||
fn feed_rates(examples: &[RatedExample]) -> (HashMap<FeedId, FeedRate>, usize) {
|
fn normalize_author(author: Option<&str>) -> Option<String> {
|
||||||
let mut rates: HashMap<FeedId, FeedRate> = HashMap::new();
|
let normalized = author?
|
||||||
|
.split_whitespace()
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join(" ")
|
||||||
|
.to_lowercase();
|
||||||
|
(!normalized.is_empty()).then_some(normalized)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn feed_rates(
|
||||||
|
examples: &[RatedExample],
|
||||||
|
) -> (HashMap<FeedId, FeedRate>, HashMap<String, FeedRate>, usize) {
|
||||||
|
let mut feed_rates: HashMap<FeedId, FeedRate> = HashMap::new();
|
||||||
|
let mut author_rates: HashMap<String, FeedRate> = HashMap::new();
|
||||||
let mut attributable = 0;
|
let mut attributable = 0;
|
||||||
for example in examples {
|
for example in examples {
|
||||||
if example.feeds.is_empty() {
|
let weight = example.weight();
|
||||||
continue;
|
if !example.feeds.is_empty() {
|
||||||
|
let feed_weight = if example.aggregator_only {
|
||||||
|
weight * AGGREGATOR_FEED_SHARE
|
||||||
|
} else {
|
||||||
|
weight
|
||||||
|
};
|
||||||
|
let credit = feed_weight / example.feeds.len() as f64;
|
||||||
|
for feed in &example.feeds {
|
||||||
|
let rate = feed_rates.entry(*feed).or_default();
|
||||||
|
rate.up += credit.max(0.0);
|
||||||
|
rate.down += (-credit).max(0.0);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
attributable += 1;
|
if let Some(author) = &example.author {
|
||||||
let credit = example.weight() / example.feeds.len() as f64;
|
let rate = author_rates.entry(author.clone()).or_default();
|
||||||
for feed in &example.feeds {
|
rate.up += weight.max(0.0);
|
||||||
let rate = rates.entry(*feed).or_default();
|
rate.down += (-weight).max(0.0);
|
||||||
rate.up += credit.max(0.0);
|
}
|
||||||
rate.down += (-credit).max(0.0);
|
if !example.feeds.is_empty() || example.author.is_some() {
|
||||||
|
attributable += 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
(rates, attributable)
|
(feed_rates, author_rates, attributable)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The interest match of §9.1 for one article.
|
/// The interest match of §9.1 for one article.
|
||||||
@@ -620,6 +674,8 @@ mod tests {
|
|||||||
decay: 1.0,
|
decay: 1.0,
|
||||||
embedding: unit(embedding),
|
embedding: unit(embedding),
|
||||||
feeds: vec![id],
|
feeds: vec![id],
|
||||||
|
author: None,
|
||||||
|
aggregator_only: false,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -845,6 +901,69 @@ mod tests {
|
|||||||
assert_eq!(state.feed(&article(9, &[99])), None);
|
assert_eq!(state.feed(&article(9, &[99])), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn aggregator_only_rating_splits_credit_between_feed_and_author() {
|
||||||
|
let mut rated = example(1, "loved", 1.0, &[1.0, 0.0]);
|
||||||
|
rated.decay = 0.4;
|
||||||
|
rated.feeds = vec![10];
|
||||||
|
rated.author = Some("example author".into());
|
||||||
|
rated.aggregator_only = true;
|
||||||
|
let state = PreferenceState::build(vec![rated], &ranking());
|
||||||
|
|
||||||
|
assert!((state.feed_credit(10).unwrap().0 - 0.25 * 0.4).abs() < 1e-9);
|
||||||
|
assert!((state.author_credit("example author").unwrap().0 - 0.4).abs() < 1e-9);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn author_affinity_applies_across_feeds_with_normalized_keys() {
|
||||||
|
let mut ranking = ranking();
|
||||||
|
ranking.feed_floor = 0;
|
||||||
|
ranking.feed_full = 1;
|
||||||
|
let mut rated = example(1, "loved", 1.0, &[1.0, 0.0]);
|
||||||
|
rated.feeds = vec![10];
|
||||||
|
rated.author = Some("Ada Lovelace".into());
|
||||||
|
let state = PreferenceState::build(vec![rated], &ranking);
|
||||||
|
let mut candidate = article(2, &[99]);
|
||||||
|
candidate.author = Some(" ADA lovelace ".into());
|
||||||
|
|
||||||
|
assert_eq!(state.examples[0].author.as_deref(), Some("ada lovelace"));
|
||||||
|
assert_eq!(state.feed_credit(99), None);
|
||||||
|
assert!((state.feed(&candidate).unwrap() - 2.0 / 3.0).abs() < 1e-9);
|
||||||
|
assert_eq!(state.author_credit(" ADA Lovelace "), Some((1.0, 0.0)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn feed_affinity_means_rated_feed_and_rated_author() {
|
||||||
|
let mut ranking = ranking();
|
||||||
|
ranking.feed_floor = 0;
|
||||||
|
ranking.feed_full = 1;
|
||||||
|
let mut feed_loved = example(1, "loved", 1.0, &[1.0, 0.0]);
|
||||||
|
feed_loved.feeds = vec![10];
|
||||||
|
let mut author_down = example(2, "not_for_me", -1.0, &[1.0, 0.0]);
|
||||||
|
author_down.feeds.clear();
|
||||||
|
author_down.author = Some("writer".into());
|
||||||
|
let state = PreferenceState::build(vec![feed_loved, author_down], &ranking);
|
||||||
|
let mut candidate = article(3, &[10]);
|
||||||
|
candidate.author = Some("Writer".into());
|
||||||
|
|
||||||
|
assert!((state.feed(&candidate).unwrap() - 0.5).abs() < 1e-9);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn aggregator_only_without_author_keeps_feed_behavior_at_reduced_credit() {
|
||||||
|
let mut ranking = ranking();
|
||||||
|
ranking.feed_floor = 0;
|
||||||
|
ranking.feed_full = 1;
|
||||||
|
let mut rated = example(1, "not_for_me", -1.0, &[1.0, 0.0]);
|
||||||
|
rated.feeds = vec![10];
|
||||||
|
rated.aggregator_only = true;
|
||||||
|
let state = PreferenceState::build(vec![rated], &ranking);
|
||||||
|
|
||||||
|
assert_eq!(state.feed_credit(10), Some((0.0, 0.25)));
|
||||||
|
assert_eq!(state.author_credit(""), None);
|
||||||
|
assert!((state.feed(&article(2, &[10])).unwrap() - 1.0 / 2.25).abs() < 1e-9);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn feed_is_absent_when_the_gate_is_closed() {
|
fn feed_is_absent_when_the_gate_is_closed() {
|
||||||
let ranking = ranking(); // feed_floor 15
|
let ranking = ranking(); // feed_floor 15
|
||||||
|
|||||||
@@ -291,16 +291,17 @@ impl Db {
|
|||||||
|
|
||||||
/// Upsert a deduped cluster by canonical URL; returns its `articles.id`.
|
/// Upsert a deduped cluster by canonical URL; returns its `articles.id`.
|
||||||
///
|
///
|
||||||
/// The denormalized fields on [`Article`] are not stored here — they come
|
/// Most denormalized fields on [`Article`] come from the joined `entries`
|
||||||
/// from the joined `entries` row when loading.
|
/// row when loading; the extracted author is stored on `articles`.
|
||||||
pub async fn upsert_article(&self, article: &Article) -> Result<ArticleId> {
|
pub async fn upsert_article(&self, article: &Article) -> Result<ArticleId> {
|
||||||
let sources = serde_json::to_string(&article.sources).unwrap_or_else(|_| "[]".into());
|
let sources = serde_json::to_string(&article.sources).unwrap_or_else(|_| "[]".into());
|
||||||
let row = sqlx::query(
|
let row = sqlx::query(
|
||||||
"INSERT INTO articles (canonical_url, title, best_entry_id, content_html, word_count,
|
"INSERT INTO articles (canonical_url, title, author, best_entry_id, content_html,
|
||||||
excerpt_only, image_count, sources_json, first_seen)
|
word_count, excerpt_only, image_count, sources_json, first_seen)
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
ON CONFLICT(canonical_url) DO UPDATE SET
|
ON CONFLICT(canonical_url) DO UPDATE SET
|
||||||
title = excluded.title,
|
title = excluded.title,
|
||||||
|
author = excluded.author,
|
||||||
best_entry_id = excluded.best_entry_id,
|
best_entry_id = excluded.best_entry_id,
|
||||||
content_html = excluded.content_html,
|
content_html = excluded.content_html,
|
||||||
word_count = excluded.word_count,
|
word_count = excluded.word_count,
|
||||||
@@ -311,6 +312,7 @@ impl Db {
|
|||||||
)
|
)
|
||||||
.bind(&article.canonical_url)
|
.bind(&article.canonical_url)
|
||||||
.bind(&article.title)
|
.bind(&article.title)
|
||||||
|
.bind(&article.author)
|
||||||
.bind((article.best_entry_id != 0).then_some(article.best_entry_id))
|
.bind((article.best_entry_id != 0).then_some(article.best_entry_id))
|
||||||
.bind(&article.content_html)
|
.bind(&article.content_html)
|
||||||
.bind(article.word_count)
|
.bind(article.word_count)
|
||||||
@@ -860,7 +862,7 @@ macro_rules! article_select {
|
|||||||
a.word_count AS word_count, a.excerpt_only AS excerpt_only,
|
a.word_count AS word_count, a.excerpt_only AS excerpt_only,
|
||||||
a.image_count AS image_count, a.sources_json AS sources_json,
|
a.image_count AS image_count, a.sources_json AS sources_json,
|
||||||
a.first_seen AS first_seen,
|
a.first_seen AS first_seen,
|
||||||
e.url AS entry_url, e.author AS author, e.feed_id AS feed_id,
|
e.url AS entry_url, COALESCE(a.author, e.author) AS author, e.feed_id AS feed_id,
|
||||||
e.feed_title AS feed_title, e.category AS category,
|
e.feed_title AS feed_title, e.category AS category,
|
||||||
e.published_at AS published_at, e.comments_url AS comments_url
|
e.published_at AS published_at, e.comments_url AS comments_url
|
||||||
FROM articles a LEFT JOIN entries e ON e.id = a.best_entry_id
|
FROM articles a LEFT JOIN entries e ON e.id = a.best_entry_id
|
||||||
@@ -1163,7 +1165,7 @@ mod tests {
|
|||||||
}],
|
}],
|
||||||
first_seen: ts("2026-08-15T05:30:00Z"),
|
first_seen: ts("2026-08-15T05:30:00Z"),
|
||||||
url: "https://example.com/1".into(),
|
url: "https://example.com/1".into(),
|
||||||
author: None,
|
author: Some("Page Writer".into()),
|
||||||
feed_id: 7,
|
feed_id: 7,
|
||||||
feed_title: "Hacker News".into(),
|
feed_title: "Hacker News".into(),
|
||||||
category: None,
|
category: None,
|
||||||
@@ -1194,11 +1196,20 @@ mod tests {
|
|||||||
assert_eq!(loaded.social.len(), 1);
|
assert_eq!(loaded.social.len(), 1);
|
||||||
assert_eq!(loaded.social[0].score, 342);
|
assert_eq!(loaded.social[0].score, 342);
|
||||||
assert_eq!(loaded.feed_title, "Hacker News");
|
assert_eq!(loaded.feed_title, "Hacker News");
|
||||||
|
assert_eq!(loaded.author.as_deref(), Some("Page Writer"));
|
||||||
// The batched lookup agrees with the single one and skips unknown ids.
|
// The batched lookup agrees with the single one and skips unknown ids.
|
||||||
let batch = db.get_articles(&[id, 9_999]).await.unwrap();
|
let batch = db.get_articles(&[id, 9_999]).await.unwrap();
|
||||||
assert_eq!(batch.len(), 1);
|
assert_eq!(batch.len(), 1);
|
||||||
assert_eq!(batch[&id], loaded);
|
assert_eq!(batch[&id], loaded);
|
||||||
assert!(db.get_articles(&[]).await.unwrap().is_empty());
|
assert!(db.get_articles(&[]).await.unwrap().is_empty());
|
||||||
|
|
||||||
|
sqlx::query("UPDATE articles SET author = NULL WHERE id = ?")
|
||||||
|
.bind(id)
|
||||||
|
.execute(db.pool())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let fallback = db.get_article(id).await.unwrap().unwrap();
|
||||||
|
assert_eq!(fallback.author.as_deref(), Some("someone"));
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
db.article_id_for_url("https://example.com/1")
|
db.article_id_for_url("https://example.com/1")
|
||||||
.await
|
.await
|
||||||
|
|||||||
@@ -380,8 +380,17 @@ fn build_article(members: Vec<(Entry, String)>, feed_urls: &FeedUrls) -> Article
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// An aggregator entry's author is usually the submitter, so it only counts
|
||||||
|
// when no direct feed carried the story; extraction may still replace it
|
||||||
|
// with the page's own byline.
|
||||||
|
let is_direct = |e: &Entry| {
|
||||||
|
classify_source_with_feed(e, feed_urls.get(&e.feed_id).map(String::as_str))
|
||||||
|
== SourceKind::Feed
|
||||||
|
};
|
||||||
|
let has_direct = members.iter().any(|(e, _)| is_direct(e));
|
||||||
let author = members
|
let author = members
|
||||||
.iter()
|
.iter()
|
||||||
|
.filter(|(e, _)| !has_direct || is_direct(e))
|
||||||
.filter_map(|(e, _)| e.author.clone())
|
.filter_map(|(e, _)| e.author.clone())
|
||||||
.find(|a| !a.trim().is_empty());
|
.find(|a| !a.trim().is_empty());
|
||||||
|
|
||||||
@@ -638,6 +647,38 @@ mod tests {
|
|||||||
assert_eq!(articles[1].sources.len(), 1);
|
assert_eq!(articles[1].sources.len(), 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn direct_feed_author_beats_aggregator_submitter() {
|
||||||
|
let mut aggregator = entry(1, "https://blog.dev/post", "A Distinct Article Title");
|
||||||
|
aggregator.author = Some("HN Submitter".into());
|
||||||
|
aggregator.raw_content = format!("<p>{}</p>", "word ".repeat(50));
|
||||||
|
|
||||||
|
let mut direct = entry(2, "https://blog.dev/post", "A Distinct Article Title");
|
||||||
|
direct.author = Some("Real Writer".into());
|
||||||
|
|
||||||
|
let feed_urls = FeedUrls::from([
|
||||||
|
(aggregator.feed_id, "https://hnrss.org/frontpage".into()),
|
||||||
|
(direct.feed_id, "https://blog.dev/feed.xml".into()),
|
||||||
|
]);
|
||||||
|
let (articles, _) = cluster_with_feeds(vec![aggregator, direct], &feed_urls);
|
||||||
|
|
||||||
|
assert_eq!(articles.len(), 1);
|
||||||
|
assert_eq!(articles[0].best_entry_id, 1);
|
||||||
|
assert_eq!(articles[0].author.as_deref(), Some("Real Writer"));
|
||||||
|
|
||||||
|
// With a direct feed present, a submitter name is not used as a fallback.
|
||||||
|
let mut aggregator = entry(3, "https://blog.dev/other", "Another Distinct Title");
|
||||||
|
aggregator.author = Some("HN Submitter".into());
|
||||||
|
let mut direct = entry(4, "https://blog.dev/other", "Another Distinct Title");
|
||||||
|
direct.author = None;
|
||||||
|
let feed_urls = FeedUrls::from([
|
||||||
|
(aggregator.feed_id, "https://hnrss.org/frontpage".into()),
|
||||||
|
(direct.feed_id, "https://blog.dev/feed.xml".into()),
|
||||||
|
]);
|
||||||
|
let (articles, _) = cluster_with_feeds(vec![aggregator, direct], &feed_urls);
|
||||||
|
assert_eq!(articles[0].author, None);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn clustering_drops_non_articles_and_keeps_short_titles_apart() {
|
fn clustering_drops_non_articles_and_keeps_short_titles_apart() {
|
||||||
let mut a = entry(1, "https://a.dev/1", "News");
|
let mut a = entry(1, "https://a.dev/1", "News");
|
||||||
|
|||||||
+157
-4
@@ -30,6 +30,9 @@ pub const MAX_WORD_CHARS: usize = 200;
|
|||||||
/// U+00AD, invisible unless the renderer actually needs to break there.
|
/// U+00AD, invisible unless the renderer actually needs to break there.
|
||||||
const SOFT_HYPHEN: char = '\u{00ad}';
|
const SOFT_HYPHEN: char = '\u{00ad}';
|
||||||
|
|
||||||
|
/// U+00A0, rendered as visible indentation even when the X4 collapses whitespace.
|
||||||
|
const NO_BREAK_SPACE: char = '\u{00a0}';
|
||||||
|
|
||||||
/// Elements whose content is code, not prose, and must be copied through
|
/// Elements whose content is code, not prose, and must be copied through
|
||||||
/// untouched — a soft hyphen inside a stylesheet would corrupt it.
|
/// untouched — a soft hyphen inside a stylesheet would corrupt it.
|
||||||
const RAW_TEXT_ELEMENTS: &[&str] = &["script", "style"];
|
const RAW_TEXT_ELEMENTS: &[&str] = &["script", "style"];
|
||||||
@@ -127,14 +130,28 @@ pub fn simplify_xhtml(xhtml: &str) -> String {
|
|||||||
break_long_words(&strip_attributes(xhtml, DROPPED_ATTRIBUTES))
|
break_long_words(&strip_attributes(xhtml, DROPPED_ATTRIBUTES))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Insert soft hyphens into words longer than [`MAX_WORD_CHARS`], in text
|
/// Insert soft hyphens into over-long words and make `<pre>` whitespace
|
||||||
/// content only (§3.10).
|
/// explicit, in text content only (§3.10).
|
||||||
fn break_long_words(html: &str) -> String {
|
fn break_long_words(html: &str) -> String {
|
||||||
let mut out = String::with_capacity(html.len());
|
let mut out = String::with_capacity(html.len());
|
||||||
let mut cursor = 0usize;
|
let mut cursor = 0usize;
|
||||||
|
let mut inside_pre = false;
|
||||||
|
let mut at_line_start = false;
|
||||||
|
// A newline right after `<pre>` (or `<pre><code>`) is not a blank line:
|
||||||
|
// browsers drop it, so we do too.
|
||||||
|
let mut pre_just_opened = false;
|
||||||
while let Some(rel) = html[cursor..].find('<') {
|
while let Some(rel) = html[cursor..].find('<') {
|
||||||
let start = cursor + rel;
|
let start = cursor + rel;
|
||||||
soften_text(&html[cursor..start], &mut out);
|
if inside_pre {
|
||||||
|
preserve_pre_whitespace(
|
||||||
|
&html[cursor..start],
|
||||||
|
&mut out,
|
||||||
|
&mut at_line_start,
|
||||||
|
&mut pre_just_opened,
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
soften_text(&html[cursor..start], &mut out);
|
||||||
|
}
|
||||||
let Some(end) = tag_end(html, start) else {
|
let Some(end) = tag_end(html, start) else {
|
||||||
out.push_str(&html[start..]);
|
out.push_str(&html[start..]);
|
||||||
return out;
|
return out;
|
||||||
@@ -142,6 +159,17 @@ fn break_long_words(html: &str) -> String {
|
|||||||
let tag = &html[start..end];
|
let tag = &html[start..end];
|
||||||
out.push_str(tag);
|
out.push_str(tag);
|
||||||
cursor = end;
|
cursor = end;
|
||||||
|
if is_element_tag(tag, "pre", false) {
|
||||||
|
inside_pre = true;
|
||||||
|
at_line_start = true;
|
||||||
|
pre_just_opened = true;
|
||||||
|
} else if is_element_tag(tag, "pre", true) {
|
||||||
|
inside_pre = false;
|
||||||
|
} else if inside_pre && is_element_tag(tag, "br", false) {
|
||||||
|
at_line_start = true;
|
||||||
|
} else if inside_pre && !is_element_tag(tag, "code", false) {
|
||||||
|
pre_just_opened = false;
|
||||||
|
}
|
||||||
// `<style>`/`<script>` bodies are not prose: copy to the closing tag verbatim.
|
// `<style>`/`<script>` bodies are not prose: copy to the closing tag verbatim.
|
||||||
if let Some(name) = raw_text_name(tag)
|
if let Some(name) = raw_text_name(tag)
|
||||||
&& let Some(close) = find_close_tag(html, cursor, name)
|
&& let Some(close) = find_close_tag(html, cursor, name)
|
||||||
@@ -150,10 +178,84 @@ fn break_long_words(html: &str) -> String {
|
|||||||
cursor = close;
|
cursor = close;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
soften_text(&html[cursor..], &mut out);
|
if inside_pre {
|
||||||
|
preserve_pre_whitespace(
|
||||||
|
&html[cursor..],
|
||||||
|
&mut out,
|
||||||
|
&mut at_line_start,
|
||||||
|
&mut pre_just_opened,
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
soften_text(&html[cursor..], &mut out);
|
||||||
|
}
|
||||||
out
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether `tag` opens or closes the named element.
|
||||||
|
fn is_element_tag(tag: &str, name: &str, closing: bool) -> bool {
|
||||||
|
let Some(mut rest) = tag.strip_prefix('<') else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
if closing {
|
||||||
|
let Some(after_slash) = rest.strip_prefix('/') else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
rest = after_slash;
|
||||||
|
} else if rest.starts_with(['/', '!', '?']) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
rest.len() >= name.len()
|
||||||
|
&& rest[..name.len()].eq_ignore_ascii_case(name)
|
||||||
|
&& rest[name.len()..].starts_with([' ', '\t', '\n', '\r', '>', '/'])
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Make whitespace in a `<pre>` text node survive renderers that ignore CSS.
|
||||||
|
fn preserve_pre_whitespace(
|
||||||
|
text: &str,
|
||||||
|
out: &mut String,
|
||||||
|
at_line_start: &mut bool,
|
||||||
|
pre_just_opened: &mut bool,
|
||||||
|
) {
|
||||||
|
let mut rest = text;
|
||||||
|
if *pre_just_opened && !rest.is_empty() {
|
||||||
|
*pre_just_opened = false;
|
||||||
|
rest = rest
|
||||||
|
.strip_prefix("\r\n")
|
||||||
|
.or_else(|| rest.strip_prefix(['\n', '\r']))
|
||||||
|
.unwrap_or(rest);
|
||||||
|
}
|
||||||
|
while !rest.is_empty() {
|
||||||
|
let plain_len = rest.find([' ', '\t', '\n', '\r']).unwrap_or(rest.len());
|
||||||
|
if plain_len > 0 {
|
||||||
|
soften_text(&rest[..plain_len], out);
|
||||||
|
*at_line_start = false;
|
||||||
|
rest = &rest[plain_len..];
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if rest.starts_with("\r\n") {
|
||||||
|
out.push_str("<br/>");
|
||||||
|
*at_line_start = true;
|
||||||
|
rest = &rest[2..];
|
||||||
|
} else if rest.starts_with(['\n', '\r']) {
|
||||||
|
out.push_str("<br/>");
|
||||||
|
*at_line_start = true;
|
||||||
|
rest = &rest[1..];
|
||||||
|
} else if rest.starts_with('\t') {
|
||||||
|
out.extend(std::iter::repeat_n(NO_BREAK_SPACE, 4));
|
||||||
|
rest = &rest[1..];
|
||||||
|
} else {
|
||||||
|
let spaces = rest.bytes().take_while(|&b| b == b' ').count();
|
||||||
|
if *at_line_start || spaces >= 2 {
|
||||||
|
out.extend(std::iter::repeat_n(NO_BREAK_SPACE, spaces));
|
||||||
|
} else {
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
rest = &rest[spaces..];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The element name when `tag` opens a raw-text element, else `None`.
|
/// The element name when `tag` opens a raw-text element, else `None`.
|
||||||
fn raw_text_name(tag: &str) -> Option<&'static str> {
|
fn raw_text_name(tag: &str) -> Option<&'static str> {
|
||||||
let rest = tag.strip_prefix('<')?;
|
let rest = tag.strip_prefix('<')?;
|
||||||
@@ -504,6 +606,57 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_whitespace_is_made_explicit_inside_nested_tags() {
|
||||||
|
let input = "<pre><code>fn main() {\n if x < 2 {\n <em>return</em>;\n }\n <strong><a href=\"#\">// done</a></strong>\n}</code></pre>";
|
||||||
|
let out = simplify_xhtml(input);
|
||||||
|
assert_eq!(
|
||||||
|
out,
|
||||||
|
concat!(
|
||||||
|
"<pre><code>fn main() {<br/>",
|
||||||
|
"\u{00a0}\u{00a0}\u{00a0}\u{00a0}if x < 2 {<br/>",
|
||||||
|
"\u{00a0}\u{00a0}\u{00a0}\u{00a0}\u{00a0}\u{00a0}\u{00a0}\u{00a0}<em>return</em>;<br/>",
|
||||||
|
"\u{00a0}\u{00a0}\u{00a0}\u{00a0}}<br/>",
|
||||||
|
"\u{00a0}<strong><a href=\"#\">// done</a></strong><br/>}</code></pre>"
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn tabs_in_pre_become_four_no_break_spaces() {
|
||||||
|
assert_eq!(
|
||||||
|
simplify_xhtml("<pre><code>one\ttwo</code></pre>"),
|
||||||
|
"<pre><code>one\u{00a0}\u{00a0}\u{00a0}\u{00a0}two</code></pre>"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn whitespace_outside_pre_is_untouched() {
|
||||||
|
let input = "<p>outside text\n next\tword</p><pre>x y</pre><p>tail text</p>";
|
||||||
|
assert_eq!(
|
||||||
|
simplify_xhtml(input),
|
||||||
|
"<p>outside text\n next\tword</p><pre>x\u{00a0}\u{00a0}y</pre><p>tail text</p>"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn newline_right_after_pre_open_is_not_a_blank_line() {
|
||||||
|
assert_eq!(
|
||||||
|
simplify_xhtml("<pre><code>\nlet a = 1;\n\nlet b = 2;</code></pre>"),
|
||||||
|
"<pre><code>let a = 1;<br/><br/>let b = 2;</code></pre>"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
simplify_xhtml("<pre>\n x</pre>"),
|
||||||
|
"<pre>\u{00a0}\u{00a0}x</pre>"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_without_a_trailing_newline_gets_no_extra_break() {
|
||||||
|
let input = "<pre><code>let answer = 42;</code></pre>";
|
||||||
|
assert_eq!(simplify_xhtml(input), input);
|
||||||
|
}
|
||||||
|
|
||||||
/// The shipped X4 stylesheet must already satisfy the X4 rules, so the
|
/// The shipped X4 stylesheet must already satisfy the X4 rules, so the
|
||||||
/// simplifier is a no-op over it (§3.10).
|
/// simplifier is a no-op over it (§3.10).
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
+112
-6
@@ -18,7 +18,7 @@ use url::Url;
|
|||||||
use crate::html::word_count;
|
use crate::html::word_count;
|
||||||
use crate::images::normalize::{normalize_img_tags, prepare_for_readability};
|
use crate::images::normalize::{normalize_img_tags, prepare_for_readability};
|
||||||
use crate::images::refs::collect_image_urls;
|
use crate::images::refs::collect_image_urls;
|
||||||
use crate::types::{Article, ExtractMethod, Extracted};
|
use crate::types::{Article, ExtractMethod, Extracted, SourceKind};
|
||||||
|
|
||||||
/// Word count at or above which Miniflux content is treated as full text (§3.3).
|
/// Word count at or above which Miniflux content is treated as full text (§3.3).
|
||||||
pub const FULL_TEXT_MIN_WORDS: i64 = 250;
|
pub const FULL_TEXT_MIN_WORDS: i64 = 250;
|
||||||
@@ -197,6 +197,7 @@ impl Extractor {
|
|||||||
|| looks_paywalled(&article.url, words, &self.paywall_domains);
|
|| looks_paywalled(&article.url, words, &self.paywall_domains);
|
||||||
Extracted {
|
Extracted {
|
||||||
content_html,
|
content_html,
|
||||||
|
author: None,
|
||||||
word_count: words,
|
word_count: words,
|
||||||
excerpt_only,
|
excerpt_only,
|
||||||
image_urls,
|
image_urls,
|
||||||
@@ -279,10 +280,11 @@ impl Extractor {
|
|||||||
body.extend_from_slice(&chunk);
|
body.extend_from_slice(&chunk);
|
||||||
}
|
}
|
||||||
let html = String::from_utf8_lossy(&body).into_owned();
|
let html = String::from_utf8_lossy(&body).into_owned();
|
||||||
let (title, html) = readable_page(&html, &final_url)?;
|
let (title, html, author) = readable_page(&html, &final_url)?;
|
||||||
Ok(Page {
|
Ok(Page {
|
||||||
title,
|
title,
|
||||||
html,
|
html,
|
||||||
|
author,
|
||||||
final_url,
|
final_url,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -294,6 +296,7 @@ impl Extractor {
|
|||||||
let image_urls = collect_image_urls(&clean, &page.final_url);
|
let image_urls = collect_image_urls(&clean, &page.final_url);
|
||||||
Extracted {
|
Extracted {
|
||||||
content_html: clean,
|
content_html: clean,
|
||||||
|
author: page.author.clone(),
|
||||||
word_count: words,
|
word_count: words,
|
||||||
excerpt_only: looks_paywalled(requested_url, words, &self.paywall_domains),
|
excerpt_only: looks_paywalled(requested_url, words, &self.paywall_domains),
|
||||||
image_urls,
|
image_urls,
|
||||||
@@ -309,6 +312,8 @@ pub struct Page {
|
|||||||
pub title: String,
|
pub title: String,
|
||||||
/// Readability's main-content markup.
|
/// Readability's main-content markup.
|
||||||
pub html: String,
|
pub html: String,
|
||||||
|
/// Readability's normalized byline for the page.
|
||||||
|
pub author: Option<String>,
|
||||||
/// Where the fetch ended up, after any redirects — the base for relative URLs.
|
/// Where the fetch ended up, after any redirects — the base for relative URLs.
|
||||||
pub final_url: String,
|
pub final_url: String,
|
||||||
}
|
}
|
||||||
@@ -321,10 +326,10 @@ pub struct Page {
|
|||||||
/// their image with them) and its lazy-image heuristic overwrites a perfectly
|
/// their image with them) and its lazy-image heuristic overwrites a perfectly
|
||||||
/// good `src` with whatever other attribute happens to contain `.jpg`.
|
/// good `src` with whatever other attribute happens to contain `.jpg`.
|
||||||
pub fn readability(html: &str, url: &str) -> Result<String, ExtractError> {
|
pub fn readability(html: &str, url: &str) -> Result<String, ExtractError> {
|
||||||
readable_page(html, url).map(|(_, content)| content)
|
readable_page(html, url).map(|(_, content, _)| content)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn readable_page(html: &str, url: &str) -> Result<(String, String), ExtractError> {
|
fn readable_page(html: &str, url: &str) -> Result<(String, String, Option<String>), ExtractError> {
|
||||||
let html = prepare_for_readability(html);
|
let html = prepare_for_readability(html);
|
||||||
let config = dom_smoothie::Config {
|
let config = dom_smoothie::Config {
|
||||||
max_elements_to_parse: 60_000,
|
max_elements_to_parse: 60_000,
|
||||||
@@ -337,7 +342,20 @@ fn readable_page(html: &str, url: &str) -> Result<(String, String), ExtractError
|
|||||||
if content.trim().is_empty() {
|
if content.trim().is_empty() {
|
||||||
return Err(ExtractError::NoContent);
|
return Err(ExtractError::NoContent);
|
||||||
}
|
}
|
||||||
Ok((parsed.title.trim().to_string(), content))
|
Ok((
|
||||||
|
parsed.title.trim().to_string(),
|
||||||
|
content,
|
||||||
|
normalize_author(parsed.byline),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn normalize_author(author: Option<String>) -> Option<String> {
|
||||||
|
let author = author?;
|
||||||
|
if author.chars().any(|c| matches!(c, '\n' | '\r')) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let author = author.split_whitespace().collect::<Vec<_>>().join(" ");
|
||||||
|
(!author.is_empty() && author.chars().count() <= 100).then_some(author)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Copy an [`Extracted`] onto its [`Article`].
|
/// Copy an [`Extracted`] onto its [`Article`].
|
||||||
@@ -348,6 +366,11 @@ pub fn apply(article: &mut Article, extracted: Extracted) {
|
|||||||
article.word_count = extracted.word_count;
|
article.word_count = extracted.word_count;
|
||||||
article.excerpt_only = extracted.excerpt_only;
|
article.excerpt_only = extracted.excerpt_only;
|
||||||
article.extract_method = extracted.method;
|
article.extract_method = extracted.method;
|
||||||
|
if let Some(author) = extracted.author
|
||||||
|
&& (article.author.is_none() || !article.came_via(SourceKind::Feed))
|
||||||
|
{
|
||||||
|
article.author = Some(author);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn merge_paywall_domains(configured: Vec<String>) -> Vec<String> {
|
fn merge_paywall_domains(configured: Vec<String>) -> Vec<String> {
|
||||||
@@ -722,11 +745,94 @@ mod tests {
|
|||||||
<article><h1>A Post</h1><p>{paragraph}</p><p>{paragraph}</p></article>\
|
<article><h1>A Post</h1><p>{paragraph}</p><p>{paragraph}</p></article>\
|
||||||
<footer>© 2026</footer></body></html>"
|
<footer>© 2026</footer></body></html>"
|
||||||
);
|
);
|
||||||
let (title, content) = readable_page(&html, "https://blog.dev/p").expect("main content");
|
let (title, content, author) =
|
||||||
|
readable_page(&html, "https://blog.dev/p").expect("main content");
|
||||||
assert_eq!(title, "A Post");
|
assert_eq!(title, "A Post");
|
||||||
|
assert_eq!(author, None);
|
||||||
assert!(content.contains("Readability keeps the body copy"));
|
assert!(content.contains("Readability keeps the body copy"));
|
||||||
let clean = sanitize_with_base(&content, "https://blog.dev/p");
|
let clean = sanitize_with_base(&content, "https://blog.dev/p");
|
||||||
assert!(word_count(&clean) > 200);
|
assert!(word_count(&clean) > 200);
|
||||||
assert!(!clean.contains("<nav"));
|
assert!(!clean.contains("<nav"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn readable_page_plumbs_meta_author_through_extracted() {
|
||||||
|
let html = format!(
|
||||||
|
"<html><head><title>A Post</title>\
|
||||||
|
<meta name=\"author\" content=\" Jane Dev \"></head>\
|
||||||
|
<body><article><h1>A Post</h1><p>{}</p></article></body></html>",
|
||||||
|
"Substantial body copy for readability. ".repeat(40)
|
||||||
|
);
|
||||||
|
let (title, content, author) =
|
||||||
|
readable_page(&html, "https://blog.dev/p").expect("main content");
|
||||||
|
let page = Page {
|
||||||
|
title,
|
||||||
|
html: content,
|
||||||
|
author,
|
||||||
|
final_url: "https://blog.dev/p".into(),
|
||||||
|
};
|
||||||
|
let extracted = Extractor::offline(vec![]).finish_readable(&page.final_url, &page);
|
||||||
|
assert_eq!(extracted.author.as_deref(), Some("Jane Dev"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn readable_page_plumbs_json_ld_author_through_extracted() {
|
||||||
|
let html = format!(
|
||||||
|
r#"<html><head><title>A Post</title>
|
||||||
|
<script type="application/ld+json">{{
|
||||||
|
"@context":"https://schema.org", "@type":"Article",
|
||||||
|
"headline":"A Post", "author":{{"@type":"Person","name":"Alex Writer"}}
|
||||||
|
}}</script></head>
|
||||||
|
<body><article><h1>A Post</h1><p>{}</p></article></body></html>"#,
|
||||||
|
"Substantial body copy for readability. ".repeat(40)
|
||||||
|
);
|
||||||
|
let (title, content, author) =
|
||||||
|
readable_page(&html, "https://blog.dev/p").expect("main content");
|
||||||
|
let page = Page {
|
||||||
|
title,
|
||||||
|
html: content,
|
||||||
|
author,
|
||||||
|
final_url: "https://blog.dev/p".into(),
|
||||||
|
};
|
||||||
|
let extracted = Extractor::offline(vec![]).finish_readable(&page.final_url, &page);
|
||||||
|
assert_eq!(extracted.author.as_deref(), Some("Alex Writer"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn apply_uses_page_author_with_feed_precedence() {
|
||||||
|
let extracted = |author: &str| Extracted {
|
||||||
|
content_html: "<p>body</p>".into(),
|
||||||
|
author: Some(author.into()),
|
||||||
|
word_count: 1,
|
||||||
|
excerpt_only: false,
|
||||||
|
image_urls: vec![],
|
||||||
|
method: ExtractMethod::Readability,
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut aggregator = article("https://blog.dev/aggregator", "");
|
||||||
|
aggregator.sources[0].kind = SourceKind::HnFrontpage;
|
||||||
|
aggregator.author = Some("Submitter".into());
|
||||||
|
apply(&mut aggregator, extracted("Page Writer"));
|
||||||
|
assert_eq!(aggregator.author.as_deref(), Some("Page Writer"));
|
||||||
|
|
||||||
|
let mut direct = article("https://blog.dev/direct", "");
|
||||||
|
direct.author = Some("Feed Writer".into());
|
||||||
|
apply(&mut direct, extracted("Page Writer"));
|
||||||
|
assert_eq!(direct.author.as_deref(), Some("Feed Writer"));
|
||||||
|
|
||||||
|
let mut missing = article("https://blog.dev/missing", "");
|
||||||
|
apply(&mut missing, extracted("Page Writer"));
|
||||||
|
assert_eq!(missing.author.as_deref(), Some("Page Writer"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn implausible_page_authors_are_dropped() {
|
||||||
|
assert_eq!(
|
||||||
|
normalize_author(Some(" Jane Dev ".into())).as_deref(),
|
||||||
|
Some("Jane Dev")
|
||||||
|
);
|
||||||
|
assert_eq!(normalize_author(Some("Jane\nDev".into())), None);
|
||||||
|
assert_eq!(normalize_author(Some("x".repeat(101))), None);
|
||||||
|
assert_eq!(normalize_author(Some(" ".into())), None);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
+4
-1
@@ -165,7 +165,7 @@ fn imported_article(
|
|||||||
sources: Vec::new(),
|
sources: Vec::new(),
|
||||||
first_seen,
|
first_seen,
|
||||||
url: page.final_url,
|
url: page.final_url,
|
||||||
author: None,
|
author: extracted.author,
|
||||||
feed_id: 0,
|
feed_id: 0,
|
||||||
feed_title: "Imported".into(),
|
feed_title: "Imported".into(),
|
||||||
category: None,
|
category: None,
|
||||||
@@ -482,6 +482,7 @@ mod tests {
|
|||||||
let page = Page {
|
let page = Page {
|
||||||
title: "A historical essay".into(),
|
title: "A historical essay".into(),
|
||||||
html: "<article><p>Useful old writing.</p><script>bad()</script><img src=\"/chart.png\"></article>".into(),
|
html: "<article><p>Useful old writing.</p><script>bad()</script><img src=\"/chart.png\"></article>".into(),
|
||||||
|
author: Some("Essay Writer".into()),
|
||||||
final_url: "https://example.com/essays/old".into(),
|
final_url: "https://example.com/essays/old".into(),
|
||||||
};
|
};
|
||||||
let extracted = extractor.finish_readable("https://example.com/old", &page);
|
let extracted = extractor.finish_readable("https://example.com/old", &page);
|
||||||
@@ -494,6 +495,7 @@ mod tests {
|
|||||||
assert_eq!(article.title, "A historical essay");
|
assert_eq!(article.title, "A historical essay");
|
||||||
assert_eq!(article.best_entry_id, 0);
|
assert_eq!(article.best_entry_id, 0);
|
||||||
assert_eq!(article.feed_title, "Imported");
|
assert_eq!(article.feed_title, "Imported");
|
||||||
|
assert_eq!(article.author.as_deref(), Some("Essay Writer"));
|
||||||
assert!(article.sources.is_empty());
|
assert!(article.sources.is_empty());
|
||||||
assert!(!article.content_html.contains("script"));
|
assert!(!article.content_html.contains("script"));
|
||||||
assert_eq!(article.image_urls, ["https://example.com/chart.png"]);
|
assert_eq!(article.image_urls, ["https://example.com/chart.png"]);
|
||||||
@@ -502,6 +504,7 @@ mod tests {
|
|||||||
assert_eq!(loaded.title, "A historical essay");
|
assert_eq!(loaded.title, "A historical essay");
|
||||||
assert_eq!(loaded.best_entry_id, 0);
|
assert_eq!(loaded.best_entry_id, 0);
|
||||||
assert_eq!(loaded.feed_title, "Imported");
|
assert_eq!(loaded.feed_title, "Imported");
|
||||||
|
assert_eq!(loaded.author.as_deref(), Some("Essay Writer"));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
|
|||||||
+3
-1
@@ -90,6 +90,8 @@ pub enum ExtractMethod {
|
|||||||
pub struct Extracted {
|
pub struct Extracted {
|
||||||
/// Sanitized XHTML-safe body markup.
|
/// Sanitized XHTML-safe body markup.
|
||||||
pub content_html: String,
|
pub content_html: String,
|
||||||
|
/// Author discovered in the fetched page, if any.
|
||||||
|
pub author: Option<String>,
|
||||||
pub word_count: i64,
|
pub word_count: i64,
|
||||||
/// True when we only have an excerpt/paywall stub — penalized in pre-filter.
|
/// True when we only have an excerpt/paywall stub — penalized in pre-filter.
|
||||||
pub excerpt_only: bool,
|
pub excerpt_only: bool,
|
||||||
@@ -119,7 +121,7 @@ pub struct Article {
|
|||||||
pub sources: Vec<SourceRef>,
|
pub sources: Vec<SourceRef>,
|
||||||
pub first_seen: Timestamp,
|
pub first_seen: Timestamp,
|
||||||
|
|
||||||
// --- denormalized, not stored on `articles` ---
|
// --- joined / derived fields ---
|
||||||
pub url: String,
|
pub url: String,
|
||||||
pub author: Option<String>,
|
pub author: Option<String>,
|
||||||
pub feed_id: FeedId,
|
pub feed_id: FeedId,
|
||||||
|
|||||||
@@ -58,7 +58,8 @@ pub fn routes() -> Router<AppState> {
|
|||||||
pub struct FeedCredit {
|
pub struct FeedCredit {
|
||||||
pub feed_id: FeedId,
|
pub feed_id: FeedId,
|
||||||
pub feed_title: String,
|
pub feed_title: String,
|
||||||
/// `value × decay / n` over the article's `n` direct feeds.
|
/// `value × decay / n` over the article's `n` direct feeds, reduced for an
|
||||||
|
/// aggregator-only article.
|
||||||
pub credit: f64,
|
pub credit: f64,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -97,7 +98,12 @@ pub fn contribution(
|
|||||||
let feed_credits = if rating.label == "cleared" || feeds.is_empty() {
|
let feed_credits = if rating.label == "cleared" || feeds.is_empty() {
|
||||||
Vec::new()
|
Vec::new()
|
||||||
} else {
|
} else {
|
||||||
let credit = weight / feeds.len() as f64;
|
let feed_weight = if article.is_some_and(crate::discovery::aggregator_only) {
|
||||||
|
weight * signals::AGGREGATOR_FEED_SHARE
|
||||||
|
} else {
|
||||||
|
weight
|
||||||
|
};
|
||||||
|
let credit = feed_weight / feeds.len() as f64;
|
||||||
feeds
|
feeds
|
||||||
.iter()
|
.iter()
|
||||||
.map(|feed_id| FeedCredit {
|
.map(|feed_id| FeedCredit {
|
||||||
|
|||||||
@@ -1,2 +1,2 @@
|
|||||||
/*! tailwindcss v4.3.3 | MIT License | https://tailwindcss.com */
|
/*! tailwindcss v4.3.3 | MIT License | https://tailwindcss.com */
|
||||||
@layer properties{@supports (((-webkit-hyphens:none)) and (not (margin-trim:inline))) or ((-moz-orient:inline) and (not (color:rgb(from red r g b)))){*,:before,:after,::backdrop{--tw-translate-x:0;--tw-translate-y:0;--tw-translate-z:0;--tw-rotate-x:initial;--tw-rotate-y:initial;--tw-rotate-z:initial;--tw-skew-x:initial;--tw-skew-y:initial;--tw-space-y-reverse:0;--tw-divide-y-reverse:0;--tw-border-style:solid;--tw-leading:initial;--tw-font-weight:initial;--tw-tracking:initial;--tw-ordinal:initial;--tw-slashed-zero:initial;--tw-numeric-figure:initial;--tw-numeric-spacing:initial;--tw-numeric-fraction:initial;--tw-shadow:0 0 #0000;--tw-shadow-color:initial;--tw-shadow-alpha:100%;--tw-inset-shadow:0 0 #0000;--tw-inset-shadow-color:initial;--tw-inset-shadow-alpha:100%;--tw-ring-color:initial;--tw-ring-shadow:0 0 #0000;--tw-inset-ring-color:initial;--tw-inset-ring-shadow:0 0 #0000;--tw-ring-inset:initial;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-offset-shadow:0 0 #0000;--tw-outline-style:solid;--tw-blur:initial;--tw-brightness:initial;--tw-contrast:initial;--tw-grayscale:initial;--tw-hue-rotate:initial;--tw-invert:initial;--tw-opacity:initial;--tw-saturate:initial;--tw-sepia:initial;--tw-drop-shadow:initial;--tw-drop-shadow-color:initial;--tw-drop-shadow-alpha:100%;--tw-drop-shadow-size:initial;--tw-duration:initial}}}@layer theme{:root,:host{--font-sans:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--font-serif:"Newsreader", "Newsreader Fallback Georgia", "Newsreader Fallback Times", "Newsreader Fallback Noto Serif", "Newsreader Fallback DejaVu Serif", "Iowan Old Style", "Palatino Linotype", Georgia, serif;--font-mono:ui-monospace, SFMono-Regular, Menlo, monospace;--spacing:.25rem;--container-md:28rem;--container-lg:32rem;--container-2xl:42rem;--container-7xl:80rem;--text-xs:.75rem;--text-xs--line-height:calc(1 / .75);--text-sm:.875rem;--text-sm--line-height:calc(1.25 / .875);--text-base:1rem;--text-base--line-height:calc(1.5 / 1);--text-lg:1.125rem;--text-lg--line-height:calc(1.75 / 1.125);--text-xl:1.25rem;--text-xl--line-height:calc(1.75 / 1.25);--text-2xl:1.5rem;--text-2xl--line-height:calc(2 / 1.5);--text-3xl:1.875rem;--text-3xl--line-height:calc(2.25 / 1.875);--text-4xl:2.25rem;--text-4xl--line-height:calc(2.5 / 2.25);--text-5xl:3rem;--text-5xl--line-height:1;--font-weight-normal:400;--font-weight-medium:500;--font-weight-semibold:600;--tracking-normal:0em;--leading-tight:1.25;--leading-snug:1.375;--leading-normal:1.5;--leading-relaxed:1.625;--radius-sm:.25rem;--default-transition-duration:.15s;--default-transition-timing-function:cubic-bezier(.4, 0, .2, 1);--default-font-family:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--default-mono-font-family:ui-monospace, SFMono-Regular, Menlo, monospace;--color-paper:var(--paper);--color-paper-2:var(--paper-2);--color-ink:var(--ink)}}@layer base{*,:after,:before,::backdrop{box-sizing:border-box;border:0 solid;margin:0;padding:0}::file-selector-button{box-sizing:border-box;border:0 solid;margin:0;padding:0}html,:host{-webkit-text-size-adjust:100%;tab-size:4;line-height:1.5;font-family:var(--default-font-family,-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Noto Sans", Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji");font-feature-settings:var(--default-font-feature-settings,normal);font-variation-settings:var(--default-font-variation-settings,normal);-webkit-tap-highlight-color:transparent}hr{height:0;color:inherit;border-top-width:1px}abbr:where([title]){-webkit-text-decoration:underline dotted;text-decoration:underline dotted}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;text-decoration:inherit}b,strong{font-weight:bolder}code,kbd,samp,pre{font-family:var(--default-mono-font-family,ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace);font-feature-settings:var(--default-mono-font-feature-settings,normal);font-variation-settings:var(--default-mono-font-variation-settings,normal);font-size:1em}small{font-size:80%}sub,sup{vertical-align:baseline;font-size:75%;line-height:0;position:relative}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit;border-collapse:collapse}:-moz-focusring:where(:not(iframe)){outline:auto}progress{vertical-align:baseline}summary{display:list-item}ol,ul,menu{list-style:none}img,svg,video,canvas,audio,iframe,embed,object{vertical-align:middle;display:block}img,video{max-width:100%;height:auto}button,input,select,optgroup,textarea{font:inherit;font-feature-settings:inherit;font-variation-settings:inherit;letter-spacing:inherit;color:inherit;opacity:1;background-color:#0000;border-radius:0}::file-selector-button{font:inherit;font-feature-settings:inherit;font-vaLine truncated
|
@layer properties{@supports (((-webkit-hyphens:none)) and (not (margin-trim:inline))) or ((-moz-orient:inline) and (not (color:rgb(from red r g b)))){*,:before,:after,::backdrop{--tw-translate-x:0;--tw-translate-y:0;--tw-translate-z:0;--tw-rotate-x:initial;--tw-rotate-y:initial;--tw-rotate-z:initial;--tw-skew-x:initial;--tw-skew-y:initial;--tw-space-y-reverse:0;--tw-divide-y-reverse:0;--tw-border-style:solid;--tw-leading:initial;--tw-font-weight:initial;--tw-tracking:initial;--tw-ordinal:initial;--tw-slashed-zero:initial;--tw-numeric-figure:initial;--tw-numeric-spacing:initial;--tw-numeric-fraction:initial;--tw-shadow:0 0 #0000;--tw-shadow-color:initial;--tw-shadow-alpha:100%;--tw-inset-shadow:0 0 #0000;--tw-inset-shadow-color:initial;--tw-inset-shadow-alpha:100%;--tw-ring-color:initial;--tw-ring-shadow:0 0 #0000;--tw-inset-ring-color:initial;--tw-inset-ring-shadow:0 0 #0000;--tw-ring-inset:initial;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-offset-shadow:0 0 #0000;--tw-outline-style:solid;--tw-blur:initial;--tw-brightness:initial;--tw-contrast:initial;--tw-grayscale:initial;--tw-hue-rotate:initial;--tw-invert:initial;--tw-opacity:initial;--tw-saturate:initial;--tw-sepia:initial;--tw-drop-shadow:initial;--tw-drop-shadow-color:initial;--tw-drop-shadow-alpha:100%;--tw-drop-shadow-size:initial;--tw-duration:initial}}}@layer theme{:root,:host{--font-sans:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--font-serif:"Newsreader", "Newsreader Fallback Georgia", "Newsreader Fallback Times", "Newsreader Fallback Noto Serif", "Newsreader Fallback DejaVu Serif", "Iowan Old Style", "Palatino Linotype", Georgia, serif;--font-mono:ui-monospace, SFMono-Regular, Menlo, monospace;--spacing:.25rem;--container-md:28rem;--container-lg:32rem;--container-2xl:42rem;--container-7xl:80rem;--text-xs:.75rem;--text-xs--line-height:calc(1 / .75);--text-sm:.875rem;--text-sm--line-height:calc(1.25 / .875);--text-base:1rem;--text-base--line-height:calc(1.5 / 1);--text-lg:1.125rem;--text-lg--line-height:calc(1.75 / 1.125);--text-xl:1.25rem;--text-xl--line-height:calc(1.75 / 1.25);--text-2xl:1.5rem;--text-2xl--line-height:calc(2 / 1.5);--text-3xl:1.875rem;--text-3xl--line-height:calc(2.25 / 1.875);--text-4xl:2.25rem;--text-4xl--line-height:calc(2.5 / 2.25);--text-5xl:3rem;--text-5xl--line-height:1;--font-weight-normal:400;--font-weight-medium:500;--font-weight-semibold:600;--tracking-normal:0em;--leading-tight:1.25;--leading-snug:1.375;--leading-normal:1.5;--leading-relaxed:1.625;--radius-sm:.25rem;--default-transition-duration:.15s;--default-transition-timing-function:cubic-bezier(.4, 0, .2, 1);--default-font-family:ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;--default-mono-font-family:ui-monospace, SFMono-Regular, Menlo, monospace;--color-paper:var(--paper);--color-paper-2:var(--paper-2);--color-ink:var(--ink)}}@layer base{*,:after,:before,::backdrop{box-sizing:border-box;border:0 solid;margin:0;padding:0}::file-selector-button{box-sizing:border-box;border:0 solid;margin:0;padding:0}html,:host{-webkit-text-size-adjust:100%;tab-size:4;line-height:1.5;font-family:var(--default-font-family,-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Noto Sans", Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji");font-feature-settings:var(--default-font-feature-settings,normal);font-variation-settings:var(--default-font-variation-settings,normal);-webkit-tap-highlight-color:transparent}hr{height:0;color:inherit;border-top-width:1px}abbr:where([title]){-webkit-text-decoration:underline dotted;text-decoration:underline dotted}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;-webkit-text-decoration:inherit;text-decoration:inherit}b,strong{font-weight:bolder}code,kbd,samp,pre{font-family:var(--default-mono-font-family,ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace);font-feature-settings:var(--default-mono-font-feature-settings,normal);font-variation-settings:var(--default-mono-font-variation-settings,normal);font-size:1em}small{font-size:80%}sub,sup{vertical-align:baseline;font-size:75%;line-height:0;position:relative}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit;border-collapse:collapse}:-moz-focusring:where(:not(iframe)){outline:auto}progress{vertical-align:baseline}summary{display:list-item}ol,ul,menu{list-style:none}img,svg,video,canvas,audio,iframe,embed,object{vertical-align:middle;display:block}img,video{max-width:100%;height:auto}button,input,select,optgroup,textarea{font:inherit;font-feature-settings:inherit;font-variation-settings:inherit;letter-spacing:inherit;color:inherit;opacity:1;background-color:#0000;border-radius:0}::file-selector-button{font:inherit;font-feature-settings:inherit;font-vaLine truncated
|
||||||
@@ -162,6 +162,7 @@
|
|||||||
.scroll-x > table { min-width:max-content; }
|
.scroll-x > table { min-width:max-content; }
|
||||||
.scroll-x.tall { @apply max-h-[70vh] overflow-y-auto; overscroll-behavior-block:contain; }
|
.scroll-x.tall { @apply max-h-[70vh] overflow-y-auto; overscroll-behavior-block:contain; }
|
||||||
|
|
||||||
|
.reader-page { @apply min-w-0; overflow-wrap:anywhere; }
|
||||||
.prose-body { @apply text-[1.0625rem] leading-[1.72] text-ink; }
|
.prose-body { @apply text-[1.0625rem] leading-[1.72] text-ink; }
|
||||||
.prose-body > * + * { @apply mt-5; }
|
.prose-body > * + * { @apply mt-5; }
|
||||||
.prose-body p, .index-summary { text-wrap:pretty; }
|
.prose-body p, .index-summary { text-wrap:pretty; }
|
||||||
@@ -171,6 +172,9 @@
|
|||||||
.prose-body img { @apply mx-auto my-7 block h-auto max-w-full rounded-sm text-center font-sans text-xs italic text-muted; outline:1px solid oklch(0 0 0 / 0.1); outline-offset:-1px; }
|
.prose-body img { @apply mx-auto my-7 block h-auto max-w-full rounded-sm text-center font-sans text-xs italic text-muted; outline:1px solid oklch(0 0 0 / 0.1); outline-offset:-1px; }
|
||||||
@media (prefers-color-scheme: dark) { :root:not([data-theme="light"]) .prose-body img { outline-color:oklch(1 0 0 / 0.1); } }
|
@media (prefers-color-scheme: dark) { :root:not([data-theme="light"]) .prose-body img { outline-color:oklch(1 0 0 / 0.1); } }
|
||||||
:root[data-theme="dark"] .prose-body img { outline-color:oklch(1 0 0 / 0.1); }
|
:root[data-theme="dark"] .prose-body img { outline-color:oklch(1 0 0 / 0.1); }
|
||||||
|
.prose-body table { @apply block max-w-full overflow-x-auto; overscroll-behavior-inline:contain; }
|
||||||
|
/* Cells keep whole words and let the table scroll instead of squeezing. */
|
||||||
|
.prose-body th, .prose-body td { overflow-wrap:normal; }
|
||||||
.prose-body pre { @apply max-w-full overflow-x-auto rounded-sm bg-paper-2 p-4 text-sm leading-relaxed; }
|
.prose-body pre { @apply max-w-full overflow-x-auto rounded-sm bg-paper-2 p-4 text-sm leading-relaxed; }
|
||||||
.prose-body :not(pre) > code { @apply rounded-sm bg-paper-2 px-1 py-0.5 text-[0.9em]; }
|
.prose-body :not(pre) > code { @apply rounded-sm bg-paper-2 px-1 py-0.5 text-[0.9em]; }
|
||||||
.prose-body blockquote { @apply my-7 border-l-2 border-rule-strong pl-5 text-ink-2; }
|
.prose-body blockquote { @apply my-7 border-l-2 border-rule-strong pl-5 text-ink-2; }
|
||||||
|
|||||||
Reference in new issue
Block a user