crawlnicle/Cargo.toml

[package]
name = "crawlnicle"
version = "0.1.0"
edition = "2021"
default-run = "crawlnicle"
authors = ["Tyler Hallada <tyler@hallada.net>"]

[lib]
name = "lib"
path = "src/lib.rs"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
ansi-to-html = "0.1"
anyhow = "1"
article_scraper = "2.0.0-alpha.0"
axum = { version = "0.6", features = ["form"] }
bytes = "1.4"
chrono = { version = "0.4", features = ["serde"] }
clap = { version = "4.3", features = ["derive", "env"] }
dotenvy = "0.15"
feed-rs = "1.3"
maud = { version = "0.25", features = ["axum"] }
notify = "6"
once_cell = "1.17"
readability = "0.2"
reqwest = { version = "0.11", features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_with = "3"
sqlx = { version = "0.7", features = [
  "runtime-tokio-native-tls",
  "postgres",
  "macros",
  "migrate",
  "chrono",
  "uuid",
] }
thiserror = "1"
tokio = { version = "1", features = ["full"] }
tokio-stream = { version = "0.1", features = ["sync"] }
tower = "0.4"
tower-livereload = "0.8"
tower-http = { version = "0.4", features = ["trace", "fs"] }
tracing = { version = "0.1", features = ["valuable", "attributes"] }
tracing-appender = "0.2"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
uuid = { version = "1.3", features = ["serde"] }
url = "2.4"
validator = { version = "0.16", features = ["derive"] }
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`[package]`
Rename to crawlnicle 2023-05-10 04:00:48 +00:00			`name = "crawlnicle"`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`version = "0.1.0"`
			`edition = "2021"`
Rename to crawlnicle 2023-05-10 04:00:48 +00:00			`default-run = "crawlnicle"`
Replace argh with clap Mostly for the more concise Config parsing and error handling. 2023-06-27 18:03:52 +00:00			`authors = ["Tyler Hallada <tyler@hallada.net>"]`
Create cli binary Just has `add-feed` command so far. 2023-05-09 04:08:55 +00:00
			`[lib]`
			`name = "lib"`
			`path = "src/lib.rs"`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00
			`# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html`

			`[dependencies]`
Streaming log page with colors Using the ansi-to-html crate. 2023-06-07 01:14:29 +00:00			`ansi-to-html = "0.1"`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`anyhow = "1"`
Fetch and save entry HTML content with metadata And render the extracted HTML on the entry page in the frontend. 2023-06-07 05:06:03 +00:00			`article_scraper = "2.0.0-alpha.0"`
Add feed form 2023-07-08 05:05:49 +00:00			`axum = { version = "0.6", features = ["form"] }`
Complete log stream implementation Sets up a watch channel to send tracing lines from tracing-subscriber to receivers in a axum handler which streams Server Sent Events to any number of connected /log/stream clients. 2023-06-03 23:03:58 +00:00			`bytes = "1.4"`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`chrono = { version = "0.4", features = ["serde"] }`
Replace argh with clap Mostly for the more concise Config parsing and error handling. 2023-06-27 18:03:52 +00:00			`clap = { version = "4.3", features = ["derive", "env"] }`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`dotenvy = "0.15"`
Add very basic crawl job Loops through feeds and adds items from each feed. 2023-05-10 03:46:42 +00:00			`feed-rs = "1.3"`
Add maud for frontend and move api to sub module 2023-06-01 04:03:00 +00:00			`maud = { version = "0.25", features = ["axum"] }`
Add (semi) live reloading for debug server 2023-06-01 04:59:46 +00:00			`notify = "6"`
Add /log page that displays server log This will eventually be behind an authorization guard so that only I have access, but it's useful to have for monitoring and testing out turbo streams. 2023-06-03 05:09:25 +00:00			`once_cell = "1.17"`
Add EntryCrawler that uses readability lib Actors delegating to actors baybeeee 2023-07-11 05:52:38 +00:00			`readability = "0.2"`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`reqwest = { version = "0.11", features = ["json"] }`
			`serde = { version = "1", features = ["derive"] }`
Improve error handling 2023-05-07 22:20:19 +00:00			`serde_with = "3"`
Upgrade sqlx to 0.7 + some tweaks 2023-07-06 01:14:31 +00:00			`sqlx = { version = "0.7", features = [`
Create cli binary Just has `add-feed` command so far. 2023-05-09 04:08:55 +00:00			`"runtime-tokio-native-tls",`
			`"postgres",`
			`"macros",`
			`"migrate",`
			`"chrono",`
Better database layout with uuid primary keys Serialize and deserialize the uuid ids as base62 strings in the URLs. 2023-06-27 18:03:52 +00:00			`"uuid",`
Create cli binary Just has `add-feed` command so far. 2023-05-09 04:08:55 +00:00			`] }`
Improve error handling 2023-05-07 22:20:19 +00:00			`thiserror = "1"`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`tokio = { version = "1", features = ["full"] }`
Complete log stream implementation Sets up a watch channel to send tracing lines from tracing-subscriber to receivers in a axum handler which streams Server Sent Events to any number of connected /log/stream clients. 2023-06-03 23:03:58 +00:00			`tokio-stream = { version = "0.1", features = ["sync"] }`
Initial commit with basic axum and sqlx API 2023-05-07 21:41:45 +00:00			`tower = "0.4"`
Add static file server, CSS styles for home Also fix the livereloading. 2023-06-09 05:02:50 +00:00			`tower-livereload = "0.8"`
			`tower-http = { version = "0.4", features = ["trace", "fs"] }`
Add an async actor FeedCrawler for fetching feed details Currently, this allows the browser to subscribe to the response of the asynchronous crawl after they add a new feed. Eventually I will also use this in the main scheduled crawls. Right now, it only upserts feed metadata based on the parsed feed. 2023-07-10 01:18:19 +00:00			`tracing = { version = "0.1", features = ["valuable", "attributes"] }`
Add /log page that displays server log This will eventually be behind an authorization guard so that only I have access, but it's useful to have for monitoring and testing out turbo streams. 2023-06-03 05:09:25 +00:00			`tracing-appender = "0.2"`
Add very basic crawl job Loops through feeds and adds items from each feed. 2023-05-10 03:46:42 +00:00			`tracing-subscriber = { version = "0.3", features = ["env-filter"] }`
Better database layout with uuid primary keys Serialize and deserialize the uuid ids as base62 strings in the URLs. 2023-06-27 18:03:52 +00:00			`uuid = { version = "1.3", features = ["serde"] }`
Add domain to entry list 2023-06-27 18:03:52 +00:00			`url = "2.4"`
Improve error handling 2023-05-07 22:20:19 +00:00			`validator = { version = "0.16", features = ["derive"] }`