thallada/crawlnicle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
My personalized news and blog aggregator
28 Commits 3 Branches 0 Tags 780 KiB
Rust 95.5%
PLpgSQL 1.7%
TypeScript 1.6%
Just 0.9%
JavaScript 0.2%
76cc87631f
Go to file
Tyler Hallada 76cc87631f Add frontend JS and improve post styling 
Frontend is built with Bun. It uses Stimulus to progressively enhance
the server-built HTML. Currently, it only replaces UTC timestamps from
the server with the time in the browser's timezone.
2023-06-27 14:03:52 -04:00
frontend Add frontend JS and improve post styling 2023-06-27 14:03:52 -04:00
migrations Better database layout with uuid primary keys 2023-06-27 14:03:52 -04:00
src Add frontend JS and improve post styling 2023-06-27 14:03:52 -04:00
static Add frontend JS and improve post styling 2023-06-27 14:03:52 -04:00
.gitignore Add frontend JS and improve post styling 2023-06-27 14:03:52 -04:00
build.rs Add frontend JS and improve post styling 2023-06-27 14:03:52 -04:00
Cargo.lock Replace argh with clap 2023-06-27 14:03:52 -04:00
Cargo.toml Replace argh with clap 2023-06-27 14:03:52 -04:00
drop_all.sql Better database layout with uuid primary keys 2023-06-27 14:03:52 -04:00
justfile Add frontend JS and improve post styling 2023-06-27 14:03:52 -04:00
README.md Add /log page that displays server log 2023-06-03 01:09:25 -04:00

README.md

crawlnicle

My personalized news and blog aggregator. Taking back control over the algorithm. Pining for the days of Google Reader. An excuse to write more Rust.

Development Install

  1. Install and run postgres.
  2. Create postgres user and database:
createuser crawlnicle
createdb crawlnicle
sudo -u postgres -i psql
postgres=# ALTER DATABASE crawlnicle OWNER TO crawlnicle;
\password crawlnicle

# Or, on Windows in PowerShell:

& 'C:\Program Files\PostgreSQL\13\bin\createuser.exe' -U postgres crawlnicle
& 'C:\Program Files\PostgreSQL\13\bin\createdb.exe' -U postgres crawlnicle
& 'C:\Program Files\PostgreSQL\13\bin\psql.exe' -U postgres
postgres=# ALTER DATABASE crawlnicle OWNER TO crawlnicle;
\password crawlnicle
  1. Save password somewhere safe and then and add a .env file to the project directory with the contents:
RUST_LOG=crawlnicle=debug,cli=debug,lib=debug,tower_http=debug,sqlx=debug
HOST=127.0.0.1
PORT=3000
DATABASE_URL=postgresql://crawlnicle:<password>@localhost/crawlnicle
DATABASE_MAX_CONNECTIONS=5
TITLE=crawlnicle
MAX_MEM_LOG_SIZE=1000000
  1. Install sqlx_cli with cargo install sqlx-cli --no-default-features --features native-tls,postgres
  2. Run sqlx migrate run which will run all the database migrations.
  3. Build the release binary by running cargo build --release.
  4. Run ./target/build/crawlnicle to start the server.

Using the CLI

This project also comes with a CLI binary which allows you to manipulate the database directly without needing to go through the REST API server. Run ./target/build/cli --help to see all of the available commands.

Running jobs

To periodically fetch new items from all of the feeds execute the cli crawl command in a cronjob.