69ed8aa8bb
Use ceil in calculating crawl_interval
...
So that it can increase from 1 minute.
2023-08-29 23:32:05 -04:00
276f0e17a8
Remove self from Crawls and Imports in actors at end of task
...
In case the user never listens to the stream so that I do not create
inifinitely growing hashmaps in the server memory.
2023-08-29 23:30:00 -04:00
ceac234ce7
Sanitize entry html content with ammonia
2023-08-29 23:04:35 -04:00
2f39be4152
Finish implemtning OPML importer
...
Now with progress messages!
2023-08-29 00:35:19 -04:00
eddf39b62e
Added importer actor, file upload still not working
2023-07-22 22:54:57 -04:00
e6a37703be
Use tokio::sync::mutex for Crawls
2023-07-19 23:03:35 -04:00
d17f909312
Add CrawlScheduler actor, shared client w/ last modified headers
2023-07-15 21:40:31 -04:00
02d5cb9976
Remove article_scraper crate
2023-07-15 02:31:45 -04:00
ea67690d1d
Start feed crawlers at startup
2023-07-15 02:30:25 -04:00
0dfde7cd31
Begin adding pagination for feeds
2023-07-15 02:12:59 -04:00
ae95921966
Update feed last_crawl_error & crawl_interval_minutes
2023-07-15 01:30:45 -04:00
4837cbb903
Add crawl metadata to feed & improve model interface
2023-07-15 00:40:10 -04:00
0fa0cfc508
Only crawl new entries
2023-07-14 21:27:23 -04:00
923776d7a1
Add DomainLocks to serialize requests to each domain
...
All async tasks must wait 1 second since the last request to make
another to the same domain.
2023-07-14 00:55:35 -04:00
b7efc61cfc
Add EntryCrawler that uses readability lib
...
Actors delegating to actors baybeeee
2023-07-11 01:52:38 -04:00
f13c7e5e70
Add an async actor FeedCrawler for fetching feed details
...
Currently, this allows the browser to subscribe to the response of the
asynchronous crawl after they add a new feed.
Eventually I will also use this in the main scheduled crawls. Right now,
it only upserts feed metadata based on the parsed feed.
2023-07-09 21:30:23 -04:00
8f4db1d8d9
Button padding tweak
2023-07-08 01:58:33 -04:00
b94555d346
Delete feed button
...
And a non-functional edit button
2023-07-08 01:52:15 -04:00
478e72d8f0
Add feed form
2023-07-08 01:05:49 -04:00
f69d0f2752
Improve feed list and add feed page
2023-07-07 16:03:59 -04:00
3f028c3088
Store entry html content outside DB in file storage
...
Since the HTML content can get quite big and can have embeded images.
2023-07-05 23:45:49 -04:00
7289151318
Upgrade sqlx to 0.7 + some tweaks
2023-07-05 21:14:31 -04:00
5169fac218
Reload frontend changes faster in dev mode
...
Now frontend changes appear as fast as Bun can bundle them instead of
needing to wait for rust to compile too.
2023-07-05 20:05:03 -04:00
74a205d51e
Also bundle css with bun so it busts caches
...
And updates the `build.rs` script so that it also builds a manifest.txt
for css files.
2023-06-29 00:38:32 -04:00
ec90069c30
Fixup the just watch command
...
And add more docs about it.
2023-06-29 00:00:26 -04:00
3f744c0c50
Add auto-reload for frontend and backend
...
Also adds more just commands and improves the README docs.
2023-06-28 01:23:11 -04:00
76cc87631f
Add frontend JS and improve post styling
...
Frontend is built with Bun. It uses Stimulus to progressively enhance
the server-built HTML. Currently, it only replaces UTC timestamps from
the server with the time in the browser's timezone.
2023-06-27 14:03:52 -04:00
7e06d23bba
Replace argh with clap
...
Mostly for the more concise Config parsing and error handling.
2023-06-27 14:03:52 -04:00
abd540d2ff
Better database layout with uuid primary keys
...
Serialize and deserialize the uuid ids as base62 strings in the URLs.
2023-06-27 14:03:52 -04:00
4e41bbd6e1
Add domain to entry list
2023-06-27 14:03:52 -04:00
97c4ae73f0
Add static file server, CSS styles for home
...
Also fix the livereloading.
2023-06-09 01:06:19 -04:00
758e644173
Add published_at to entries, begin to support pagination
...
Articles will be sorted by their published_at dates for now.
2023-06-08 01:20:21 -04:00
3f29138bd1
Fetch and save entry HTML content with metadata
...
And render the extracted HTML on the entry page in the frontend.
2023-06-07 01:06:03 -04:00
786f3a194f
Streaming log page with colors
...
Using the ansi-to-html crate.
2023-06-06 21:14:29 -04:00
6713a7a440
Complete log stream implementation
...
Sets up a watch channel to send tracing lines from tracing-subscriber to
receivers in a axum handler which streams Server Sent Events to any
number of connected /log/stream clients.
2023-06-03 19:03:58 -04:00
951d6d23e2
Add /log page that displays server log
...
This will eventually be behind an authorization guard so that only I
have access, but it's useful to have for monitoring and testing out
turbo streams.
2023-06-03 01:09:25 -04:00
ea236dff4e
Add rudimentary entry page, fix main frame link navigation
2023-06-02 00:31:25 -04:00
0b7acadd60
Add main turbo-frame
...
Nav link navigations now only replace the main content area while
leaving the header in place.
2023-06-02 00:18:06 -04:00
f4da3f3ab3
Adding title to config and utilizing it in layout
2023-06-02 00:08:20 -04:00
effccfdbbc
Add config extension module
...
Better way to parse environment variables and pass around config vars.
2023-06-01 23:01:11 -04:00
a67ffbbbed
Add layout and feeds page, add hotwire
2023-06-01 22:48:09 -04:00
74f353b894
Add (semi) live reloading for debug server
2023-06-01 00:59:46 -04:00
e501c15c66
Add header partial
2023-06-01 00:59:39 -04:00
1698ef9c60
Add maud for frontend and move api to sub module
2023-06-01 00:03:00 -04:00
9059894021
Make titles optional on feeds and entries
2023-05-17 23:10:23 -04:00
bf40b803a9
Rename item to entry
2023-05-17 23:10:09 -04:00
a331e63466
Add README
2023-05-10 00:16:30 -04:00
6f364b4c44
Rename to crawlnicle
2023-05-10 00:00:48 -04:00
ae8f15f19b
Add very basic crawl job
...
Loops through feeds and adds items from each feed.
2023-05-09 23:55:42 -04:00
89fdf8f95a
Create cli binary
...
Just has `add-feed` command so far.
2023-05-09 00:08:55 -04:00
