thallada/crawlnicle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
103 Commits 3 Branches 0 Tags
main
Commit Graph

103 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
thallada 276f0e17a8 Remove self from Crawls and Imports in actors at end of task 
In case the user never listens to the stream so that I do not create
inifinitely growing hashmaps in the server memory.
 2023-08-29 23:30:00 -04:00
thallada ceac234ce7 Sanitize entry html content with ammonia 2023-08-29 23:04:35 -04:00
thallada 2f39be4152 Finish implemtning OPML importer 
Now with progress messages!
 2023-08-29 00:35:19 -04:00
thallada eddf39b62e Added importer actor, file upload still not working 2023-07-22 22:54:57 -04:00
thallada e6a37703be Use tokio::sync::mutex for Crawls 2023-07-19 23:03:35 -04:00
thallada d17f909312 Add CrawlScheduler actor, shared client w/ last modified headers 2023-07-15 21:40:31 -04:00
thallada 02d5cb9976 Remove article_scraper crate 2023-07-15 02:31:45 -04:00
thallada ea67690d1d Start feed crawlers at startup 2023-07-15 02:30:25 -04:00
thallada 0dfde7cd31 Begin adding pagination for feeds 2023-07-15 02:12:59 -04:00
thallada ae95921966 Update feed last_crawl_error & crawl_interval_minutes 2023-07-15 01:30:45 -04:00
thallada 4837cbb903 Add crawl metadata to feed & improve model interface 2023-07-15 00:40:10 -04:00
thallada 0fa0cfc508 Only crawl new entries 2023-07-14 21:27:23 -04:00
thallada 923776d7a1 Add DomainLocks to serialize requests to each domain 
All async tasks must wait 1 second since the last request to make
another to the same domain.
 2023-07-14 00:55:35 -04:00
thallada b7efc61cfc Add EntryCrawler that uses readability lib 
Actors delegating to actors baybeeee
 2023-07-11 01:52:38 -04:00
thallada f13c7e5e70 Add an async actor FeedCrawler for fetching feed details 
Currently, this allows the browser to subscribe to the response of the
asynchronous crawl after they add a new feed.

Eventually I will also use this in the main scheduled crawls. Right now,
it only upserts feed metadata based on the parsed feed.
 2023-07-09 21:30:23 -04:00
thallada 8f4db1d8d9 Button padding tweak 2023-07-08 01:58:33 -04:00
thallada b94555d346 Delete feed button 
And a non-functional edit button
 2023-07-08 01:52:15 -04:00
thallada 478e72d8f0 Add feed form 2023-07-08 01:05:49 -04:00
thallada f69d0f2752 Improve feed list and add feed page 2023-07-07 16:03:59 -04:00
thallada 3f028c3088 Store entry html content outside DB in file storage 
Since the HTML content can get quite big and can have embeded images.
 2023-07-05 23:45:49 -04:00
thallada 7289151318 Upgrade sqlx to 0.7 + some tweaks 2023-07-05 21:14:31 -04:00
thallada 5169fac218 Reload frontend changes faster in dev mode 
Now frontend changes appear as fast as Bun can bundle them instead of
needing to wait for rust to compile too.
 2023-07-05 20:05:03 -04:00
thallada 74a205d51e Also bundle css with bun so it busts caches 
And updates the `build.rs` script so that it also builds a manifest.txt
for css files.
 2023-06-29 00:38:32 -04:00
thallada ec90069c30 Fixup the just watch command 
And add more docs about it.
 2023-06-29 00:00:26 -04:00
thallada 3f744c0c50 Add auto-reload for frontend and backend 
Also adds more just commands and improves the README docs.
 2023-06-28 01:23:11 -04:00
thallada 76cc87631f Add frontend JS and improve post styling 
Frontend is built with Bun. It uses Stimulus to progressively enhance
the server-built HTML. Currently, it only replaces UTC timestamps from
the server with the time in the browser's timezone.
 2023-06-27 14:03:52 -04:00
thallada 7e06d23bba Replace argh with clap 
Mostly for the more concise Config parsing and error handling.
 2023-06-27 14:03:52 -04:00
thallada abd540d2ff Better database layout with uuid primary keys 
Serialize and deserialize the uuid ids as base62 strings in the URLs.
 2023-06-27 14:03:52 -04:00
thallada 4e41bbd6e1 Add domain to entry list 2023-06-27 14:03:52 -04:00
thallada 97c4ae73f0 Add static file server, CSS styles for home 
Also fix the livereloading.
 2023-06-09 01:06:19 -04:00
thallada 758e644173 Add published_at to entries, begin to support pagination 
Articles will be sorted by their published_at dates for now.
 2023-06-08 01:20:21 -04:00
thallada 3f29138bd1 Fetch and save entry HTML content with metadata 
And render the extracted HTML on the entry page in the frontend.
 2023-06-07 01:06:03 -04:00
thallada 786f3a194f Streaming log page with colors 
Using the ansi-to-html crate.
 2023-06-06 21:14:29 -04:00
thallada 6713a7a440 Complete log stream implementation 
Sets up a watch channel to send tracing lines from tracing-subscriber to
receivers in a axum handler which streams Server Sent Events to any
number of connected /log/stream clients.
 2023-06-03 19:03:58 -04:00
thallada 951d6d23e2 Add /log page that displays server log 
This will eventually be behind an authorization guard so that only I
have access, but it's useful to have for monitoring and testing out
turbo streams.
 2023-06-03 01:09:25 -04:00
thallada ea236dff4e Add rudimentary entry page, fix main frame link navigation 2023-06-02 00:31:25 -04:00
thallada 0b7acadd60 Add main turbo-frame 
Nav link navigations now only replace the main content area while
leaving the header in place.
 2023-06-02 00:18:06 -04:00
thallada f4da3f3ab3 Adding title to config and utilizing it in layout 2023-06-02 00:08:20 -04:00
thallada effccfdbbc Add config extension module 
Better way to parse environment variables and pass around config vars.
 2023-06-01 23:01:11 -04:00
thallada a67ffbbbed Add layout and feeds page, add hotwire 2023-06-01 22:48:09 -04:00
thallada 74f353b894 Add (semi) live reloading for debug server 2023-06-01 00:59:46 -04:00
thallada e501c15c66 Add header partial 2023-06-01 00:59:39 -04:00
thallada 1698ef9c60 Add maud for frontend and move api to sub module 2023-06-01 00:03:00 -04:00
thallada 9059894021 Make titles optional on feeds and entries 2023-05-17 23:10:23 -04:00
thallada bf40b803a9 Rename item to entry 2023-05-17 23:10:09 -04:00
thallada a331e63466 Add README 2023-05-10 00:16:30 -04:00
thallada 6f364b4c44 Rename to crawlnicle 2023-05-10 00:00:48 -04:00
thallada ae8f15f19b Add very basic crawl job 
Loops through feeds and adds items from each feed.
 2023-05-09 23:55:42 -04:00
thallada 89fdf8f95a Create cli binary 
Just has `add-feed` command so far.
 2023-05-09 00:08:55 -04:00
thallada b2a5bf5882 Add feed model, link items to feeds 2023-05-07 21:25:22 -04:00