thallada/crawlnicle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
73 Commits 3 Branches 0 Tags
e59c6d596e16d7198a77bc14356e2b1a1cd86904
Commit Graph

73 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
thallada e59c6d596e Finish up email verification form 2023-09-29 23:48:37 -04:00
thallada c95334a7e2 Email verification form submit 2023-09-29 20:56:43 -04:00
thallada cdc8eb9b02 Basic email verification done 
Still need to add forms to confirmation page to allow resending and add
rate limiting.
 2023-09-28 23:53:46 -04:00
thallada f938a6b46b WIP email sending for registration 2023-09-27 23:11:52 -04:00
thallada 8d1bffc899 Update bun types 2023-09-27 22:41:10 -04:00
thallada a72bfa15bd Use HX-Target, not HX-Boost for Layout render 
HX-Boost is not always sent in all AJAX requests that htmx sends, only
those initiated by an element with hx-boost enabled. It was not showing
up on requests following an HX-Redirect response.

After reading the docs more, I realized HX-Target was what I wanted. If
I can see that the request is targeting `#main-content` then I know to
only return HTML inside that element. Simple.
 2023-09-27 01:28:53 -04:00
thallada bea3529e22 Rename signup to register and center forms 2023-09-27 00:23:28 -04:00
thallada 092a38ad52 Add hx-boosted support to Layout 
It will now skip rendering the layout if the request is coming from an
hx-boosted link or form and only update the head title with a
hx-swap-oob.
 2023-09-26 23:36:31 -04:00
thallada 81b4ef860e Fix login/signup redirection 
HTMX needs special treatment for redirects on submit.
 2023-09-26 01:48:05 -04:00
thallada 6fd2f150a0 Store sessions in redis 2023-09-26 01:47:50 -04:00
thallada 306059c355 Add basic user auth 2023-09-25 01:38:18 -04:00
thallada ec394fc170 Implement entry and feed pagination 2023-09-02 14:01:18 -04:00
thallada 0607b46283 Add a footer 2023-09-01 20:13:44 -04:00
thallada a8742c882e Install htmx via bun instead of unpkg CDN 2023-09-01 19:59:56 -04:00
thallada e50e165b87 Remove dbg! statements 2023-09-01 19:59:56 -04:00
thallada 21cae62714 Add bun to prereqs in README.md 2023-09-01 01:23:52 -04:00
thallada ac12ca0fd9 Add eslint and prettier to frontend 
And format files
 2023-09-01 01:05:06 -04:00
thallada 1d6f98c6bb Replace hotwire with htmx 
In the process, also improve the feedback from the import/add feed
forms.

I also replaced the frontend code to replace utc timestamps with local
time strings with @hotwired/stimulus with vanilla js.
 2023-09-01 00:25:05 -04:00
thallada ff0b218da1 Adjust entry html/css 2023-08-29 23:36:05 -04:00
thallada 69ed8aa8bb Use ceil in calculating crawl_interval 
So that it can increase from 1 minute.
 2023-08-29 23:32:05 -04:00
thallada 276f0e17a8 Remove self from Crawls and Imports in actors at end of task 
In case the user never listens to the stream so that I do not create
inifinitely growing hashmaps in the server memory.
 2023-08-29 23:30:00 -04:00
thallada ceac234ce7 Sanitize entry html content with ammonia 2023-08-29 23:04:35 -04:00
thallada 2f39be4152 Finish implemtning OPML importer 
Now with progress messages!
 2023-08-29 00:35:19 -04:00
thallada eddf39b62e Added importer actor, file upload still not working 2023-07-22 22:54:57 -04:00
thallada e6a37703be Use tokio::sync::mutex for Crawls 2023-07-19 23:03:35 -04:00
thallada d17f909312 Add CrawlScheduler actor, shared client w/ last modified headers 2023-07-15 21:40:31 -04:00
thallada 02d5cb9976 Remove article_scraper crate 2023-07-15 02:31:45 -04:00
thallada ea67690d1d Start feed crawlers at startup 2023-07-15 02:30:25 -04:00
thallada 0dfde7cd31 Begin adding pagination for feeds 2023-07-15 02:12:59 -04:00
thallada ae95921966 Update feed last_crawl_error & crawl_interval_minutes 2023-07-15 01:30:45 -04:00
thallada 4837cbb903 Add crawl metadata to feed & improve model interface 2023-07-15 00:40:10 -04:00
thallada 0fa0cfc508 Only crawl new entries 2023-07-14 21:27:23 -04:00
thallada 923776d7a1 Add DomainLocks to serialize requests to each domain 
All async tasks must wait 1 second since the last request to make
another to the same domain.
 2023-07-14 00:55:35 -04:00
thallada b7efc61cfc Add EntryCrawler that uses readability lib 
Actors delegating to actors baybeeee
 2023-07-11 01:52:38 -04:00
thallada f13c7e5e70 Add an async actor FeedCrawler for fetching feed details 
Currently, this allows the browser to subscribe to the response of the
asynchronous crawl after they add a new feed.

Eventually I will also use this in the main scheduled crawls. Right now,
it only upserts feed metadata based on the parsed feed.
 2023-07-09 21:30:23 -04:00
thallada 8f4db1d8d9 Button padding tweak 2023-07-08 01:58:33 -04:00
thallada b94555d346 Delete feed button 
And a non-functional edit button
 2023-07-08 01:52:15 -04:00
thallada 478e72d8f0 Add feed form 2023-07-08 01:05:49 -04:00
thallada f69d0f2752 Improve feed list and add feed page 2023-07-07 16:03:59 -04:00
thallada 3f028c3088 Store entry html content outside DB in file storage 
Since the HTML content can get quite big and can have embeded images.
 2023-07-05 23:45:49 -04:00
thallada 7289151318 Upgrade sqlx to 0.7 + some tweaks 2023-07-05 21:14:31 -04:00
thallada 5169fac218 Reload frontend changes faster in dev mode 
Now frontend changes appear as fast as Bun can bundle them instead of
needing to wait for rust to compile too.
 2023-07-05 20:05:03 -04:00
thallada 74a205d51e Also bundle css with bun so it busts caches 
And updates the `build.rs` script so that it also builds a manifest.txt
for css files.
 2023-06-29 00:38:32 -04:00
thallada ec90069c30 Fixup the just watch command 
And add more docs about it.
 2023-06-29 00:00:26 -04:00
thallada 3f744c0c50 Add auto-reload for frontend and backend 
Also adds more just commands and improves the README docs.
 2023-06-28 01:23:11 -04:00
thallada 76cc87631f Add frontend JS and improve post styling 
Frontend is built with Bun. It uses Stimulus to progressively enhance
the server-built HTML. Currently, it only replaces UTC timestamps from
the server with the time in the browser's timezone.
 2023-06-27 14:03:52 -04:00
thallada 7e06d23bba Replace argh with clap 
Mostly for the more concise Config parsing and error handling.
 2023-06-27 14:03:52 -04:00
thallada abd540d2ff Better database layout with uuid primary keys 
Serialize and deserialize the uuid ids as base62 strings in the URLs.
 2023-06-27 14:03:52 -04:00
thallada 4e41bbd6e1 Add domain to entry list 2023-06-27 14:03:52 -04:00
thallada 97c4ae73f0 Add static file server, CSS styles for home 
Also fix the livereloading.
 2023-06-09 01:06:19 -04:00