thallada/crawlnicle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
103 Commits 3 Branches 0 Tags
4a5d514cc79149fac31a8f00fca6874852f0548e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
thallada 4a5d514cc7 Make stdout logging pretty 2024-05-11 16:43:22 -04:00
thallada 27ab839df4 Update frontend packages 2024-05-11 16:02:22 -04:00
thallada 3f97c0e2ca Upgrade packages and make sessions more secure 
Sign session cookies with base64 encoded secret from .env
 2024-05-09 00:45:46 -04:00
thallada 2fab68241e Add last/next crawl times to feeds, improve local time rendering 2024-02-01 00:13:09 -05:00
thallada 457aafbfe3 Fix opml import form 2024-01-31 22:32:54 -05:00
thallada 2e5ff1337c Fix crawl_feed_interval_minutes update logic 
Was just going down to minimum for all feeds that weren't updating. I need to use the current timestamp if there is no new entry.
 2024-01-31 22:25:05 -05:00
thallada 4eee21caed Switch to tailwind for css styling 
Could use more partials to reduce some of the current repetition (especially forms), but this is a start with everything converted.
 2024-01-07 19:45:20 -05:00
thallada 89f37279e5 Package updates & don't load livereload snippet on hx-requests 2024-01-03 00:07:41 -05:00
thallada c9a631a1f2 Update all packages, switch to tower-sessions 2023-12-20 00:42:13 -05:00
thallada 6c23b3aaa3 Cleanup the other auth forms 
Somewhat kinda progressively enhanced, but at least I'm using page partials now... mostly.
 2023-12-19 01:33:19 -05:00
thallada 7abffb2729 Progressively enhanced register form 
Thanks copilot for all the shit I didn't have to write.
 2023-12-18 22:31:33 -05:00
thallada 5881412b59 Login & add feed form progressive enhancement 2023-12-16 01:03:04 -05:00
thallada ea4784ff07 Fix login and add feed form 
Add feed form still WIP
 2023-11-12 18:40:05 -05:00
thallada 575f7d45b9 Restart the server on deploy 2023-10-24 23:00:02 -04:00
thalladaandGitHub 6980ff5f9f Turn off verbose logging in deploy.yml 2023-10-20 01:26:00 -04:00
thalladaandGitHub 7bff966ac0 Just rsync backend binary in deploy.yml 2023-10-20 01:12:30 -04:00
thallada d6946573a4 Fix cargo-bin cache in deploy.yml 2023-10-20 00:18:03 -04:00
thallada 3cb33a5d34 Build and ship frontend in deploy.yml 2023-10-20 00:16:08 -04:00
thallada 68fed43a5a Cache sqlx-cli between deploy runs 2023-10-19 23:54:06 -04:00
thallada b6d33a0f62 Remove sqlx prepare --check from deploy.yml 
Since it requires a running database for some reason?
 2023-10-19 23:46:56 -04:00
thallada 4b729de5fe Install sqlx-cli in deploy.yml 2023-10-19 23:32:02 -04:00
thallada f8a717d92f Prepare sqlx queries and add content dir 2023-10-19 23:26:10 -04:00
thalladaandGitHub 1305cf490a Create deploy.yml for deploying via Github actions 2023-10-19 22:53:09 -04:00
thallada 8feb2ec10e Prevent decoding empty uuid strings 2023-10-17 01:09:17 -04:00
thallada 5f9d64f2d9 Prevent Base62Uuid decoding from panicing 
An extra long encoded uuid could crash the server, now it's handled as
an error.
 2023-10-17 00:59:01 -04:00
thallada 7f86612899 Switch to async-fred-session, default config vals 
Fixup some issues in README too
 2023-10-17 00:15:44 -04:00
thallada 835e9dc748 Do reset password operations in db transaction 
And modify signature of model methods to accept an executor instead of a
pool connection which will also allow transactions.
 2023-10-13 14:44:40 +02:00
thallada 60671d5865 Complete reset password flow 2023-10-13 14:07:38 +02:00
thallada d5c5185351 Fix sessions persisting across server restart 
Need a non-random session secret.
 2023-10-07 00:54:15 -04:00
thallada 609f6d3d9f Fixes to email verification process 
Also make registration form progressively enhanced.
 2023-10-06 17:50:08 -04:00
thallada e59c6d596e Finish up email verification form 2023-09-29 23:48:37 -04:00
thallada c95334a7e2 Email verification form submit 2023-09-29 20:56:43 -04:00
thallada cdc8eb9b02 Basic email verification done 
Still need to add forms to confirmation page to allow resending and add
rate limiting.
 2023-09-28 23:53:46 -04:00
thallada f938a6b46b WIP email sending for registration 2023-09-27 23:11:52 -04:00
thallada 8d1bffc899 Update bun types 2023-09-27 22:41:10 -04:00
thallada a72bfa15bd Use HX-Target, not HX-Boost for Layout render 
HX-Boost is not always sent in all AJAX requests that htmx sends, only
those initiated by an element with hx-boost enabled. It was not showing
up on requests following an HX-Redirect response.

After reading the docs more, I realized HX-Target was what I wanted. If
I can see that the request is targeting `#main-content` then I know to
only return HTML inside that element. Simple.
 2023-09-27 01:28:53 -04:00
thallada bea3529e22 Rename signup to register and center forms 2023-09-27 00:23:28 -04:00
thallada 092a38ad52 Add hx-boosted support to Layout 
It will now skip rendering the layout if the request is coming from an
hx-boosted link or form and only update the head title with a
hx-swap-oob.
 2023-09-26 23:36:31 -04:00
thallada 81b4ef860e Fix login/signup redirection 
HTMX needs special treatment for redirects on submit.
 2023-09-26 01:48:05 -04:00
thallada 6fd2f150a0 Store sessions in redis 2023-09-26 01:47:50 -04:00
thallada 306059c355 Add basic user auth 2023-09-25 01:38:18 -04:00
thallada ec394fc170 Implement entry and feed pagination 2023-09-02 14:01:18 -04:00
thallada 0607b46283 Add a footer 2023-09-01 20:13:44 -04:00
thallada a8742c882e Install htmx via bun instead of unpkg CDN 2023-09-01 19:59:56 -04:00
thallada e50e165b87 Remove dbg! statements 2023-09-01 19:59:56 -04:00
thalladaandGitHub 21cae62714 Add bun to prereqs in README.md 2023-09-01 01:23:52 -04:00
thallada ac12ca0fd9 Add eslint and prettier to frontend 
And format files
 2023-09-01 01:05:06 -04:00
thallada 1d6f98c6bb Replace hotwire with htmx 
In the process, also improve the feedback from the import/add feed
forms.

I also replaced the frontend code to replace utc timestamps with local
time strings with @hotwired/stimulus with vanilla js.
 2023-09-01 00:25:05 -04:00
thallada ff0b218da1 Adjust entry html/css 2023-08-29 23:36:05 -04:00
thallada 69ed8aa8bb Use ceil in calculating crawl_interval 
So that it can increase from 1 minute.
 2023-08-29 23:32:05 -04:00
thallada 276f0e17a8 Remove self from Crawls and Imports in actors at end of task 
In case the user never listens to the stream so that I do not create
inifinitely growing hashmaps in the server memory.
 2023-08-29 23:30:00 -04:00
thallada ceac234ce7 Sanitize entry html content with ammonia 2023-08-29 23:04:35 -04:00
thallada 2f39be4152 Finish implemtning OPML importer 
Now with progress messages!
 2023-08-29 00:35:19 -04:00
thallada eddf39b62e Added importer actor, file upload still not working 2023-07-22 22:54:57 -04:00
thallada e6a37703be Use tokio::sync::mutex for Crawls 2023-07-19 23:03:35 -04:00
thallada d17f909312 Add CrawlScheduler actor, shared client w/ last modified headers 2023-07-15 21:40:31 -04:00
thallada 02d5cb9976 Remove article_scraper crate 2023-07-15 02:31:45 -04:00
thallada ea67690d1d Start feed crawlers at startup 2023-07-15 02:30:25 -04:00
thallada 0dfde7cd31 Begin adding pagination for feeds 2023-07-15 02:12:59 -04:00
thallada ae95921966 Update feed last_crawl_error & crawl_interval_minutes 2023-07-15 01:30:45 -04:00
thallada 4837cbb903 Add crawl metadata to feed & improve model interface 2023-07-15 00:40:10 -04:00
thallada 0fa0cfc508 Only crawl new entries 2023-07-14 21:27:23 -04:00
thallada 923776d7a1 Add DomainLocks to serialize requests to each domain 
All async tasks must wait 1 second since the last request to make
another to the same domain.
 2023-07-14 00:55:35 -04:00
thallada b7efc61cfc Add EntryCrawler that uses readability lib 
Actors delegating to actors baybeeee
 2023-07-11 01:52:38 -04:00
thallada f13c7e5e70 Add an async actor FeedCrawler for fetching feed details 
Currently, this allows the browser to subscribe to the response of the
asynchronous crawl after they add a new feed.

Eventually I will also use this in the main scheduled crawls. Right now,
it only upserts feed metadata based on the parsed feed.
 2023-07-09 21:30:23 -04:00
thallada 8f4db1d8d9 Button padding tweak 2023-07-08 01:58:33 -04:00
thallada b94555d346 Delete feed button 
And a non-functional edit button
 2023-07-08 01:52:15 -04:00
thallada 478e72d8f0 Add feed form 2023-07-08 01:05:49 -04:00
thallada f69d0f2752 Improve feed list and add feed page 2023-07-07 16:03:59 -04:00
thallada 3f028c3088 Store entry html content outside DB in file storage 
Since the HTML content can get quite big and can have embeded images.
 2023-07-05 23:45:49 -04:00
thallada 7289151318 Upgrade sqlx to 0.7 + some tweaks 2023-07-05 21:14:31 -04:00
thallada 5169fac218 Reload frontend changes faster in dev mode 
Now frontend changes appear as fast as Bun can bundle them instead of
needing to wait for rust to compile too.
 2023-07-05 20:05:03 -04:00
thallada 74a205d51e Also bundle css with bun so it busts caches 
And updates the `build.rs` script so that it also builds a manifest.txt
for css files.
 2023-06-29 00:38:32 -04:00
thallada ec90069c30 Fixup the just watch command 
And add more docs about it.
 2023-06-29 00:00:26 -04:00
thallada 3f744c0c50 Add auto-reload for frontend and backend 
Also adds more just commands and improves the README docs.
 2023-06-28 01:23:11 -04:00
thallada 76cc87631f Add frontend JS and improve post styling 
Frontend is built with Bun. It uses Stimulus to progressively enhance
the server-built HTML. Currently, it only replaces UTC timestamps from
the server with the time in the browser's timezone.
 2023-06-27 14:03:52 -04:00
thallada 7e06d23bba Replace argh with clap 
Mostly for the more concise Config parsing and error handling.
 2023-06-27 14:03:52 -04:00
thallada abd540d2ff Better database layout with uuid primary keys 
Serialize and deserialize the uuid ids as base62 strings in the URLs.
 2023-06-27 14:03:52 -04:00
thallada 4e41bbd6e1 Add domain to entry list 2023-06-27 14:03:52 -04:00
thallada 97c4ae73f0 Add static file server, CSS styles for home 
Also fix the livereloading.
 2023-06-09 01:06:19 -04:00
thallada 758e644173 Add published_at to entries, begin to support pagination 
Articles will be sorted by their published_at dates for now.
 2023-06-08 01:20:21 -04:00
thallada 3f29138bd1 Fetch and save entry HTML content with metadata 
And render the extracted HTML on the entry page in the frontend.
 2023-06-07 01:06:03 -04:00
thallada 786f3a194f Streaming log page with colors 
Using the ansi-to-html crate.
 2023-06-06 21:14:29 -04:00
thallada 6713a7a440 Complete log stream implementation 
Sets up a watch channel to send tracing lines from tracing-subscriber to
receivers in a axum handler which streams Server Sent Events to any
number of connected /log/stream clients.
 2023-06-03 19:03:58 -04:00
thallada 951d6d23e2 Add /log page that displays server log 
This will eventually be behind an authorization guard so that only I
have access, but it's useful to have for monitoring and testing out
turbo streams.
 2023-06-03 01:09:25 -04:00
thallada ea236dff4e Add rudimentary entry page, fix main frame link navigation 2023-06-02 00:31:25 -04:00
thallada 0b7acadd60 Add main turbo-frame 
Nav link navigations now only replace the main content area while
leaving the header in place.
 2023-06-02 00:18:06 -04:00
thallada f4da3f3ab3 Adding title to config and utilizing it in layout 2023-06-02 00:08:20 -04:00
thallada effccfdbbc Add config extension module 
Better way to parse environment variables and pass around config vars.
 2023-06-01 23:01:11 -04:00
thallada a67ffbbbed Add layout and feeds page, add hotwire 2023-06-01 22:48:09 -04:00
thallada 74f353b894 Add (semi) live reloading for debug server 2023-06-01 00:59:46 -04:00
thallada e501c15c66 Add header partial 2023-06-01 00:59:39 -04:00
thallada 1698ef9c60 Add maud for frontend and move api to sub module 2023-06-01 00:03:00 -04:00
thallada 9059894021 Make titles optional on feeds and entries 2023-05-17 23:10:23 -04:00
thallada bf40b803a9 Rename item to entry 2023-05-17 23:10:09 -04:00
thallada a331e63466 Add README 2023-05-10 00:16:30 -04:00
thallada 6f364b4c44 Rename to crawlnicle 2023-05-10 00:00:48 -04:00
thallada ae8f15f19b Add very basic crawl job 
Loops through feeds and adds items from each feed.
 2023-05-09 23:55:42 -04:00
thallada 89fdf8f95a Create cli binary 
Just has `add-feed` command so far.
 2023-05-09 00:08:55 -04:00
thallada b2a5bf5882 Add feed model, link items to feeds 2023-05-07 21:25:22 -04:00