Commit graph

64 commits

Author SHA1 Message Date
phiresky
9dc4de3cf2 federation: parallel sending per instance (#4623)
* federation: parallel sending

* federation: some comments

* lint and set force_write true when a request fails

* inbox_urls return vec

* split inbox functions into separate file

* cleanup

* extract sending task code to separate file

* move federation concurrent config to config file

* off by one issue

* improve msg

* fix both permanent stopping of federation queues and multiple creation of the same federation queues

* fix after merge

* lint fix

* Update crates/federate/src/send.rs

Co-authored-by: dullbananas <dull.bananas0@gmail.com>

* comment about reverse ordering

* remove crashable, comment

* comment

* move comment

* run federation tests twice

* fix test run

* prettier

* fix config default

* upgrade rust to 1.78 to fix diesel cli

* fix clippy

* delay

* add debug to make localhost urls not valid in ap crate, add some debug logs

* federation tests: ensure server stop after test and random activity id

* ci fix

* add test to federate 100 events

* fix send 100 test

* different data every time so activities are distinguishable

* allow out of order receives in test

* lint

* comment about https://github.com/LemmyNet/lemmy/pull/4623#discussion_r1565437391

* move sender for clarity, add comment

* move more things to members

* update test todo comment, use same env var as worker test but default to 1

* remove else below continue

* some more cleanup

* handle todo about smooth exit

* add federate inboxes collector tests

* lint

* actor max length

* don't reset fail count if activity skipped

* fix some comments

* reuse vars

* format

* Update .woodpecker.yml

* fix recheck time

* fix inboxes tests under fast mode

* format

* make i32 and ugly casts

* clippy

---------

Co-authored-by: dullbananas <dull.bananas0@gmail.com>
2024-07-23 12:20:14 +02:00
Nutomic
c96017c009
Configure max comment width in clippy (#4738)
* Configure max comment width in clippy

* update default config
2024-05-23 08:46:26 -04:00
TechVest
b0370ae2fd
chore: fix some comments (#4637)
Signed-off-by: TechVest <techdashen@qq.com>
2024-04-17 14:35:54 +02:00
Nutomic
e8a52d3a5c
Rewrite images to use local proxy (#4035)
* Add markdown rule to add rel=nofollow for all links

* Add markdown image rule to add local image proxy (fixes #1036)

* comments

* rewrite markdown image links working

* add comment

* perform markdown image processing in api/apub receivers

* clippy

* add db table to validate proxied links

* rewrite link fields for avatar, banner etc

* sql fmt

* proxy links received over federation

* add config option

* undo post.url rewriting, move http route definition

* add tests

* proxy images through pictrs

* testing

* cleanup request.rs file

* more cleanup (fixes #2611)

* include url content type when sending post over apub (fixes #2611)

* store post url content type in db

* should be media_type

* get rid of cache_remote_thumbnails setting, instead automatically
take thumbnail from federation data if available.

* fix tests

* add setting disable_external_link_previews

* federate post url as image depending on mime type

* change setting again

* machete

* invert

* support custom emoji

* clippy

* update defaults

* add image proxy test, fix test

* fix test

* clippy

* revert accidental changes

* address review

* clippy

* Markdown link rule-dess (#4356)

* Extracting opengraph_data to its own type.

* A few additions for markdown-link-rule.

---------

Co-authored-by: Nutomic <me@nutomic.com>

* fix setting

* use enum for image proxy setting

* fix test configs

* add config backwards compat

* clippy

* machete

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-25 09:22:11 -05:00
Nutomic
7d9b59c467
Reduce default db pool size to 30, remove db timeout (ref #4282) (#4301)
* Reduce default db pool size to 30 (ref #4282)

* remove db timeout
2024-01-04 12:16:51 -05:00
Nutomic
a790a24c4d
Revert debug auth (#4232)
* Revert "Some changes to help debug auth problems on lemmy.ml (#4220)"

This reverts commit 16ac893e15.

* Rename auth cookie back to jwt
2023-12-05 11:22:08 -05:00
Nutomic
16ac893e15
Some changes to help debug auth problems on lemmy.ml (#4220)
* Some changes to help debug auth problems on lemmy.ml

* fix

* clippy
2023-12-01 09:18:29 -05:00
İsmail Karslı
2070381e81
Add upload_timeout to PictrsConfig (#4159)
* Add upload timeout to PictrsConfig

* Bad space 🤔

* Update PictrsConfig upload timeout to include units.
2023-11-17 04:01:32 -05:00
Nutomic
1d23df37d8
Rename cache_remote_images setting to cache_external_link_previews (ref #4035) (#4148) 2023-11-13 09:19:09 -05:00
Dessalines
1b751a8cac
Adding cors_origin to settings. Fixes #3665 (#4095)
* Adding cors_origin to settings. Fixes #3665

* Fix result to option.

* Forgot to update config defaults.

* Setting a cors origin doku default.

* Adding comments for CORS.
2023-10-25 16:46:34 +02:00
Nutomic
50b7322ff3
Mention max length for site name in config (fixes #3006) (#4010)
* Mention max length for site name in config (fixes #3006)

* ci
2023-10-04 15:09:25 +02:00
Nutomic
9bb99bec5e
Increase default database connections (fixes #3394) (#3994) 2023-09-28 09:56:39 -04:00
Nick Webster
a57658d99c
Adding a new config flag to disable pictrs caching for thumbnails (#3897)
* add logic to prevent downloading remote pictrs images

* apply formatting

* Do not attempt a pictrs fetch if the remote image is also on a pictrs instance

* Do not attempt a pictrs fetch if the remote image is also on a pictrs instance and cache_federated_images is false

* Generalising the no caching option to handle all remote images

* rustfmt

* Return None if the URL is not an image

* Updating defaults.hjson

* fixing typo

* Fixing typo

* Skip cloning the Url unless we need to

* using a HEAD request for checking the content type, saving bandwidth/improving perf

* Removing early returns

* Switching back to GET requests for Content-Type because pictrs does not handle HEAD requests

* Simplifying logic and using metadata_image instead of url if we do not get a pictrs thumbnail

* Removing unused import

* Return None as a thumbnail if caching is disabled

* formatting

---------

Co-authored-by: Djones4822 <david.jones4822@gmail.com>
2023-08-31 10:36:39 -04:00
Andrew Fields
1e99e8b9dc
Add Prometheus endpoint (#3456)
Add a server for serving Prometheus metrics. Include a configuration
block in the config file. Provide HTTP metrics on the API, along with
process-level metrics and DB pool metrics.
2023-07-05 13:25:19 +02:00
cetra3
d7da911a48
Remove actix_rt & use standard tokio spawn (#3158)
* Remove `actix_rt` & use standard tokio spawn

* Adjust rust log back down

* Format correctly

* Update cargo lock

* Add DB settings

* Change name and update to latest rev

* Clean up formatting changes

* Move `worker_count` and `worker_retry_count` to settings

* Update defaults

* Use `0.4.4` instead of git branch
2023-06-26 10:24:11 +02:00
Charles Hall
f50325e78a
update doku (#3003)
Fixes malformed documentation generated by previous version.
2023-06-12 05:55:47 -04:00
Charles Hall
14c18dbdae
allow specifying db uri in config file (#2956)
* allow specifying db uri in config file

* succumb to a bug in doku

See <https://github.com/anixe/doku/issues/33>.
2023-06-09 08:18:22 -04:00
Nutomic
1eaf2c8a03
Add feature to embed pictrs in lemmy binary (fixes #2627) (#2633)
* Add feature to embed pictrs in lemmy binary (fixes #2627)

* Add pictrs config

* add protobuf

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-01-20 12:46:49 -05:00
Nutomic
24756af84b
Live reload settings (fixes #2508) (#2543)
* Live reload rate limit settings (fixes #2508)

* fix tests
2022-11-16 14:06:22 -05:00
Nutomic
83d2959b4b
Update doku dependency for easier formatting of defaults.hjson (#2553) 2022-11-11 16:01:29 -05:00
Nutomic
ceae0f5993
Use urlencoding for db url params (fixes #2532) (#2537) 2022-11-07 21:29:32 -05:00
Dessalines
235cc8b228
Moving settings to Database. (#2492)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Fixing federation tests.

* Trying to fix check features 1.

* Addressing PR comments.

* Adding check_apub to all verify functions.
2022-10-27 09:24:07 +00:00
Dessalines
265dc54627
Increase default search rate limit. (#2424) 2022-09-03 11:37:27 +02:00
Nutomic
16fe149a6d
Change config pictrs key name (#2360) 2022-07-14 14:25:10 -04:00
Nutomic
d0a3d99636
Config changes, remove unused image purge function (#2343)
* Throw error if old pictrs config item is used, make api key optional

* use doku url feature

* fix doku version

* fix clippy

* remove unused image purge function, remove config migration
2022-07-11 16:38:37 -04:00
Nutomic
cd4d235171
Move setting http_fetch_retry_limit into federation block (#2314) 2022-06-14 01:20:30 -04:00
Dessalines
4e12e25c59
Adding admin purging of DB items and pictures. #904 #1331 (#1809)
* First pass at adding admin purge. #904 #1331

* Breaking out purge into 4 tables for the 4 purgeable types.

* Using CommunitySafe instead in view

* Fix db_schema features flags.

* Attempting to pass API key.

* Adding pictrs image purging

- Added pictrs_config block, for API_KEY
- Clear out image columns after purging

* Remove the remove_images field from a few of the purge API calls.

* Fix some suggestions by @nutomic.

* Add separate pictrs reqwest client.

* Update defaults.hjson

Co-authored-by: Nutomic <me@nutomic.com>
2022-06-13 19:15:04 +00:00
Nutomic
fcaf7a084c
Upgrade activitypub_federation to 0.2.0, add setting federation.debug (#2300) 2022-06-08 11:45:39 -04:00
Nutomic
7bc2f9fd6d
Remove unused setup config vars (#2302) 2022-06-08 11:44:53 -04:00
Nutomic
00b0b26dfa
Add default post listing type (fixes #2195) (#2209)
* Add default post listing type (fixes #2195)

* review fixes

* change column type
2022-04-19 19:05:08 +00:00
Dessalines
0a36b16e29
Fix allowlist / blocklist description location. Fixes #2214 (#2215)
* Fix allowlist / blocklist description location. Fixes #2214

* Fix allowlist check
2022-04-13 18:27:56 +00:00
Dessalines
e16821116e
Lowering search rate limit. Fixes #2153 (#2154)
* Lowering search rate limit. Fixes #2153

* Adding a search rate limit.

* Forgot to add the websocket search rate limit

* Fix wrong op
2022-03-29 15:46:03 +00:00
Nutomic
3c5275638f
Increase default worker count (#2120) 2022-03-07 22:28:50 +00:00
Nutomic
ef1e164cc5
Make activity queue worker count configurable, log stats (#2113) 2022-03-03 18:54:33 +00:00
Nutomic
9f5183fe98
Add site option for default theme (#2104) 2022-02-23 11:40:36 -05:00
Nutomic
f1aef63149
Use doku(skip) for opentelemetry_url config value (ref #2085) (#2091) 2022-02-07 10:39:37 -05:00
Aode (lion)
6058b11f6e Add comment about opentelemetry_url 2022-02-03 17:24:19 -06:00
Dessalines
1372827b41
Adding starttls support. Fixes #1997 (#2051)
* Adding starttls support. Fixes #1997

* Change name to tls_type, make a string
2022-01-26 16:42:43 +00:00
Riley
3032a2745b
Opentelemetry (#1992)
* Add otlp export

* Add jaeger to dev docker

* More instrument

* Update deps

* Merge from main.

* Add comment about viewing traces

* Fixing reqwest-tracing dep.

* Update pre-commit regex

* Move imports to top

Co-authored-by: Dessalines <tyhou13@gmx.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2022-01-06 19:10:20 +00:00
Dessalines
c883a49a40
First pass at invite-only migration. (#1949)
* First pass at invite-only migration.

* Implement email verification (fixes #219)

* remove unwrap

* Adding views and functionality to registration application. #209

* Add private instance site column, and back end checks.

* Adding some message fields to LoginResponse

* Adding private instance to site setup.

* A few additions:

- Add a DeleteAccount response.
- RegistrationApplicationView now has the safe LocalUserSettings.
- Adding VerifyEmail to websocket API, added a proper response type.

* Adding and reorganizing some email helpers.

* A few fixes for private sites:

- Added a check_registration_application function.
- Only send a verification email if its been changed.
- VerifyEmail now returns LoginResponse.
- Deleting the old tokens after a successful email verify.
- If port is missing on email config, display a better error message.

* Version 0.15.0-rc.3

* Adding published to email_verification table.

* Adding fixes from comments.

* Version 0.15.0-rc.4

* Adding modlog private site check.

* Version 0.15.0-rc.6

Co-authored-by: Felix Ableitner <me@nutomic.com>
2021-12-15 14:49:59 -05:00
Dessalines
161d41999d
Adding min setup password length to the docs. Fixes #1989 (#1990) 2021-12-12 16:42:24 +00:00
Dessalines
040770d7ba
Fix retry infinite loops. Fixes #1964 (#1967)
* Fix retry infinite loops. Fixes #1964

* Moving retry_limit to settings
2021-12-05 10:03:13 -05:00
layla
7fe7062c47 Implement rate limits on comments 2021-11-18 13:40:16 +00:00
Dessalines
b18ea3e0cc
Renaming to slur_filter. Fixes #1773 (#1801)
* Renaming to slur_filter. Fixes #1773

* Fixing config

* Updating doku for slur filter.

* Updating doku for slur filter again.

* Remove comment.
2021-10-28 20:47:25 +00:00
Nutomic
7bfe59a89f
Format config/defaults.hjson before committing (#1860) 2021-10-22 13:29:47 -04:00
Nutomic
dd0ba10b44
Pleroma federation2 (#1855)
* Allow fetching person from Pleroma, including test case (ref #1461)

* Added test case for parsing community from apub json

- fixed a bug with objectid (de)serialization
- fixed a bug with outbox fetching (ref #1582)

* Added apub test for post

* Ignore errors when reading community outbox (fixes #1582)

* Dont fetch community outbox/moderators during tests

* added test for lemmy comment

* Added federation test for pleroma comment

* Added html2md crate to parse comment html from pleroma (fixes #1461)

* some fixes for update_apub_test_files.sh

* Add tests for ToApub, private message, remove update script

* Delete objects from db at the end of each test
2021-10-21 13:25:35 -04:00
Nutomic
21304e6751
Create a custom pre-commit hook, generates config/defaults.hjson (#1857)
* Create a custom pre-commit hook, generates config/defaults.hjson (fixes #1836)

* invalid defaults file

* Revert "invalid defaults file"

This reverts commit 76d2c303e2.
2021-10-21 12:57:09 -04:00
Nutomic
454d3987a5
Generate config docs from code (#1786)
* Generate config docs from code

* update library

* move settings comments from config.hjson to code

* updte doku to 0.10

* update doku
2021-10-08 11:07:24 -04:00
Felix Ableitner
cf214ff583 Move jwt secret from config to database (fixes #1728) 2021-09-27 11:25:09 +02:00
Dessalines
6af75492a9
Swap out iframely (#1706)
* Replace Iframely. Fixes #1681

* Add post_link_tags to nginx

* Adding post_link_tags route

* Cleaning up post_link_tags

* Changing PostLink to SiteMetadata, adding it to the API.

* Fixing issue when local has no openssl certs.

* Fixing an issue with pictrs errors

* Revert "Fixing issue when local has no openssl certs."

This reverts commit dbf7d1b1ee03846e5ef7b7156e618424f1150e1d.

* Add ca-certs to dockerfile for volume mount.

* Cleaning up fetch_pictrs request

* Changing to fetch_site_data
2021-08-19 14:12:49 +00:00