Commit graph

196 commits

Author SHA1 Message Date
Dessalines
a3bf2f1cf1
Auto resolve reports on removing a comment or post. Fixes #4390 (#4402)
* Automatically resolve report when post/comment is removed (#3850)

* Automatically resolve report when post/comment is removed

* also handle apub removes

* Removing auto-resolve report triggers.

* Dont allow creating reports for deleted / removed items.

* Running pgformat.

* Fixing test.

* Addressing PR comments.

* Forgot comment report.

---------

Co-authored-by: Nutomic <me@nutomic.com>
2024-02-15 08:52:04 -05:00
Nutomic
0f414a95d5
Local only community (#4350)
* Add support for local only community (fixes #1576)

* add filters and tests to db views

* dont federate local only community

* test get apub community http

* tests

* more checks

* wip

* api test

* fix tests

* change community.local_only column to visibility enum
(for private communities)

* sql fmt

* rename vars

* clippy

* fix tests

* update lib

* review

* fix js client version

* update client
2024-01-25 11:04:25 -05:00
Nutomic
e8a52d3a5c
Rewrite images to use local proxy (#4035)
* Add markdown rule to add rel=nofollow for all links

* Add markdown image rule to add local image proxy (fixes #1036)

* comments

* rewrite markdown image links working

* add comment

* perform markdown image processing in api/apub receivers

* clippy

* add db table to validate proxied links

* rewrite link fields for avatar, banner etc

* sql fmt

* proxy links received over federation

* add config option

* undo post.url rewriting, move http route definition

* add tests

* proxy images through pictrs

* testing

* cleanup request.rs file

* more cleanup (fixes #2611)

* include url content type when sending post over apub (fixes #2611)

* store post url content type in db

* should be media_type

* get rid of cache_remote_thumbnails setting, instead automatically
take thumbnail from federation data if available.

* fix tests

* add setting disable_external_link_previews

* federate post url as image depending on mime type

* change setting again

* machete

* invert

* support custom emoji

* clippy

* update defaults

* add image proxy test, fix test

* fix test

* clippy

* revert accidental changes

* address review

* clippy

* Markdown link rule-dess (#4356)

* Extracting opengraph_data to its own type.

* A few additions for markdown-link-rule.

---------

Co-authored-by: Nutomic <me@nutomic.com>

* fix setting

* use enum for image proxy setting

* fix test configs

* add config backwards compat

* clippy

* machete

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-25 09:22:11 -05:00
dullbananas
d8f9e8a64c
Post view: move cursor pagination to separate library, add backward pagination to PostQuery (#4320)
* stuff

* stuff

* crates.io

* Update up.sql

* Rerun federation tests

* Update post_view.rs

* Update post_view.rs

* Update up.sql

* Update utils.rs

* Fix precision loss

* Update up.sql

* Update down.sql

* remove unwrap

* Update post_view.rs

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-24 10:50:11 -05:00
dullbananas
759f6d8a9a
Better query plan viewing experience (#4285)
* stuff

* stuff including batch_upsert function

* stuff

* do things

* stuff

* different timestamps

* stuff

* Revert changes to comment.rs

* Update comment.rs

* Update comment.rs

* Update post_view.rs

* Update utils.rs

* Update up.sql

* Update up.sql

* Update down.sql

* Update up.sql

* Update main.rs

* use anyhow macro

* replace get(0) with first()

* as_slice

* Update series.rs

* Update db_perf.sh

* Update and rename crates/db_schema/src/utils/series.rs to crates/db_perf/src/series.rs

* Update utils.rs

* Update main.rs

* Update main.rs

* Update .woodpecker.yml

* fmt main.rs

* Update .woodpecker.yml

* Instance::delete at end

* Update main.rs

* Update Cargo.toml

---------

Co-authored-by: Nutomic <me@nutomic.com>
2024-01-24 10:22:33 -05:00
İsmail Karslı
8670403a67
Add local_subscribers field to CommunityAggregates. Fixes #4144 (#4166)
* Add upload timeout to PictrsConfig

* Bad space 🤔

* Update PictrsConfig upload timeout to include units.

* Add local_subscribers field to CommunityAggregates
struct and schema

* sql format

* local_subscribers test

* fix local_subscribers test

* Revert "fix local_subscribers test"

This reverts commit 4bbac5ce4a.

* Revert "local_subscribers test"

This reverts commit 735107e1f7.

* Create trigger for local_subscribers

* Rename variable

* re-trigger ci

* re-trigger ci

* Add local_subscribers count to follow.spec.ts

* Rename local_subscribers to subscribers_local

* Add subscribers_local to community_aggregates

* added subscribers_local to the aggregate tests

* Check if person exists on community_follower trigger

* Delete community follows before deleting person

* Update lemmy-js-client in api_tests

* Refactor local_subscriber migration

* fix format

* Move migration files date to now

* Fix test to wait for aggregates to federate

* re-trigger ci

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2024-01-24 10:22:05 -05:00
Nutomic
53147596b4
Drop unique constraint for site.name (fixes #4329) (#4342)
* Drop unique constraint for site.name (fixes #4329)

* working down migration
2024-01-03 12:47:39 -05:00
Benjamin Barbeau
60849355db
Add voters to active users (#4235)
* Add voters to active users

* Edit formatting

* Edit formatting

* Edit formatting in down.sql

* Fix person table inner joins

* Remove post read from calculations
2023-12-11 09:38:17 -05:00
Dessalines
6e4278b1c9
Setting rank column defaults to a small number. Fixes #4178 (#4191)
* Setting rank column defaults to a small number. Fixes #4178

* Fixing tests.
2023-11-23 20:39:46 -05:00
Nutomic
a00313e680
Merge /site_inbox into /inbox, remove unique constraint for inboxes (#4138)
* Merge /site_inbox into /inbox (fixes #4137)

Get rid of different inboxes, only use /inbox

Remove shared_inbox_url db columns

add code migration

move to db migration, fixes

machete

fix sql

drop inbox url unique constraints

Dont create auth cookie in backend (#4136)

dont change individual inboxes to shared inbox

Dont send comment reply to user who has community blocked. Fixes #3684 (#4096)

* Dont send comment reply to user who has community blocked. Fixes #3684

* Adding source instance block check.

* Adding api test.

* Addressing PR comments.

* move site inbox rewrite to db

* fix test

* clippy

* clippy 2

* fix test
2023-11-16 14:22:40 +01:00
Dessalines
a53892d2bb
Fixing an issue with timezone migration. (#4153) 2023-11-16 14:19:31 +01:00
dullbananas
8e2cbc9a0f
Remove id column and use different primary key on some tables (#4093)
* post_saved

* fmt

* remove unique and not null

* put person_id first in primary key and remove index

* use post_saved.find

* change captcha_answer

* remove removal of not null

* comment_aggregates

* comment_like

* comment_saved

* aggregates

* remove "\"

* deduplicate site_aggregates

* person_post_aggregates

* community_moderator

* community_block

* community_person_ban

* custom_emoji_keyword

* federation allow/block list

* federation_queue_state

* instance_block

* local_site_rate_limit, local_user_language, login_token

* person_ban, person_block, person_follower, post_like, post_read, received_activity

* community_follower, community_language, site_language

* fmt

* image_upload

* remove unused newtypes

* remove more indexes

* use .find

* merge

* fix site_aggregates_site function

* fmt

* Primary keys dess (#17)

* Also order reports by oldest first (ref #4123) (#4129)

* Support signed fetch for federation (fixes #868) (#4125)

* Support signed fetch for federation (fixes #868)

* taplo

* add federation queue state to get_federated_instances api (#4104)

* add federation queue state to get_federated_instances api

* feature gate

* move retry sleep function

* move stuff around

* Add UI setting for collapsing bot comments. Fixes #3838 (#4098)

* Add UI setting for collapsing bot comments. Fixes #3838

* Fixing clippy check.

* Only keep sent and received activities for 7 days (fixes #4113, fixes #4110) (#4131)

* Only check auth secure on release mode. (#4127)

* Only check auth secure on release mode.

* Fixing wrong js-client.

* Adding is_debug_mode var.

* Fixing the desktop image on the README. (#4135)

* Delete dupes and add possibly missing unique constraint on person_aggregates.

* Fixing clippy lints.

---------

Co-authored-by: Nutomic <me@nutomic.com>
Co-authored-by: phiresky <phireskyde+git@gmail.com>

* fmt

* Update community_block.rs

* Update instance_block.rs

* Update person_block.rs

* Update person_block.rs

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Nutomic <me@nutomic.com>
Co-authored-by: phiresky <phireskyde+git@gmail.com>
2023-11-13 08:14:07 -05:00
Dessalines
97a4fb9a72
Add UI setting for collapsing bot comments. Fixes #3838 (#4098)
* Add UI setting for collapsing bot comments. Fixes #3838

* Fixing clippy check.
2023-11-06 16:09:12 -05:00
phiresky
b9b65c9c18
add federation queue state to get_federated_instances api (#4104)
* add federation queue state to get_federated_instances api

* feature gate

* move retry sleep function

* move stuff around
2023-11-06 16:07:04 -05:00
Nutomic
8c85f35b19
Support signed fetch for federation (fixes #868) (#4125)
* Support signed fetch for federation (fixes #868)

* taplo
2023-11-06 16:02:01 -05:00
Dessalines
c11e9446c6
Fix hot_rank algorithm pushing downvoted content off the feed. (#4085)
* Fix hot_rank algorithm pushing downvoted content off the feed.

- Max hot_rank algorithm now uses max(2, score) rather than greatest(1, score + 3)
- Fixes #4084

* Fixing SQL format.

* Adding 2 to bias the positive scores.
2023-10-30 10:47:57 +01:00
Nutomic
6d27bfed08
Handle invalid ban expires values (fixes #4045) (#4046)
* Handle invalid ban expires values (fixes #4045)

* Adding a few missing expire time checks. Fixing up time conversions. (#4051)

* Adding a few missing expire time checks. Fixing up time conversions.

* Increase settings export wait time.

* get rid of RemoveCommunity.expires

* fmt

* tests

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-10-17 13:25:35 -04:00
SleeplessOne1917
cf1c32d2ab
Enable animated avatars setting (#4040)
* feat: Add user setting for toggling avatar gif animations

* Add forgotten post listing mode option to settings form

* Fix compilation error

* Fix formatting

* Change name of column to be more general purpose

* Fix sql format.

* Fix comments

* Running format.

---------

Co-authored-by: Dessalines <tyhou13@gmx.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-10-17 10:52:34 -04:00
Nutomic
56322c75f0
Read community follower count from home instance (fixes #1440) (#4013)
* Read community follower count from home instance (fixes #1440)

* fmt

* prettier

* fix tests

* fmt

* rename fn

* fmt

* Run prettier

* increase timeout

* ci

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Dessalines <tyhou13@gmx.com>
2023-10-17 10:34:38 -04:00
Nutomic
6d7b38f4de
Implement user data import/export (#3976)
* Implement endpoints for user data import/export

* add test

* exclude avatar/banner

* increase import url count, add rate limit

* also export/import saved posts

* rate limit

* rename

* saved posts also exist

* rename routes

* fix test

* error handling

* clippy

* limit parallelism

* clippy

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-10-11 16:47:22 +02:00
Hamzah Mansour
6470ea81a7
add keyboard navigation setting (#3569)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Dessalines <tyhou13@gmx.com>
2023-10-10 12:17:27 +02:00
Nutomic
dc327652a5
Add db table for login tokens which allows for invalidation (#3818)
* wip

* stuff

* fmt

* fmt 2

* fmt 3

* fix default feature

* use Authorization header

* store ip and user agent for each login

* add list_logins endpoint

* serde(skip) for token

* fix api tests

* A few suggestions for login_token (#3991)

* A few suggestions.

* Fixing SQL format.

* review

* review

* rename cookie

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-10-09 12:46:12 +02:00
phiresky
2dbaf67b7c
Merge branch 'main' into add_federation_worker_index 2023-09-20 18:06:44 +02:00
Nutomic
22608ae983
Rework the way 2FA is enabled/disabled (fixes #3309) (#3959)
* Rework the way 2FA is enabled/disabled (fixes #3309)

* postgres format

* change algo to sha1 for better compat

* review comments

* review

* clippy

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-09-20 10:49:54 -04:00
Nutomic
50f81cf157
User can block instances (fixes #2397) (#3869)
* User can block instances (fixes #2397)

* update comments

* review comments

* use route

* update

* add api test

* update tests

* fix

* fix test

* ci

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-09-20 11:56:13 +02:00
phiresky
88931227b0 Merge remote-tracking branch 'origin/main' into add_federation_worker_index 2023-09-18 16:45:23 +00:00
phiresky
544d30f0d4
Fix Posts List Performance + cursor-based pagination (#3872)
* add token-based pagination + fast subscribed post view

* add migrations

* fix failing heuristic

* revert

* output pagination token as next_page, fix off-by-one, restructure

* more cleanup

* clean

* format sql

* fix comment

* fix tests

* e

* empty

* move last page thing

* restructure a bit for readability

* rename page_cursor

* update for scaled sort

* fix

* sql format

* fix

* get rid of macros

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-09-18 15:44:48 +02:00
Dessalines
4f1240487e Fixing high CPU usage on federation worker recheck. Fixes #3958 2023-09-12 15:54:49 -04:00
phiresky
375d9a2a3c
Persistent, performant, reliable federation queue (#3605)
* persistent activity queue

* fixes

* fixes

* make federation workers function callable from outside

* log federation instances

* dead instance detection not needed here

* taplo fmt

* split federate bin/lib

* minor fix

* better logging

* log

* create struct to hold cancellable task for readability

* use boxfuture for readability

* reset submodule

* fix

* fix lint

* swap

* remove json column, use separate array columns instead

* some review comments

* make worker a struct for readability

* minor readability

* add local filter to community follower view

* remove separate lemmy_federate entry point

* fix remaining duration

* address review comments mostly

* fix lint

* upgrade actitypub-fed to simpler interface

* fix sql format

* increase delays a bit

* fixes after merge

* remove selectable

* fix instance selectable

* add comment

* start federation based on latest id at the time

* rename federate process args

* dead instances in one query

* filter follow+report activities by local

* remove synchronous federation

remove activity sender queue

* lint

* fix federation tests by waiting for results to change

* fix fed test

* fix comment report

* wait some more

* Apply suggestions from code review

Co-authored-by: SorteKanin <sortekanin@gmail.com>

* fix most remaining tests

* wait until private messages

* fix community tests

* fix community tests

* move arg parse

* use instance_id instead of domain in federation_queue_state table

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: SorteKanin <sortekanin@gmail.com>
2023-09-09 12:25:03 -04:00
Dessalines
9785b20843
Adding a scaled sort, to boost smaller communities. (#3907)
* Adding a scaled sort, to boost smaller communities.

- Previously referred to as *best* .
- Fixes #3622

* Fixing scheduled task update.

* Converting hot_rank integers to floats.

* Altering hot_rank psql function to default to zero after a week.

* Setting scaled_rank to zero, where hot_rank is zero.

* Adding image_upload table.
2023-09-06 13:43:27 -04:00
Anon
fe3ebea95a
Add logging for pictrs uploads (#3927)
* Add logging for pictrs uploads

* cleanup
2023-09-06 09:13:30 -04:00
Nutomic
a0ea8dbc00
Auto resolve reports trigger (#3871)
* Revert "Automatically resolve report when post/comment is removed (#3850)"

This reverts commit f7f6766650.

* Automatically resolve reports using db trigger

* lint

* use mod log tables

* fix migration

* fix ci

* fix clippy
2023-09-05 05:35:10 -04:00
Dessalines
56e26fc3d4
Adding a post_view mode. Fixes #3730 (#3731)
* Adding a post_view mode. Fixes #3730

* Fix test.

* Addressing PR comments.

* Adding a post_view mode. Fixes #3730

* Fix test.

* Addressing PR comments.

* Fixing column order.

* Fix default Ok returns.

* Removing return Err(... where feasible.
2023-08-31 09:01:08 -04:00
biosfood
384e55f0e4
update moderator view (#3820)
* update api tests for new moderator view

* chage moderator view to be a listing type in get posts

Note: Internally, the listing type is called ListingType.ModeratorView,
but it's called "Moderator View" in the api endpoint

* fix formatting

* add support for moderator view to list comments

* add api test for moderator view when listing comments

* fix api test formatting

* retry tests

* don't filter out blocked users and communities when using moderator view

* fix cargo tests failing

* fix formatting

* fix previous merge

* Adding ModeratorView to listing_type_enums

* Fixing fmt.

* Adding a default to ListingType.

* Upgrading to use new lemmy-js-client.

---------

Co-authored-by: Nutomic <me@nutomic.com>
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: Dessalines <tyhou13@gmx.com>
2023-08-31 13:07:45 +02:00
phiresky
514f2222e0
Fix time zone handling (#3496)
* convert naive time to utc time

* compounding fixes

* cargo fmt

* fix the rest

* fix down migration

* fix migrations

* fix after merge

* clippy fix

* ap-fed 0.5.0

---------

Co-authored-by: Nutomic <me@nutomic.com>
2023-08-24 11:27:00 -04:00
Nutomic
6047257bfc
Move admin flag from person to local_user (fixes #3060) (#3403)
* Move admin flag from person to local_user (fixes #3060)

The person table is for federated data, but admin flag can only
apply to local users. Thats why it really belongs in the local_user
table. This will also prevent the federation code from accidentally
overwriting the admin flag

* fmt

* try to fix api tests

* lint

* fix person view

* ci

* ci

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
2023-08-24 05:40:08 -04:00
Dessalines
70fae9d68d
Fixing broken SQL migration formatting. (#3800)
* Fixing SQL format.

* Fixing clippy lints.

* Fixing clippy lint 2.
2023-08-03 11:07:35 +02:00
Nutomic
2d0f77af59
Dont use sha hash for password reset token (fixes #3491) (#3795) 2023-08-02 13:02:53 -04:00
Dessalines
be1389420b
Adding SQL format checking via pg_format / pgFormatter (#3740)
* SQL format checking, 1.

* SQL format checking, 2.

* SQL format checking, 3.

* SQL format checking, 4.

* SQL format checking, 5.

* Running pg_format

* Getting rid of comment.

* Upping pg_format version.

* Using git ls-files for sql format check.

* Fixing sql lints.

* Addressing PR comments.
2023-08-02 12:44:51 -04:00
phiresky
e315092ee3
remove n^2 part of person triggers, improve community aggregate trigger (#3739)
* remove n^2 part of person triggers, improve community aggregate trigger

* comment out comment_score tests since previously they only accidentally succeeded

* empty
2023-07-27 23:36:51 +02:00
phiresky
1253a2a0d5
remove performance-problematic and buggy duplicate site aggregates (#3732) 2023-07-27 23:35:53 +02:00
phiresky
9bfa86d162
add trigram index to search (#3719)
* add trigram index to search

* add community index
2023-07-27 12:11:00 +02:00
Pavlos Smith
c890797b37
Add controversial ranking (#3205)
* Added controversy rank property to posts and comments, and ability to sort by it

* Triggers instead of schedules tasks, integer -> double, TODO: comments don't seem to get updated with floats, divide SortTypes

* Created PersonSortType

* PersonSortType::MostComments case

* Removed unused PartialOrd trait

* Added new person sort type mappings

* SortType -> PersonSortType

* fixes

* cargo fmt

* fixes after merge with main

* Fixed bug in controversy rank trigger, removed TopX sorts from PersonSortType and added CommentScore instead

* Uncovered enum case

* clippy

* reset translation changes

* translations

* translations

* Added additional hot ordering on controversial posts and comments

* featured local and featured community added to controversy rank index, additional order_by removed (?), added post_score and post_count to PersonSortType

* Woodpecker rerun

* cargo fmt

* woodpecker rerun

* fixed controversy_rank order

* fix

* Readded migration as latest, removed second update statement for setting controversy rank
2023-07-26 13:07:05 -04:00
Domenic Horner
f9351b6512
UI Settings - Blur NSFW & Auto Expand (#3377)
* add new flag to api

* add new ui settings for local user

* remove extraneous def

* add props to application reg.

* fix clippy updated these

* re-order db schema entries

* remove dupe

* update lemmy sdk

* update lemmy js client

---------

Co-authored-by: Nutomic <me@nutomic.com>
2023-07-26 07:53:45 -04:00
Dessalines
623d81139e
Make sure comments are sorted by hot_rank, then score. (#3667) 2023-07-21 11:44:47 +02:00
Sander Saarend
b511c2e6cb
Denormalize community_id into post_aggregates for a 1000x speed-up when loading posts (#3653)
* Denormalize community_id into post_aggregates for a 1000x speed-up when loading posts

* Remove unused index

* Add creator_id to post_aggregates

* Use post_aggregates as main table for PostQuery

* Make post_aggregates the main table for PostView

* Reformat SQL
2023-07-20 11:13:21 -04:00
Dessalines
ced3aa5bd8
Fixing hot_ranks and scores to append a published sort. (#3618)
- #3428
2023-07-20 16:44:23 +02:00
Sander Saarend
6688a8a5d4
Optimize hot rank updates (#3617) 2023-07-17 11:05:55 +02:00
Nutomic
e9e76549a8
Split activity table into sent and received parts (fixes #3103) (#3583)
* Split activity table into sent and received parts (fixes #3103)

The received activities are only stored in order to avoid processing
the same incoming activity multiple times. For this purpose it is
completely unnecessary to store the data. So we can split the
table into sent_activity and received_activity parts, where
only sent_activity table needs to store activity data. This should
reduce storage use significantly.

Also reduces activity storage duration to three months, we can reduce
this further if necessary.

Additionally the id columns of activity tables are removed because
they are completely unused and risk overflowing (fixes #3560).

* address review

* move insert_received_activity() methods to verify handlers

* remove unnecessary conflict line

* clippy

* use on conflict, add tests
2023-07-14 11:17:06 -04:00
Simon Bordeyne
9b5e765364
Add infinite scroll user option (#3572) 2023-07-12 09:12:01 -04:00