Commit graph

2011 commits

Author SHA1 Message Date
kim
6fe96a5611
update go-fastcopy to v1.1.3 (#3200) 2024-08-14 16:56:56 +00:00
kim
f8d5b9f49c
update go-structr to v0.8.8 (#3199) 2024-08-14 12:08:24 +00:00
kim
9cd27b412d
[security] harden account update logic (#3198)
* on account update, ensure that public key has not changed

* change expected error message

* also support the case of changing account keys when expired (not waiting for handshake)

* tweak account update hardening logic, add tests for updating account with pubkey expired

* add check for whether incoming data was via federator, accepting keys if so

* use freshest window for federated account updates + comment about it
2024-08-13 15:37:09 +00:00
kim
5212a1057e
[bugfix] relax missing preferred_username, instead using webfingered username (#3189)
* support no preferred_username, instead using webfingered username

* add tests for the new preferred_username behaviour
2024-08-13 09:01:50 +00:00
kim
4cb3e4d3e6
[bugfix] incorrect AP serialize function used serializing worker data (#3196) 2024-08-12 18:23:24 +02:00
dependabot[bot]
871d062bd6
[chore]: Bump golang.org/x/image from 0.18.0 to 0.19.0 (#3191) 2024-08-12 10:12:52 +00:00
dependabot[bot]
745b80259f
[chore]: Bump golang.org/x/net from 0.27.0 to 0.28.0 (#3194)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.27.0 to 0.28.0.
- [Commits](https://github.com/golang/net/compare/v0.27.0...v0.28.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-12 11:49:45 +02:00
kim
865b3aeaac
[bugfix] updated pinned counts on status delete (#3188)
* include pinned status when incrementing / decrementing status counts

* remove the pinned increment on status creation

* code comments

* microoptimize decr
2024-08-11 11:23:36 +02:00
kim
4a3ece0c6c
[bugfix] ensure testrig package only compiled-in when debug enabled (#3185)
* ensure testrig package only compiled-in (including init) when debug enabled

* add code comment to testrig init to indicate WebAssembly compilation
2024-08-09 10:53:29 +00:00
kim
f77005128a
[performance] move thumbnail generation to go code where possible (#3183)
* wrap thumbnailing code to handle generation natively where possible

* more code comments!

* add even more code comments!

* add code comments about blurhash generation

* maintain image rotation if contained in exif data

* move rotation before resizing

* ensure pix_fmt actually selected by ffprobe, check for alpha layer with gifs

* use linear instead of nearest-neighbour for resizing

* work with image "orientation" instead of "rotation". use default 75% quality for both webp and jpeg generation

* add header to new file

* use thumb extension when getting thumb mime type

* update test models and tests with new media processing

* add suggested code comments

* add note about thumbnail filter count reducing memory usage
2024-08-08 17:12:13 +00:00
John Winston
94c615d417
[feature] Add db-postgres-connection-string option (#3178)
* handle db-url

* lint and add doc

* add more doc

* fix config test

* return error

* change name from db-url to db-postgres-connection-string
2024-08-08 14:00:19 +02:00
tobi
b19cfee7ae
[feature] Use gifv type for short soundless mp4 videos (#3182) 2024-08-08 08:12:16 +00:00
kim
3045782b49
updates our ffmpreg version, heh (#3181) 2024-08-07 15:38:02 +02:00
Leah Neukirchen
4697271cef
[bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket (#3169)
* [bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket

Chrome expects the selected Sec-Websocket-Protocol to be sent back
on the WebSocket upgrade request (RFC6455 1.9).

* fiddle a bit to avoid getting headers multiple times

* add some explanatory notes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-06 11:52:27 +02:00
dependabot[bot]
b78be9fd4a
[chore]: Bump golang.org/x/oauth2 from 0.21.0 to 0.22.0 (#3171)
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.21.0 to 0.22.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.21.0...v0.22.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-05 09:52:20 +00:00
kim
b85a9983d0
[bugfix] fix emoji recaching operations (#3167)
* add test for emoji update image

* update emoji recache to set the instance account id

* don't refresh emoji if only not cached. in that case literally just recache

* code comment

* rename + move a few things

* add some more code comments, and rename some functions to make logic a bit clearer

* remove unnecessary nil check (the value can be nil)

* comment wording

* remove test data output

* handle the case of caching an emoji which has been refreshed then uncached

* allow overwriting on testrig storage as we do now on regular storage

* fix emoji category ID not getting updated

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-03 19:05:38 +02:00
tobi
fa59c3713c
[chore] Add media-ffmpeg-pool-size config var (#3164) 2024-08-03 16:40:26 +02:00
tobi
09f239d7e3
[chore/frontend] Update namerole rendering on skinny devices (#3166) 2024-08-03 16:39:07 +02:00
Daenney
9b50151f17
[feature] Beef up our AI opt-outs (#3165)
* [chore] Synchronise our robots.txt with upstream

* [feature] Add headers to escape AI crawlers

This adds 2 headers that a number of AI crawlers respect to signal that
content should not be included in their datasets.
2024-08-02 18:22:39 +02:00
kim
e5e996b28a
[bugfix] close files before error return (#3163)
* close files before error return

* use defer statements

* shuffle around some defers
2024-08-02 15:11:24 +01:00
kim
0f734a2410
[chore] move PopulateAccountStats() nil check often performed into function itself (#3158)
* move PopulateAccountStats() nil check often performed into function itself

* fix test to take in mind we don't repopulate account stats if not-nil
2024-08-02 13:15:11 +01:00
kim
94e87610c4
[chore] add back exif-terminator and use only for jpeg,png,webp (#3161)
* add back exif-terminator and use only for jpeg,png,webp

* fix arguments passed to terminateExif()

* pull in latest exif-terminator

* fix test

* update processed img

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-02 12:46:41 +01:00
tobi
7b5917d6ae
[feature] Allow import of following and blocks via CSV (#3150)
* [feature] Import follows + blocks via settings panel

* test import follows
2024-08-02 12:41:46 +01:00
tobi
697261da53
[chore] Take account of rotation data when calculating full size image dimensions (#3159)
* [chore] Take account of rotation data when calculating full size image dimensions

* boobies
2024-07-31 18:43:39 +00:00
Vyr Cossont
fd837776e2
[feature] Implement Mastodon-compatible roles (#3136)
* Implement Mastodon-compatible roles

- `Account.role` should only be available through verify_credentials for checking current user's permissions
- `Account.role` now carries a Mastodon-compatible permissions bitmap and a marker for whether it should be shown to the public
- `Account.roles` added for *public* display roles (undocumented but stable since Mastodon 4.1)
- Web template now uses only public display roles (no user-visible change here, we already special-cased the `user` role)

* Handle verify_credentials case for default role

* Update JSON exact-match tests

* Address review comments

* Add blocks bit to admin permissions bitmap
2024-07-31 09:26:09 -07:00
tobi
2f7d654380
[docs] Update system requirements, move things around a bit (#3157) 2024-07-31 15:12:47 +01:00
tobi
38f041cea1
[feature] Allow users to export data via the settings panel (#3140)
* [feature] Allow users to export data via the settings panel

* rename/move some stuff
2024-07-31 15:03:34 +01:00
CDN
43519324b3
[feature] Object store custom URL (S3) (#3046)
* tweaks

* boobs

* fix variable name + typo

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-31 13:44:18 +01:00
rdelaage
26022c2733
Fix no rows in result set error in emoji list command (#3152)
Co-authored-by: Romain de Laage <romain.delaage@rdelaage.ovh>
2024-07-30 18:12:06 +01:00
kim
0b4a951b48
[performance] limit ffprobe read time to at most 1s after start of file (#3155) 2024-07-30 16:18:29 +02:00
kim
87cff71af9
[feature] persist worker queues to db (#3042)
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup

* ensure the tasks are sorted by creation time before pushing them

* add migration to insert WorkerTask{} into database, add test for worker task persistence

* add test for recovering worker queues from database

* quick tweak

* whoops we ended up with double cleaner job scheduling

* insert each task separately, because bun is throwing some reflection error??

* add specific checking of cancelled worker contexts

* add http request signing to deliveries recovered from database

* add test for outgoing public key ID being correctly set on delivery

* replace select with Queue.PopCtx()

* get rid of loop now we don't use it

* remove field now we don't use it

* ensure that signing func is set

* header values weren't being copied over 🤦

* use ptr for httpclient.Request in delivery

* move worker queue filling to later in server init process

* fix rebase issues

* make logging less shouty

* use slices.Delete() instead of copying / reslicing

* have database return tasks in ascending order instead of sorting them

* add a 1 minute timeout to persisting worker queues
2024-07-30 13:58:31 +02:00
tobi
42932f9820
[chore] replace UniqueStrings with Deduplicate (#3154) 2024-07-30 12:20:06 +02:00
kim
47c26818d6
[chore] add some more slice related utility functions + remove duplicated functions (#3149) 2024-07-30 11:29:32 +02:00
Vyr Cossont
a237e2b295
[feature] Implement following hashtags (#3141)
* Implement followed tags API

* Insert statuses with followed tags into home timelines

* Test following and unfollowing tags

* Correct Swagger path params

* Trim conversation caches

* Migration for followed_tags table

* Followed tag caches and DB implementation

* Lint and tests

* Add missing tag info endpoint, reorganize tag API

* Unwrap boosts when timelining based on tags

* Apply visibility filters to tag followers

* Address review comments
2024-07-29 19:26:31 +01:00
kim
368c97f0f8
[bugfix] take into account rotation when generating thumbnail (#3147)
* take into account rotation when generating thumbnail, simplify ffprobe output to show only fields we need

* only show rotation side data

* remove unnecessary comment

* fix code comments

* remove debug logging
2024-07-28 21:10:41 +02:00
kim
58f8082795
[bugfix] moves file rename to earlier in media pipeline so ffmpeg calls ALWAYS have extension (#3146) 2024-07-28 10:31:49 +02:00
tobi
87ee64afa0
[chore] log delivery worker stop/start at debug level (#3145) 2024-07-27 13:43:18 +02:00
kim
ecc114fc00
[chore] replaces nested ifs with switch cases, removes defer 'onFail()' being passed to funcs (#3143) 2024-07-27 11:09:02 +02:00
kim
d9e70b942f
ensure ffmpeg muxer gets set correctly by setting input file extension before clean (#3142) 2024-07-26 16:30:24 +01:00
tobi
ecfea10e35
[bugfix] Use punycode for host part of resource query param when doing webfinger requests (#3133)
* [bugfix] use punycode when webfingering

* account for punycode when checking if final URI matches expected

* hmm

* fix test
2024-07-26 13:11:07 +02:00
tobi
8ab2b19a94
[feature] Federate interaction policies + Accepts; enforce policies (#3138)
* [feature] Federate interaction policies + Accepts; enforce policies

* use Acceptable type

* fix index

* remove appendIRIStrs

* add GetAccept federatingdb function

* lock on object IRI
2024-07-26 12:04:28 +02:00
tobi
f8d399cf6a
[docs] Add separate migration section + instructions for moving to GtS and not just away (#3139)
* [docs] Add separate migration section + instructions for moving to GtS and not just away

* update headings
2024-07-24 19:07:36 +02:00
tobi
c9b6220fef
[chore] Add interaction filter to complement existing visibility filter (#3111)
* [chore] Add interaction filter to complement existing visibility filter

* pass in ptr to visibility and interaction filters to Processor{} to ensure shared

* use int constants for for match type, cache db calls in filterctx

* function name typo 😇

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-07-24 12:27:42 +01:00
kim
63fc9b6c3e
[chore] renames the GTS caches to DB caches (#3127)
* renames the `GTS` caches to `DB` caches, as it better references what they are

* change remaining Caches.GTS uses to Caches.DB
2024-07-24 10:41:43 +02:00
tobi
325b4a2b4a
[feature] Allow user to set "bot" flag; show bot icon on profile (#3135)
* [feature] Allow user to set "bot" flag; show bot icon on profile

* tweak

* update customs
2024-07-24 10:40:56 +02:00
Vyr Cossont
db0a47126e
[bugfix] Serialize empty conversation account list as empty list, not null (#3137) 2024-07-24 10:40:01 +02:00
tobi
6533531cf1
[chore] Add some log lines to recent migrations warning not to interrupt (#3134)
* [chore] Add some log lines to recent migrations warning not to interrupt

* arse
2024-07-23 23:06:20 +02:00
Vyr Cossont
86a59db711
Remove content and related fields from boosts (#3131)
These duplicate the content of the target and aren't necessary for anything.

- Stops copying some fields from target when boosting or processing a remote boost
- Adds a migration to null out existing duplicate data
- Updates tests
2024-07-23 20:51:57 +01:00
Vyr Cossont
2d921d9d7c
Explicitly propagate filter results from statuses to their boosts in API responses (#3130)
Related to #3128
2024-07-23 20:51:07 +01:00
Vyr Cossont
8fdd358f4b
[feature] Conversations API (#3013)
* Implement conversations API

* Sort and page conversations by last status ID

* Appease linter

* Fix deleting conversations and statuses

* Refactor to make migrations automatic

* Lint

* Update tests post-merge

* Fixes from live-fire testing

* Linter caught a format problem

* Refactor tests, fix cache

* Negative test for non-DMs

* Run conversations advanced migration on testrig startup as well as regular server startup

* Document (lack of) side effects of API method for deleting a conversation

* Make not-found check less nested for readability

* Rename PutConversation to UpsertConversation

* Use util.Ptr instead of IIFE

* Reduce cache used by conversations

* Remove unnecessary TableExpr/ColumnExpr

* Use struct tags for both unique constraints on Conversation

* Make it clear how paging with GetDirectStatusIDsBatch should be used

* Let conversation paging skip conversations it can't render

* Use Bun NewDropTable

* Convert delete raw query to Bun

* Convert update raw query to Bun

* Convert latestConversationStatusesTempTable raw query partially to Bun

* Convert conversationStatusesTempTable raw query partially to Bun

* Rename field used to store result of MaxDirectStatusID

* Move advanced migrations to their own tiny processor

* Catch up util function name with main

* Remove json.… wrappers

* Remove redundant check

* Combine error checks

* Replace map with slice of structs

* Address processor/type converter comments

- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation

* Add error context when dropping temp tables
2024-07-23 20:44:31 +01:00