Commit graph

806 commits

Author SHA1 Message Date
kim
6ac1dda96f
[chore] small changes missed in previous dereferencer.GetAccount() PRs ()
* small formatting changes, rewrite fetchRemoteMedia to use separate funcs + use mutex lock correctly

* move url parsing before acquiring mutex locks

* use wrapped mutexes to allow safe unlocking. (previously i did a fucky and passed mutex by value...)

* remove unused code

* use consistent map keying for dereferencing headers/avatars

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-10 20:15:23 +00:00
tobi
e5e257c259
[bugfix] Fix error on searching for account w/accountDomain by host () 2023-02-09 10:34:44 +01:00
tobi
95715f9251
[performance] Don't fetch avatar + header if uri hasn't changed () 2023-02-09 09:27:07 +01:00
tobi
27e95fd123
[chore/bugfix] Serve + throttle publickey separately from rest of ActivityPub API ()
* serve publickey separately from AP, don't throttle it

* update nginx cache documentation, cache main-key too

* throttle public key, but separately from other endpoints
2023-02-08 15:10:56 +01:00
tobi
4e4da19720
[bugfix] Use SignatureCheck middleware for web profile endpoints too () 2023-02-07 14:57:09 +01:00
Sam Lade
ad6ab037e4
[bugfix] don't trash emoji in profile fields on edit () 2023-02-07 09:58:36 +01:00
kim
ac2bdbbc62
[bugfix] fix file range length calculation being off by 1 ()
* small formatting change

* fix range handling new length calculation

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-07 08:51:15 +00:00
Sam Lade
6a6647d68b
Ignore missing files when cleaning up media () 2023-02-06 14:58:54 +00:00
kim
02767bfc7d
[performance] remove local copying of file for satisfying range headers ()
* remove local copying of file for satisfying range headers

Signed-off-by: kim <grufwub@gmail.com>

* remove unused import

Signed-off-by: kim <grufwub@gmail.com>

* fix range header reader limit calculation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-06 09:50:16 +01:00
tobi
75e1b9c896
[bugfix] fix old password hash staying in cache () 2023-02-06 09:44:13 +01:00
tobi
7f3245738d
[chore] stub /api/v1/featured_tags endpoint () 2023-02-05 10:50:09 +01:00
tobi
80c26d61f7
[bugfix] Allow instance thumbnail description to be set separately from image () 2023-02-04 15:53:11 +01:00
kim
abe9447d28
fix cache startup ()
Signed-off-by: kim <grufwub@gmail.com>
2023-02-03 20:16:11 +00:00
kim
33aee1b1e9
[chore] reformat GetAccount() functionality, support updating accounts based on last_fetch ()
* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove now unused error type

* return nil when wrapping nil error

* update expected error messages

* return correct url for foss satan webfinger

* add AP model for Some_User

* normalize local domain

* return notretrievable where appropriate

* expose NewErrNotRetrievable

* ensure webfinger for new accounts searched by uri

* update local account short circuit

* allow enrich to fail for already-known accounts

* remove unused LastWebfingeredAt

* expose test maps on mock http client

* update Update test

* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)

* remove mutex unlock on transport init fail (it hasn't yet been locked!)

* woops add back the error wrapping to use ErrNotRetrievable

* caches were never being started... 🙈

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-02-03 20:03:05 +00:00
Autumn!
49beb17a8f
[chore] Text formatting overhaul ()
* Implement goldmark debug print for hashtags and mentions

* Minify HTML in FromPlain

* Convert plaintext status parser to goldmark

* Move mention/tag/emoji finding logic into formatter

* Combine mention and hashtag boundary characters

* Normalize unicode when rendering hashtags
2023-02-03 11:58:58 +01:00
tobi
271da016b9
[bugfix] Read Bookwyrm Articles more thoroughly () 2023-02-02 16:41:02 +01:00
tobi
382512a5a6
[feature] Implement /api/v2/instance endpoint ()
* interim: start adding /api/v2/instance

* finish up
2023-02-02 14:08:13 +01:00
tobi
4ee4cd2da1
[chore/performance] use only 1 sqlite db connection regardless of multiplier () 2023-02-01 11:55:34 +01:00
tobi
b80be48fed
[chore] Use 'immediate' lock for sqlite transactions ()
* [chore] Use 'immediate' lock for sqlite transactions

* allow 1 connection regardless of cpu amount
2023-01-31 13:46:45 +01:00
tobi
3283900b0d
[feature] Federate reports to remote instance as Flag (if desired) ()
* reports federate out, we did it lxds

* fix optional line start (should be optional slash)
2023-01-27 14:48:11 +01:00
tobi
782169da76
[chore] set max open / idle conns + conn max lifetime for both postgres and sqlite ()
* [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite

* reduce cache size default to 8MiB, reduce connections to 2 * cpu

* introduce max open conns multiplier, tune sqlite and pg separately

* go fmt
2023-01-26 15:12:48 +01:00
f0x52
17eecfb6d9
[feature] Public list of suspended domains ()
* basic rendered domain blocklist (unauthenticated!)

* style basic domain block list

* better formatting for domain blocklist

* add opt-in config option for showing suspended domains

* format/linter

* re-use InstancePeersGet for web-accessible domain blocklist

* reword explanation, border styling

* always attach blocklist handler, update error message

* domain blocklist error message grammar
2023-01-25 18:06:41 +01:00
tobi
993aae5e48
[feature] Accept incoming federated Flag activity ()
* start working on handling incoming Flag activity

* interim commit

* federate Flag in successfully
2023-01-25 11:12:27 +01:00
tobi
faeb7ded3b
[feature] Implement reports admin API so admins can view + close reports ()
* add admin report api endpoints + tests

* [chore] remove funky duplicate attachment in testrig
2023-01-25 11:12:17 +01:00
tobi
36f62d6e60
[chore] remove funky duplicate attachment in testrig ()
* [chore] remove funky duplicate attachment in testrig

* whoops
2023-01-23 13:39:01 +01:00
tobi
e9747247d5
[feature] Implement /api/v1/reports endpoints on client API ()
* start adding report client api

* route + test reports get

* start report create endpoint

* you can create reports now babyy

* stub account report processor

* add single reportGet endpoint

* fix test

* add more filtering params to /api/v1/reports GET

* update swagger

* use marshalIndent in tests

* add + test missing Link info
2023-01-23 13:14:21 +01:00
즈눅
13ec15d2bb
[chore] extending maximumPasswordLength to 256 ()
* [chore] extending maximumPasswordLength to 256

Resolves: 

* Make long test longer
2023-01-22 23:25:11 +01:00
tobi
974ec80a20
[chore] Change default sqlite busy timeout to 5m () 2023-01-18 14:37:52 +01:00
tobi
627b8eeae6
[feature] Tune sqlite pragmas ()
* sqlite pragma tuning

* use formatuint

* add sqlite busy timeout

* fix incorrect cache size format

* update envparsing test

* add sqlite tuning flags to cli

* set sqlite timeout to 30s default
2023-01-17 12:29:44 +00:00
tobi
d4cddf460a
[bugfix] Parse video metadata more accurately; allow Range in fileserver ()
* don't serve unused fields for video attachments

* parse video bitrate + duration more accurately

* use ServeContent where appropriate to respect Range

* abstract temp file seeker into its own function
2023-01-16 16:19:17 +01:00
tobi
132c73883f
[bugfix] Mount bookmarks endpoint correctly () 2023-01-13 17:07:52 +01:00
tobi
eafd73c292
[chore] Remove omitempty on account source; refactor tests to use prettyprint json ()
* remove omitEmpty tag on account source items

* update tests
2023-01-13 17:02:45 +01:00
kim
5318054808
[performance] media processing improvements ()
* media processor consolidation and reformatting, reduce amount of required syscalls

Signed-off-by: kim <grufwub@gmail.com>

* update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling

Signed-off-by: kim <grufwub@gmail.com>

* fix duration not being set, fix mp4 test expecting error

Signed-off-by: kim <grufwub@gmail.com>

* fix test expecting media files with different extension

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code

Signed-off-by: kim <grufwub@gmail.com>

* fix expected storage paths in tests, update expected test thumbnails

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* fix cached presigned s3 url fetching

Signed-off-by: kim <grufwub@gmail.com>

* fix tests

Signed-off-by: kim <grufwub@gmail.com>

* fix test models

Signed-off-by: kim <grufwub@gmail.com>

* update media processing to use sync.Once{} for concurrency protection

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* fix passing in KVStore GetStream() as stream to PutStream()

Signed-off-by: kim <grufwub@gmail.com>

* fix unlocks of storage keys

Signed-off-by: kim <grufwub@gmail.com>

* whoops, return the error...

Signed-off-by: kim <grufwub@gmail.com>

* pour one out for tobi's code <3

Signed-off-by: kim <grufwub@gmail.com>

* add back the byte slurping code

Signed-off-by: kim <grufwub@gmail.com>

* check for both ErrUnexpectedEOF and EOF

Signed-off-by: kim <grufwub@gmail.com>

* add back links to file format header information

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2023-01-11 12:13:13 +01:00
Sleep
3512325e46
[feature] Add local user and post count to nodeinfo responses ()
* Add local user and post count to nodeinfo responses

This fixes  (at least partially). The nodeinfo endpoint should now
return the total users on an instance, along with their post count.

* Update NodeInfoUsers docstring and swagger yaml file
2023-01-11 11:36:36 +01:00
tobi
d6487933c7
[feature] Implement Report database model and utility functions ()
* implement report database model

* implement report cache + config changes

* implement report database functions

* report uri / regex functions

* update envparsing test

* remove unnecessary uri index

* remove unused function + cache lookup

* process error when storing report
2023-01-10 14:19:05 +00:00
kim
1bda6a2002
[bugfix] return early in websocket upgrade handler ()
* launch websocket streaming in goroutine to allow upgrade handler to return

* don't send any message on ping, improved close check on failed read

* use context to signal wsconn close, ensure canceled in read goroutine

Signed-off-by: kim <grufwub@gmail.com>
2023-01-08 11:43:08 +00:00
kim
2bf9bfa24f
[bugfix] fix panic during status delete loop by breaking out early on len(statuses) == 0 ()
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2023-01-08 11:33:03 +01:00
tobi
0dbe6c514f
[chore] Update/add license headers for 2023 () 2023-01-05 12:43:00 +01:00
tobi
ff46dd485a
[chore] Fix emoji notnull constraint on initial gtsmodel () 2023-01-05 12:37:50 +01:00
tobi
90a14abb0c
[feature] HTTP request throttling middleware ()
* [feature] Add throttling middleware to AP endpoints

* refactor a lil bit

* use config setting, start updating docs

* doc updates

* use relative links in faq doc

* small docs fixes

* return code 503 instead of 429 when throttled

* throttle other endpoints too

* simplify token channel prefills
2023-01-04 11:57:59 +01:00
kim
71dfea7e47
[chore] shuffle middleware to split rate limitting into client/s2s/fileserver, share gzip middleware globally ()
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2023-01-03 11:50:59 +01:00
tobi
941893a774
[chore] The Big Middleware and API Refactor (tm) ()
* interim commit: start refactoring middlewares into package under router

* another interim commit, this is becoming a big job

* another fucking massive interim commit

* refactor bookmarks to new style

* ambassador, wiz zeze commits you are spoiling uz

* she compiles, we're getting there

* we're just normal men; we're just innocent men

* apiutil

* whoopsie

* i'm glad noone reads commit msgs haha :blob_sweat:

* use that weirdo go-bytesize library for maxMultipartMemory

* fix media module paths
2023-01-02 12:10:50 +00:00
tobi
1659f75ae6
[feature] For video attachments, store + return fps, bitrate, duration ()
* start messing about with different mp4 metadata extraction

* heyyooo it works

* add test cow

* move useful multierror to gtserror package

* error out if video doesn't seem to be a real mp4

* test parsing mkv in disguise as mp4

* tidy up error handling

* remove extraneous line

* update framerate formatting

* use float32 for aspect

* fixy mctesterson
2022-12-22 11:48:28 +01:00
tobi
eabb906268
[bugfix] fix media create error not being checked () 2022-12-21 17:52:07 +01:00
tobi
6ebdc306ed
[bugfix] Close reader gracefully when streaming recache of remote media to fileserver api caller ()
* close pipereader on failed data function

* gently slurp the bytes

* readability updates

* go fmt

* tidy up file server tests + add more cases

* start moving io wrappers to separate iotools package. Remove use of buffering while piping recache stream

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-12-21 11:17:43 +01:00
tobi
2bbc64be43
[feature] Enable basic video support (mp4 only) ()
* [feature] basic video support

* fix missing semicolon

* replace text shadow with stacked icons

Co-authored-by: f0x <f0x@cthu.lu>
2022-12-17 04:38:56 +00:00
tobi
0f38e7c9b0
[chore] fix some little config whoopsies () 2022-12-16 22:38:41 +00:00
kim
da751c02fb
update go-cache to v3.2.0 with support for ignoring errors () 2022-12-16 22:36:52 +00:00
Autumn!
eb08529f35
[chore/bugfix] Switch markdown from blackfriday to goldmark ()
Co-authored-by: Autumn! <autumnull@posteo.net>
2022-12-16 12:20:22 +01:00
tobi
1d24c1c283
[bugfix] Use null for empty api status language ()
* [bugfix] Use null for empty api status `language`

* update swagger docs
2022-12-15 09:41:49 +01:00