Commit graph

939 commits

Author SHA1 Message Date
kim
ede8f43635
[performance] temporarily cache account status counts to reduce no. account counts (#2620)
* temporarily cache account status counts to reduce no. account counts

* whoops, forgot to initAccountCounts()

* use already fetched cache capacity value

* make cache a ptr type

* whoops, use count instead of just select

* fix to correctly use the transaction

* properly wrap that tx 😇

* correctly wrap both tx types

* outline retryOnBusy() to allow the fast path to be inlined

* return err on context cancelled

* remove unnecessary storage of context in stmt, fix Exec and Query interface implementations

* shutup linter
2024-02-12 11:52:12 +00:00
tobi
e890169e6f
use pointer for freshness window (#2614) 2024-02-09 14:24:49 +00:00
tobi
7a7746701d
[chore] Move DoOnce func wrapper to util (#2613) 2024-02-09 11:38:51 +00:00
kim
6738fd5bb0
[feature/performance] sqlite pragma optimize on close (#2596)
* wrap database drivers in order to handle error processing, hooks, etc

* remove dead code

* add code comment, remove unused blank imports
2024-02-07 14:43:27 +00:00
tobi
b6fe8e7a5b
[bugfix] Ensure activities sender always = activities actor (#2608) 2024-02-06 12:59:37 +01:00
tobi
aa396c78d3
[feature] serdes for moved/also_known_as (#2600)
* [feature] serdes for moved/also_known_as

* document `alsoKnownAs` and `movedTo` properties

* only implicitly populate AKA uris from DB for local accounts

* don't let remotes store more than 20 AKA uris to avoid shenanigans
2024-02-06 09:45:46 +00:00
kim
c13eb87ae7
stop paged endpoints returning null for empty items (#2597) 2024-02-01 12:45:02 +00:00
tobi
b614d33c40
[feature] Try HTTP signature validation with and without query params for incoming requests (#2591)
* [feature] Verify signatures both with + without query params

* Bump to tagged version
2024-01-31 14:15:28 +00:00
Vyr Cossont
c675d47a8c
Improve context descendant sorting (#2579)
* Improve context descendant sorting

Topologically sort replies, then move self-replies to top of list

* Unify descendant sort passes

* Correct test package name

* Preallocate maps
2024-01-31 13:40:41 +00:00
kim
c9452f32f3
[bugfix] fix possible infinite loops in media / emoji cleanup (#2590)
* update media / emoji cleaner funcs to use new paging package, check for same returned maxID

* fix other calls of getattachments and getmojis not using paging

* use alternative order-by function
2024-01-31 13:31:53 +00:00
kim
0f7a2024c3
[bugfix] parent status replied to status not dereferenced sometimes (#2587)
* much simplified DereferenceStatusAncestors(), also handles edge cases now

* perform status acceptibility check before handling even as forward

* don't further dereference ancestors if they're up to date

* call enrichStatusSafely() directly to ensure we get error messages

* change getStatusByURI() semantics to return error + old model on failed update, fix deref ancestor to check for staleness before refetch

* perform a nil-check on the status.Local variable, in case it hasn't been set on new status attempting refresh

* more consistently set returned parent status, don't check if updated

* only home-timeline statuses if explicitly visible AND not explicitly invisible!

* fix broken test now that status acceptibility checks happen on forwarded statuses
2024-01-31 13:29:47 +00:00
tobi
aa8bbe6ad2
[bugfix] Fix Postgres emoji delete, emoji category change (#2570)
* [bugfix] Fix Postgres emoji delete, emoji category change

* revert trace logging

* caching issue

* update tests
2024-01-29 14:57:22 +00:00
tobi
14b684b2b5
[chore] Add a couple tests for updating list entries (#2580) 2024-01-28 13:01:12 +01:00
Vyr Cossont
7e0a203173
[bugfix] Fix EmptyJSONObject/EmptyJSONArray (#2576)
* Fix EmptyJSONObject/EmptyJSONArray

These are meant to be the bytes representing an empty object and array in JSON: `{}` and `[]`. They are actually the strings `"{}"` and `"[]"`. This causes clients expecting an object or array to not be able to parse the response.

* Use json.RawMessage instead of []byte
2024-01-28 11:49:04 +01:00
tobi
e3052e8c82
[bugfix] Don't return Account or Status if new and dereferencing failed, other small fixes (#2563)
* tidy up account, status, webfingering logic a wee bit

* go fmt

* invert published check

* alter resp initialization

* get Published from account in typeutils

* don't instantiate error for no darn good reason

* shadow err

* don't repeat error codes in wrapped errors

* don't wrap error unnecessarily
2024-01-26 14:17:10 +01:00
tobi
138cbe4d60
[feature] Ratelimit + serve emoji images on separate router group (#2548)
* [feature] Serve + rate limit emoji files separately from attachments

* add a wee little warning about uploading loads of emojis
2024-01-22 15:17:04 +00:00
tobi
d9729e7d28
[bugfix] Don't return Internal Server Error when searching for URIs that don't return AP JSON (#2550)
* [bugfix] Don't return Internal Server Error when searching for URIs that don't return AP JSON

* don't pass map pointer
2024-01-22 14:38:45 +00:00
tobi
9d80f7fd68
[feature] Allow "charset=utf8" in incoming AP POST requests (#2564)
* [feature] Allow "charset=utf8" in incoming AP POST requests

* changed my mind

* document POSTing to a GtS inbox

* correct link
2024-01-22 14:33:01 +00:00
tobi
4e0488acfe
[bugfix] fix array type for also_known_as_uris (#2553) 2024-01-21 16:34:38 +01:00
Daenney
74380ae8a1
[chore] Downgrade OTEL libraries to 1.20 (#2546)
For some reason httpconv seems to have disappeared from 1.21, which
results in a 1.21 runtime with 1.20 semconv/httpconv which seems to
break some things.

For now, this rolls the OTEL dependencies back to 1.20 which should fix
the observability issues. We'll need to take a look at how to upgrade
safely and correctly in the future.

Relates to #2503.
2024-01-21 11:34:12 +01:00
tobi
b2cacd6b01
[bugfix] Prevent URL + URI for same account being used as alias target (#2545)
* [bugfix] Ensure URL and URI for same account can't both be provided as alias

* test whoopsie from previous PR
2024-01-20 12:45:43 +01:00
tobi
33dbd3ab7a
[bugfix] Ensure domain block side effects skipped if allow in place (blocklist mode) (#2542) 2024-01-19 13:13:24 +00:00
tobi
5ca86b1c57
[chore] Harden up boolptr logic on Accounts, warn if not set (#2544) 2024-01-19 13:02:04 +00:00
kim
7ec1e1332e
[performance] overhaul struct (+ result) caching library for simplicity, performance and multiple-result lookups (#2535)
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial

* use actual go-structr release version (not just commit hash)

* revert go toolchain changes (damn you go for auto changing this)

* fix go mod woes

* ensure %w is used in calls to errs.Appendf()

* fix error checking

* fix possible panic

* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop

* fix copy-paste artifact... 😇

* fix all comment copy-paste artifacts

* remove dropID() function, now we can just use slices.DeleteFunc()

* use util.Deduplicate() instead of collate(), move collate to util

* move orderByIDs() to util package and "generify"

* add a util.DeleteIf() function, use this to delete entries on failed population

* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)

* add note about how collate differs from deduplicate
2024-01-19 12:57:29 +00:00
kim
67e11a1a61
[chore] chore rationalise http return codes for activitypub handlers (#2540)
* some small code fixups and changes

* add check in ResolveIncomingActivity for transient activity types (i.e. activity ID is nil)

* update test to handle new transient behaviour
2024-01-18 17:11:13 +01:00
kim
906639ad7e
[chore] update viper version (#2539)
* update viper version

* removes our last uses of the slice package

* fix tests
2024-01-17 14:54:30 +00:00
tobi
c5eced5fd1
[bugfix] Better Postgres search case insensitivity (#2526)
* [bugfix] Better Postgres search case insensitivity

* use ilike for postgres
2024-01-16 18:50:17 +01:00
tobi
c36f9ac37b
[feature] Account alias / move API + db models (#2518)
* [feature] Account alias / move API + db models

* go fmt

* fix little cherry-pick issues

* update error checking, formatting

* add and use new util functions to simplify alias logic
2024-01-16 16:22:44 +00:00
tobi
ebf550b7c1
[chore] Move to codeberg's exif-terminator (#2536) 2024-01-15 20:54:32 +00:00
tobi
07aee57af2
[bugfix] Replace named unique constraint on header filter header with generic unique directive (#2525)
* [bugfix] Replace named unique constraint on header filter header with generic unique directive

* add migration retry

* the old fixie uppie

* fix constraint name

* my goodness
2024-01-15 15:39:10 +00:00
tobi
9607b482cc
[chore/docs] Replace specific year range of copyright notice (#2520) 2024-01-13 16:33:53 +01:00
kim
dfc7656579
[bugfix] fix higher-level explicit domain rules causing issues with lower-level domain blocking (#2513)
* fix the sort direction of domain cache child nodes ...

* add more domain cache test cases

* add specific test for this bug to database domain test suite (thanks for writing this @tsmethurst!)

* remove unused field (this was a previous attempt at a fix)

* remove debugging println statements 😇
2024-01-09 13:12:43 +00:00
kim
10660e566d
[bugfix] misc dereferencer fixes (#2475)
* only perform status-up-to-date checks if no statusable has been provided

* copy over the same style of freshness checking from status deref -> accounts

* change some var names

* check for empty account domain
2024-01-09 10:42:39 +01:00
tobi
7cce1a7cc6
[bugfix] Outdent placeholder from <aside> when returning unknown media attachments (#2485)
* [bugfix] Outdent placeholder from `<aside>`

* visually separate placeholder note from body

* fix test
2024-01-09 10:41:32 +01:00
tobi
aad3384c98
[feature] Log pubKeyID for http-signed requests (#2501) 2024-01-09 10:41:15 +01:00
tobi
72d0f46b0b
[feature] Allow webp emoji uploads / derefs (#2484) 2024-01-05 13:39:53 +01:00
tobi
d5e3996a18
[feature] Parse instance descriptors as markdown, show T&C on /about (#2481)
* [feature] Parse instance descriptors as markdown, show T&C on /about

* lint

* remove unnecessary nullzero tags
2024-01-05 13:39:31 +01:00
kim
511ad97fe7
[bugfix] fix check for closed poll to account for non-zero closed time but in the future (#2486) 2024-01-03 18:42:58 +01:00
kim
31481fad35
[bugfix] increases sleep time before check in throttle test, to give more leeway (#2482) 2024-01-03 10:27:55 +00:00
tobi
0ff52b71f2
[chore] Refactor HTML templates and CSS (#2480)
* [chore] Refactor HTML templates and CSS

* eslint

* ignore "Local"

* rss tests

* fiddle with OG just a tiny bit

* dick around with polls a bit more so SR stops saying "clickable"

* remove break

* oh lord

* don't lazy load avatar

* fix ogmeta tests

* clean up some cruft

* catch remaining calls to c.HTML

* fix error rendering + stack overflow in tag

* allow templating attributes

* fix indent

* set aria-hidden on status complementary content, since it's already present in the label anyway

* tidy up templating calls a little

* try to make styling a bit more consistent + readable

* fix up some remaining CSS issues

* fix up reports
2023-12-27 11:23:52 +01:00
kim
97a1fd9a29
[bugfix] 😇 (#2476) 2023-12-18 16:40:29 +00:00
kim
8ebb7775a3
[feature] request blocking by http headers (#2409) 2023-12-18 14:18:25 +00:00
Sam Lade
07bd848028
fix poll total vote double count (#2464) 2023-12-16 19:12:25 +00:00
Sam Lade
285d55dda8
[feature] Push status edit messages into open streams (#2418)
* push status edit messages into open streams

* fix a few comments

* test++

* commented out code? moi?
2023-12-16 12:55:49 +01:00
Daenney
fbe4e60232
[feature] Run ANALYZE after migrations on SQLite (#2428)
* [feature] Run ANALYZE after migrations on SQLite

This ensures that at the end of migrations, we run ANALYZE if we're
using SQLite. This should be relatively quick and guarantees that the
table and index statistics have been updated. This helps to ensure the
query planner makes better choices when it comes to picking which
indexes are used when running queries.

* [chore] use ExecContext

Uses ExecContext so we pass the context through, this is helpful for
anyone running with tracing enabled
2023-12-16 12:54:53 +01:00
kim
d56a8d095e
[performance] simpler throttling logic (#2407)
* reduce complexity of throttling logic to use 1 queue and an atomic int

* use atomic add instead of CAS, add throttling test
2023-12-16 12:53:42 +01:00
kim
f4fcffc8b5
[bugfix] use a much shorter refresh limit for statuses with polls (#2453)
* specifically use a much shorter refresh limit for statuses with polls

* allow specifying whether status must be upToDate in calls to Get(Visible)?TargetStatusBy_(), limit force refresh to 5 minute cooldown

* remove the PollID check from statusUpToDate()

* remove unnecessary force flag checks

* remove unused field

* check refresh status error

* use argument name 'refresh' instead of 'upToDate' to better fit with the codebase

* add statuses_poll_id_idx

* remove the definitely-not copy-pasted comment i accidentally typed out in full

* only synchronously refresh if the refresh flag is provided, otherwise do async

* fix wrong force value being provided for async

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2023-12-15 15:24:39 +01:00
tobi
d0bb8f0973
[bugfix] Let templates deref pointers, as a treat (#2448) 2023-12-12 14:44:54 +00:00
kim
ac48192562
[bugfix] poll vote count fixes (#2444)
* don't drop all vote counts if hideCounts is set, refactors poll option extraction slightly

* omit voters_count when not set

* make voters_count a ptr to ensure it is omit unless definitely needed

* handle case of expires_at, voters_count and option.votes_count being nilable

* faster isNil check

* remove omitempty tags since mastodon API marks things as nullable but still sets them in outgoing json
2023-12-12 13:47:07 +00:00
kim
2191c7dee5
[bugfix] ensure the 'Closing' flag doesn't get cached (#2443) 2023-12-11 12:05:54 +00:00