Commit graph

498 commits

Author SHA1 Message Date
kegsay
6284790f98
Use PDU in even more places (#3074)
- No longer rely on *Event returning from NewEventFrom... functions
 
Requires https://github.com/matrix-org/gomatrixserverlib/pull/377
2023-05-03 10:21:27 +01:00
kegsay
f5b3144dc3
Use PDU not *Event in HeaderedEvent (#3073)
Requires https://github.com/matrix-org/gomatrixserverlib/pull/376

This has numerous upsides:
 - Less type casting to `*Event` is required.
- Making Dendrite work with `PDU` interfaces means we can swap out Event
impls more easily.
 - Tests which represent weird event shapes are easier to write.

Part of a series of refactors on GMSL.
2023-05-02 15:03:16 +01:00
Devon Hudson
696cbb70b8
Pass federation API to roomserver in PurgeRoom tests 2023-05-01 21:28:10 -06:00
Till
6b47cf0f6a
Remove PerformError (#3066)
This removes `PerformError`, which was needed when we still had
polylith.

This removes quite a bunch of
```go
if err != nil {
	return err
}
if err := res.Error; err != nil {
	return err.JSONResponse()
}
```

Hopefully can be read commit by commit. [skip ci]
2023-04-28 17:46:01 +02:00
kegsay
6171310307
Use PDU interface (#3070)
We only use it in a few places currently, enough to get things to
compile and run. We should be using it in much more places.

Similarly, in some places we cast []PDU back to []*Event, we need to not
do that. Likewise, in some places we cast PDU to *Event, we need to not
do that. For now though, hopefully this is a start.
2023-04-27 16:35:19 +01:00
kegsay
b189edf4f4
Remove gmsl.HeaderedEvent (#3068)
Replaced with types.HeaderedEvent _for now_. In reality we want to move
them all to gmsl.Event and only use HeaderedEvent when we _need_ to
bundle the version/event ID with the event (seriailsation boundaries,
and even then only when we don't have the room version).

Requires https://github.com/matrix-org/gomatrixserverlib/pull/373
2023-04-27 12:54:20 +01:00
Till
2475cf4b61
Add some roomserver UTs (#3067)
Adds tests for `QueryRestrictedJoinAllowed`, `IsServerAllowed` and
`PerformRoomUpgrade`. Refactors the `QueryRoomVersionForRoom` method to
accept a string and return a `gmsl.RoomVersion` instead of req/resp
structs.
Adds some more caching for `GetStateEvent`

This should also fix #2912 by ignoring state events belonging to other
users.
2023-04-27 08:07:13 +02:00
kegsay
4679098a64
Use IRoomVersion (#3064)
This is a step towards allowing arbitrary room version impls.
2023-04-24 11:50:37 +01:00
kegsay
1647213fac
Implement new RoomVersionImpl API (#3062)
As outlined in https://github.com/matrix-org/gomatrixserverlib/pull/368

The main change Dendrite side is that `RoomVersion` no longer has any
methods on it. Instead, you need to bounce via `gmsl.GetRoomVersion`.

It's very interesting to see where exactly Dendrite cares about this.
For some places it's creating events (fine) but others are way more
specific. Those areas will need to migrate to GMSL at some point.
2023-04-21 17:06:29 +01:00
kegsay
71eeccf34a
refactor: funnel event creation through room versions (#3060)
In preparation of interfacing up the room version value.
2023-04-20 19:07:31 +01:00
kegsay
72285b2659
refactor: update GMSL (#3058)
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364

Read this commit by commit to avoid going insane.
2023-04-19 15:50:33 +01:00
devonh
f66862958d
Remove event building duplication & push to GMSL (#3056)
Removes event building duplication and moves the funcionality into GMSL
since all the sub-steps are already there.
2023-04-14 15:03:07 +00:00
kegsay
ca63b414da
Update GMSL: use static Check functions (#3052)
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/359 

A nice side effect is that we don't need to re-parse the events in some
cases.
2023-04-14 12:32:42 +01:00
kegsay
0db43f13a6
refactor: use latest GMSL which splits fed client from matrix room logic (#3051)
Part of a series of refactors on GMSL.
2023-04-06 09:55:01 +01:00
kegsay
3691423626
Move GMSL client types to Dendrite (#3045)
GMSL is intended for Federation only. Sister PR to
https://github.com/matrix-org/gomatrixserverlib/pull/357
2023-04-04 17:16:53 +00:00
Till
682a7d0a66
Add tests for /turnServer, /capabilities and /3pid/ (#3038)
Threepid seems to be pretty out of date, several missing endpoints.
Should also fix #3037, where we were still listening on the `/unstable`
prefix, while Element Web uses `/r0`
2023-04-03 21:42:46 +02:00
Till
fa7710315a
Add tests for the Dendrite admin APIs (#3028)
Contains a breaking change, since the endpoints `/_dendrite/admin/evacuateRoom/{roomID}` and `/_dendrite/admin/evacuateUser/{userID}` are now using `POST` instead of `GET`
2023-03-27 15:39:33 +02:00
Till
5e85a00cb3
Remove BaseDendrite (#3023)
Removes `BaseDendrite` to, hopefully, make testing and composing of
components easier in the future.
2023-03-22 09:21:32 +01:00
Till
5579121c6f
Preparations for removing BaseDendrite (#3016)
Preparations to actually remove/replace `BaseDendrite`.
Quite a few changes:
- SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed
from `BaseDendrite`)
- Caches are removed from `BaseDendrite`
- Introduces a `Router` struct (likely to change)
  - also fixes #2903
- Introduces a `sqlutil.ConnectionManager`, which should remove
`base.DatabaseConnection` later on
- probably more
2023-03-17 11:09:45 +00:00
Till
232aef016c
Add basic runtime tracing (#2996)
This allows us in almost all places to use regions to further trace down
long running tasks.
Also removes an unused function.
2023-03-13 16:45:14 +01:00
Till
70322699ab
Unset RoomServerEvent, since we can't be sure that Set actually updates the cached entry (#3002)
This should deflake UTs and be more correct in terms of getting
`Events`.
`Events` tries to fetch the event from the cache first and may get an
unredacted event from it, while it should already be redacted.
2023-03-09 09:52:13 +01:00
Till Faelligen
baef523cb0
Fix invalid roomNID returned 2023-03-07 15:35:08 +01:00
Till Faelligen
a684b850b9
Actually ignore the error if we were able to backfill events 2023-03-06 17:45:21 +01:00
Till Faelligen
56b28b01db
Update the cache with the redacted event 2023-03-03 14:49:41 +01:00
Till
9bcd0a2105
Make redaction check easier to read (#2995)
We need to check the redaction PL in Dendrite, if we do it in GMSL, we
end up not sending the event to the output stream because it will be
rejected.

---------

Co-authored-by: kegsay <kegan@matrix.org>
2023-03-03 14:03:17 +01:00
Till
6c20f8f742
Refactor StoreEvent, add MaybeRedactEvent, create an EventDatabase (#2989)
This PR changes the following:
- `StoreEvent` now only stores an event (and possibly prev event),
instead of also doing redactions
- Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should
be called after storing events
- a few other things
2023-03-01 17:06:47 +01:00
Till Faelligen
3d31b131fc
Cache all the things 2023-02-24 11:45:01 +01:00
Till
ad07b169b8
Refactor StoreEvent and create a new RoomDatabase interface (#2985)
This PR changes a few things:
- It pulls out the creation of several NIDs from the `StoreEvent`
function to make the functions more reusable
- Uses more caching when using those NIDs to avoid DB round trips
2023-02-24 09:40:20 +01:00
Till
4594233f89
Merge keyserver & userapi (#2972)
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
Till
11d9b9db0e
Remove polylith/API mode (#2967)
This removes most of the code used for polylith/API mode.

This removes the `/api` internal endpoints entirely. 

Binary size change roughly 5%: 
```
51437560 Feb 13 10:15 dendrite-monolith-server # old
48759008 Feb 13 10:15 dendrite-monolith-server # new
```
2023-02-14 12:47:47 +01:00
Till
eb29a31550
Optimize /sync and history visibility (#2961)
Should fix the following issues or make a lot less worse when using
Postgres:

The main issue behind #2911: The client gives up after a certain time,
causing a cascade of context errors, because the response couldn't be
built up fast enough. This mostly happens on accounts with many rooms,
due to the inefficient way we're getting recent events and current state

For #2777: The queries for getting the membership events for history
visibility were being executed for each room (I think 185?), resulting
in a whooping 2k queries for membership events. (Getting the
statesnapshot -> block nids -> actual wanted membership event)

Both should now be better by:
- Using a LATERAL join to get all recent events for all joined rooms in
one go (TODO: maybe do the same for room summary and current state etc)
- If we're lazy loading on initial syncs, we're now not getting the
whole current state, just to drop the majority of it because we're lazy
loading members - we add a filter to exclude membership events on the
first call to `CurrentState`.
- Using an optimized query to get the membership events needed to
calculate history visibility

---------

Co-authored-by: kegsay <kegan@matrix.org>
2023-02-07 14:31:23 +01:00
Till
baf118b08c
Add Sytest/Complement coverage to scheduled runs (#2962)
This adds Sytest and Complement coverage reporting to the nightly
scheduled CI runs.

Fixes a few API mode related issues as well, since we seemingly never
really ran them with Complement.

Also fixes a bug related to device list changes: When we pass in an
empty `newlyLeftRooms` slice, we got a list of all currently joined
rooms with the corresponding members. When we then got the
`newlyJoinedRooms`, we wouldn't update the `changed` slice, because we
already got the user from the `newlyLeftRooms` query. This is fixed by
simply ignoring empty `newlyLeftRooms`.
2023-02-03 13:42:35 +01:00
devonh
4738fe656f
Roomserver published pkey migration (#2960)
Adds a missed migration to update the primary key on the
roomserver_published table in postgres.
Primary key was changed in #2836.
2023-02-01 16:32:31 +00:00
Catalan Lover
25cb65acdb
Change Default Room version to 10 (#2933)
This PR implements
[MSC3904](https://github.com/matrix-org/matrix-spec-proposals/pull/3904).
This PR is almost identical to #2781 but this PR is also filed well
technically 1 day before the MSC passes FCP but well everyone knows this
MSC is expected to have passed FCP on monday so im refiling this change
today on saturday as i was doing prep work for monday. I assume that
this PR wont be counted as clogging the queue since by the next time i
expect to be a work day for this project this PR will be implementing an
FCP passed disposition merge MSC.

Also as for the lack of tests i belive that this simple change does not
need to pass new tests due to that these tests are expected to already
have been passed by the successful use of Dendrite with Room version 10
already.

### Pull Request Checklist

* [X] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [X] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: Catalan Lover <catalanlover@protonmail.com>

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2023-01-20 15:41:29 +01:00
Till
caf310fd79
AWSY missing federation tests (#2943)
In an attempt to fix the missing AWSY tests and to get to 100%
server-server compliance.
2023-01-20 15:18:06 +01:00
Neil
738686ae68
Add /_dendrite/admin/purgeRoom/{roomID} (#2662)
This adds a new admin endpoint `/_dendrite/admin/purgeRoom/{roomID}`. It
completely erases all database entries for a given room ID.

The roomserver will start by clearing all data for that room and then
will generate an output event to notify downstream components (i.e. the
sync API and federation API) to do the same.

It does not currently clear media and it is currently not implemented
for SQLite since it relies on SQL array operations right now.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Till Faelligen <2353100+S7evinK@users.noreply.github.com>
2023-01-19 21:02:32 +01:00
Till
2e1fe58937
Fix backfilling (#2926)
This should fix https://github.com/matrix-org/dendrite/issues/2923
2023-01-05 09:24:00 +01:00
Till
5eed31fea3
Handle guest access [1/2?] (#2872)
Needs https://github.com/matrix-org/sytest/pull/1315, as otherwise the
membership events aren't persisted yet when hitting `/state` after
kicking guest users.

Makes the following tests pass:
```
Guest users denied access over federation if guest access prohibited
Guest users are kicked from guest_access rooms on revocation of guest_access
Guest users are kicked from guest_access rooms on revocation of guest_access over federation
```

Todo (in a follow up PR):
- Restrict access to CS API Endpoints as per
https://spec.matrix.org/v1.4/client-server-api/#client-behaviour-14

Co-authored-by: kegsay <kegan@matrix.org>
2022-12-22 13:05:59 +01:00
Till
7d2344049d
Cleanup stale device lists for users we don't share a room with anymore (#2857)
The stale device lists table might contain entries for users we don't
share a room with anymore. This now asks the roomserver about left users
and removes those entries from the table.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-12-12 08:20:59 +01:00
Till
e245a26f6b
Enable/Disable internal metrics (#2899)
Basically enables us to use `test.WithAllDatabases` when testing
internal HTTP APIs, as this would otherwise result in Prometheus
complaining about already registered metric names.
2022-12-05 13:53:36 +01:00
Neil Alexander
9b8bb55430
Don't get blacklisted hosts when querying joined servers (#2880)
Otherwise we just waste time/CPU.
2022-11-15 17:21:16 +00:00
Neil Alexander
6650712a1c
Federation fixes for virtual hosting 2022-11-15 15:05:23 +00:00
Till
2a77a910eb
Handle remote room upgrades (#2866)
Makes the following tests pass
```
/upgrade moves remote aliases to the new room
Local and remote users' homeservers remove a room from their public directory on upgrade
```
2022-11-14 12:07:13 +00:00
Till
1e79b0557e
Use a writer to assign state key NIDs (#2877) 2022-11-14 12:06:27 +00:00
Till Faelligen
e177e0ae73
Fix oops, add simple UT 2022-11-11 16:44:59 +01:00
Till
c648c671a3
Fix issue with missing user NIDs (#2874)
This should fix #2696 and possibly other related issues regarding
missing user NIDs.
(https://github.com/matrix-org/dendrite/issues/2094?)
2022-11-11 10:52:43 +01:00
Till
0193549201
Send presence to newly added servers (#2869)
This should make `New federated private chats get full presence
information (SYN-115)` happy.
2022-11-11 10:35:17 +01:00
Neil Alexander
16c2a95900
Improve logging for processEventWithMissingState 2022-11-02 11:30:49 +00:00
Neil Alexander
6663728eb1
Fix SQLite roomserver_published migration 2022-11-01 16:08:13 +00:00
Till
2acc1d65fb
Optimize history visibility checks (#2848)
This optimizes history visibility checks by (mostly) avoiding database
hits.
Possibly solves https://github.com/matrix-org/dendrite/issues/2777

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-11-01 15:07:17 +00:00
Neil Alexander
7bd6631935
Move code for calculating auth difference into GMSL 2022-11-01 10:12:11 +00:00
Neil Alexander
f10c6f26e5
Add /_dendrite/admin/downloadState/{serverName}/{roomID} admin endpoint 2022-10-31 09:13:28 +00:00
Till Faelligen
a785532463
Fix upgrade appservices 2022-10-27 16:01:51 +02:00
Till
444b4bbdb8
Add AS specific public room list endpoints (#2836)
Adds `PUT
/_matrix/client/v3/directory/list/appservice/{networkId}/{roomId}` and
`DELTE
/_matrix/client/v3/directory/list/appservice/{networkId}/{roomId}`
support, as well as the ability to filter `/publicRooms` on networkID
and including all networks.
2022-10-27 14:40:35 +02:00
Neil Alexander
f6dea712d2
Initial support for multiple server names (#2829)
This PR is the first step towards virtual hosting by laying the
groundwork for multiple server names being configured.
2022-10-26 12:59:19 +01:00
Neboer
2a4c7f45b3
Add support for config "auto_join_rooms" (#2823)
Add support for config "auto_join_rooms". Now new accounts can join the
rooms in config file automatically.

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have justified why this PR doesn't need tests.
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)

Signed-off-by: `Rubin Poster <rubinposter@gmail.com>`
2022-10-26 11:04:53 +02:00
Neil Alexander
cd8f7e1251
Set inactivity threshold on durable consumers in the roomserver input API (#2795)
This prevents us from holding onto durable consumers indefinitely for
rooms that have long since turned inactive, since they do have a bit of
a processing overhead in the NATS Server. If we clear up a consumer and
then a room becomes active again, the consumer gets recreated as needed.

The threshold is set to 24 hours for now, we can tweak it later if needs
be.
2022-10-14 15:14:29 +01:00
Till
088ad1dd21
Fix outliers whose auth_events are in a different room are correctly rejected (#2791)
Fixes `outliers whose auth_events are in a different room are correctly
rejected`, by validating that auth events are all from the same room and
not using rejected events for event auth.
2022-10-14 09:14:54 +02:00
Till
3c1474f68f
Fix /get_missing_events for rooms with joined/invited history_visibility (#2787)
Sytest was using a wrong `history_visibility` for `invited`
(https://github.com/matrix-org/sytest/pull/1303), so `invited` was
passing for the wrong reason (-> defaulted to `shared`, as `invite`
wasn't understood).
This change now handles missing events like Synapse, if a server isn't
allowed to see the event, it gets a redacted version of it, making the
`get_missing_events` tests pass.
2022-10-11 16:04:02 +02:00
Till
1ca3f3efb5
Fix issue with DMs shown as normal rooms (#2776)
Fixes #2121, test added in
https://github.com/matrix-org/complement/pull/494
2022-10-07 16:00:12 +02:00
Neil Alexander
8e231130e9
Revert "tDatabase transaction tweaks in roomserver"
This reverts commit 8d8f4689a0.
2022-10-07 14:05:06 +01:00
Neil Alexander
8d8f4689a0
tDatabase transaction tweaks in roomserver 2022-10-07 12:21:55 +01:00
Neil Alexander
3f82bceb70
Don't try to talk to ourselves when finding missing events 2022-10-06 10:51:06 +01:00
Neil Alexander
c85bc3434f
Optimise QuerySharedUsers so that we can only work on local users (#2766)
Otherwise the sync API key change consumer wastes a lot of time trying
to wake up the notifiers for non-local users.
2022-10-05 12:47:53 +01:00
Till
8c0c3441d8
Add RoomEventType nats.Header to avoid unneeded unmarshalling (#2765) 2022-10-05 12:12:42 +02:00
Neil Alexander
ae10aac456
Don't perform a federated join after invite if we are already joined to the room (#2762)
If we are already joined to the room then it shouldn't matter if you
were invited or not, so this looks like a bug.
2022-10-04 15:40:04 +01:00
Neil Alexander
f022fc1397
Remove origin field from PDUs (#2737)
This nukes the `origin` field from PDUs as per
matrix-org/matrix-spec#998, matrix-org/gomatrixserverlib#341.
2022-09-26 17:35:35 +01:00
Neil Alexander
fc1d8e479b
Ensure that all state event IDs are included in the added section when rewriting state (#2725)
This should hopefully fix an entire class of problems where components
downstream from the roomserver (i.e. the sync API) could just lose a
whole bunch of state after a rewrite operation like a federated join.

The root of the bug is that we set `RewritesState` in the output event
which instructs downstream components to purge their copy of any room
state, but then didn't send the entire state snapshot in
`adds_state_event_ids` so the downstream state ends up being incomplete
as a result.
2022-09-16 10:35:32 +01:00
Neil Alexander
7f89fed1e4
Revert 482914aef4 2022-09-14 09:55:50 +01:00
Neil Alexander
482914aef4
Use AckNone on the ephemeral room input consumer 2022-09-13 15:25:02 +01:00
Neil Alexander
b05e028f7d
Tweak LoadMembershipAtEvent behaviour when state not known (#2716)
Previously `LoadMembershipAtEvent` would fail if the state before one of
the events was not known, i.e. because it was an outlier. This modifies
it so that it gracefully handles not knowing the state and returns no
memberships instead, so that history visibility doesn't freak out and
kill `/sync` requests dead.
2022-09-13 12:52:09 +01:00
Neil Alexander
3e55856254
Always resolve state in QueryStateAfterEvents 2022-09-13 09:37:38 +01:00
Till
100fa9b235
Check unique constraint errors when manually inserting migrations (#2712)
This should avoid unnecessary logging on startup if the migration (were
we need `InsertMigration`) was already executed.
This now checks for "unique constraint errors" for SQLite and Postgres
and fails the startup process if the migration couldn't be manually
inserted for some other reason.
2022-09-13 08:07:43 +02:00
Neil Alexander
2792d0490f
Fix missing signature check on the /get_missing_events response 2022-09-12 13:30:51 +01:00
Neil Alexander
c0e17bbe1b
Fix transactions around assigning NIDs 2022-09-09 13:30:09 +01:00
Till
8196b29657
Change detection of already executed migrations (#2665)
This changes the detection of already executed migrations for the
roomserver state block and keychange refactor. It now uses schema tables
provided by the database engine to check if the column was already
removed. We now also store the migration in the migrations table.

This should stop e.g. Postgres from logging errors like `ERROR: column
"event_nid" does not exist at character 8`.
2022-09-09 13:14:52 +01:00
Neil Alexander
31f4ae8997
Use a single context instead 2022-09-07 16:24:43 +01:00
Neil Alexander
5014b35bd7
Update state reset capture to Sentry 2022-09-07 16:23:22 +01:00
Neil Alexander
cd22ba22b0
Improve Sentry reporting 2022-09-05 17:25:11 +01:00
Neil Alexander
ecee5f10f4
Tweak logging for detected state resets 2022-09-05 17:08:54 +01:00
Neil Alexander
d1f87e63f1
Move SetLatestEvents call 2022-09-05 13:16:14 +01:00
Neil Alexander
8ff3f1a7c9
Remove a couple unnecessary Sentry captures from backfill 2022-08-25 11:01:07 +01:00
Neil Alexander
cd7fa34595
Tweak logging and Sentry reporting for roomserver input 2022-08-25 10:57:27 +01:00
Neil Alexander
16156b0b09
Fix 500s on /state, /state_ids when state not known (#2672)
This was due to bad error bubbling.
2022-08-25 09:51:36 +01:00
Neil Alexander
522bd2999f
Allow un-rejecting events on reprocessing 2022-08-24 14:03:06 +01:00
Neil Alexander
14fea600bb
Detect types.MissingStateError in CheckServerAllowedToSeeEvent (#2667)
This will hopefully stop some 500 errors on `/event` where there is no state-before known.
2022-08-23 13:57:11 +01:00
Neil Alexander
2668050e53
Tweak soft-failure handling in roomserver
commit 1929b688e31987c46e0c8a546f0f9cb0a46bf9a3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:09:44 2022 +0100

    Still process state-before for soft-failed events

commit e83c0b701d40d78b92072c4643f6bc6f71b72800
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:06:50 2022 +0100

    Improve logging

commit 29e26124bc27cb83d449de2a4214b253c594aa93
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 09:58:13 2022 +0100

    Don't store soft-failed events as rejected
2022-08-22 10:34:07 +01:00
Till
365da70a23
Set historyVisibility for backfilled events over federation (#2656)
This should hopefully deflake Backfill works correctly with history visibility set to joined as we were using the default shared visibility, even if the events are set to joined (or something else)
2022-08-19 11:04:26 +02:00
Neil Alexander
6b48ce0d75
State handling tweaks (#2652)
This tweaks how rejected events are handled in room state and also to not apply checks we can't complete to outliers.
2022-08-18 17:06:13 +01:00
Neil Alexander
59bc0a6f4e
Reprocess rejected input events (#2647)
* Reprocess outliers that were previously rejected

* Might as well do all events this way

* More useful errors

* Fix queries

* Tweak condition

* Don't wrap errors

* Report more useful error

* Flatten error on `r.Queryer.QueryStateAfterEvents`

* Some more debug logging

* Flatten error in `QueryRestrictedJoinAllowed`

* Revert "Flatten error in `QueryRestrictedJoinAllowed`"

This reverts commit 1238b4184c.

* Tweak `QueryStateAfterEvents`

* Handle MissingStateError too

* Scope to room

* Clean up

* Fix the error

* Only apply rejection check to outliers
2022-08-18 10:37:47 +01:00
Till
b4647fbb7e
Show/hide users in user directory (#2637)
* CS API changes

* Query remote profiles

* Add passing tests

* Don't create a new FullyQualifiedProfile

* Handle sql.ErrNoRows

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 13:33:31 +02:00
Neil Alexander
a01af55ec6
Restore the room version cache in the roomserver internal API HTTP client 2022-08-11 17:34:09 +01:00
Till
05cafbd197
Implement history visibility on /messages, /context, /sync (#2511)
* Add possibility to set history_visibility and user AccountType

* Add new DB queries

* Add actual history_visibility changes for /messages

* Add passing tests

* Extract check function

* Cleanup

* Cleanup

* Fix build on 386

* Move ApplyHistoryVisibilityFilter to internal

* Move queries to topology table

* Add filtering to /sync and /context
Some cleanup

* Add passing tests; Remove failing tests :(

* Re-add passing tests

* Move filtering to own function to avoid duplication

* Re-add passing test

* Use newly added GMSL HistoryVisibility

* Update gomatrixserverlib

* Set the visibility when creating events

* Default to shared history visibility

* Remove unused query

* Update history visibility checks to use gmsl
Update tests

* Remove unused statement

* Update migrations to set "correct" history visibility

* Add method to fetch the membership at a given event

* Tweaks and logging

* Use actual internal rsAPI, default to shared visibility in tests

* Revert "Move queries to topology table"

This reverts commit 4f0d41be9c.

* Remove noise/unneeded code

* More cleanup

* Try to optimize database requests

* Fix imports

* PR peview fixes/changes

* Move setting history visibility to own migration, be more restrictive

* Fix unit tests

* Lint

* Fix missing entries

* Tweaks for incremental syncs

* Adapt generic changes

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-11 18:23:35 +02:00
Neil Alexander
371336c6b5
Set default room version to 9 2022-08-11 16:31:44 +01:00
Neil Alexander
c45d0936b5
Generic-based internal HTTP API (#2626)
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)

* Add `PerformInvite`

* More tweaks

* Fix metric name

* Fix LookupStateIDs

* Lots of changes to clients

* Some serverside stuff

* Some error handling

* Use paths as metric names

* Revert "Use paths as metric names"

This reverts commit a9323a6a34.

* Namespace metric names

* Remove duplicate entry

* Remove another duplicate entry

* Tweak error handling

* Some more tweaks

* Update error behaviour

* Some more error tweaking

* Fix API path for `PerformDeleteKeys`

* Fix another path

* Tweak federation client proxying

* Fix another path

* Don't return typed nils

* Some more tweaks, not that it makes any difference

* Tweak federation client proxying

* Maybe fix the key backup test
2022-08-11 15:29:33 +01:00
Till
03ddd98f5e
Fix issues with migrations not getting executed (#2628)
* Fix issues with migrations not getting executed

* Check actual postgres error

* Return error if it's not "column does not exist"
2022-08-08 10:18:57 +02:00
Till
1b7f84250a
Fix linter issues (#2624)
* Try that again

* All hail the mighty linter?

* And once again

* goimport all the things
2022-08-05 11:12:41 +02:00
Neil Alexander
3bf5ae5ffe
Try more servers when calling /state_ids (#2610)
* Try more servers when calling `/state_ids`

* More logging

* Maybe fix concurrent map write

* Revert "Maybe fix concurrent map write"

This reverts commit da0dbb8362.

* Enforce a limit of 20s per server, 5 mins total
2022-08-03 17:37:27 +01:00
Neil Alexander
2250768be1
Remove roominfo cache (#2615)
* Remove roominfo cache

It's the source of a number of race conditions which are seemingly causing bugs and CI failures.

* Make the linter less sad
2022-08-03 17:14:21 +01:00
Neil Alexander
f4345dafde
Fix data race in lookupMissingStateViaStateIDs 2022-08-02 13:01:03 +01:00