Commit graph

1266 commits

Author SHA1 Message Date
Neil Alexander
fed3ebd2f1
CodeQL analysis 2020-09-30 21:18:35 +01:00
Neil Alexander
0caad67abc
Consider old keys in handleLocalKeys (#1454) 2020-09-30 19:52:36 +01:00
Neil Alexander
05e5386fb0
Update gobind database paths 2020-09-30 15:57:31 +01:00
Neil Alexander
135b5e264f
Fix panic on verifySigError in fetching missing events 2020-09-30 13:51:54 +01:00
Neil Alexander
beaf4cc00f
Fix perspective fetching 2020-09-29 17:34:48 +01:00
Neil Alexander
d63d7c5640
Tweak log level of a fairly common log line 2020-09-29 17:08:47 +01:00
Neil Alexander
f290e92a34
Remove TLS fingerprints, improve perspective unmarshal handling (#1452)
* Add prefer_direct_fetch option

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Don't deal in TLS fingerprints anymore
2020-09-29 17:08:18 +01:00
Neil Alexander
43cdba9a69
Ignore depth in federation API (#1451) 2020-09-29 14:07:59 +01:00
Neil Alexander
b0d5d1cc9f
Fix old verify keys 2020-09-29 14:00:12 +01:00
Neil Alexander
738b829a23
Fetch missing auth events, implement QueryMissingAuthPrevEvents, try other servers in room for /event and /get_missing_events (#1450)
* Try to ask other servers in the room for missing events if the origin won't provide them

* Logging

* More logging

* Implement QueryMissingAuthPrevEvents

* Try to get missing auth events badly

* Use processEvent

* Logging

* Update QueryMissingAuthPrevEvents

* Try to find missing auth events

* Patchy fix for test

* Logging tweaks

* Send auth events as outliers

* Update check in QueryMissingAuthPrevEvents

* Error responses

* More return codes

* Don't return error on reject/soft-fail since it was ultimately handled

* More tweaks

* More error tweaks
2020-09-29 13:40:29 +01:00
bn4t
4ff7ac7b65
Fix a small typo (#1448)
Signed-off-by: Benjamin Nater <me@bn4t.me>
2020-09-29 10:07:23 +01:00
bn4t
da89f2bd1f
Finish implementing the capabilities endpoint (#1449)
Closes #1310

Signed-off-by: Benjamin Nater <me@bn4t.me>
2020-09-29 10:06:59 +01:00
Neil Alexander
ce318f53bc
Use workers when fetching events from /state_ids, use /state only if significant portion of events missing (#1447)
* Don't fall back to /state on incoming /send

* Event workers for /state_ids, use /state only if significant percentage of events are missing
2020-09-28 11:32:59 +01:00
Neil Alexander
40dd16a6e6
Don't fall back to /state on incoming /send (#1446) 2020-09-28 10:03:18 +01:00
Matthew Hodgson
15fe61ed58 oops 2020-09-27 22:24:17 +01:00
Matthew Hodgson
5bbe73747c unbreak CORS on /capabilities.
fixes https://github.com/vector-im/element-web/issues/15297
2020-09-27 22:23:42 +01:00
Neil Alexander
63af00d5d5
Update gomatrixserverlib to matrix-org/gomatrixserverlib#225 2020-09-25 17:53:16 +01:00
Neil Alexander
6275669e65
Set default room version to v6 (#1438) 2020-09-25 12:59:57 +01:00
Neil Alexander
145db37d89
Allow configuring old verify keys (#1443)
* Allow configuring old verify keys

* Update sample config

* Update sample config

* Fix config population

* Key ID formatting validity of old_verify_keys

* Update comment
2020-09-25 10:58:53 +01:00
Neil Alexander
6fbf89a166
Return the correct error codes for v6 invite JSON violations (#1440)
* Return the correct error codes for v6 invite JSON violations

* Update sytest-whitelist
2020-09-24 17:16:59 +01:00
Neil Alexander
3013ade84f
Reject make_join for empty rooms (#1439)
* Sanity-check room version on RS event input

* Update gomatrixserverlib

* Reject make_join when no room members are left

* Revert some changes from wrong branch

* Distinguish between room not existing and room being abandoned on this server

* nolint
2020-09-24 16:18:13 +01:00
Sam
a6700331ce
Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)
* Replace all usages of txn.Stmt with sqlutil.TxStmt

Signed-off-by: Sam Day <me@samcday.com>

* Fix sign off link in PR template.

Signed-off-by: Sam Day <me@samcday.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-24 11:10:14 +01:00
Neil Alexander
60524f4b99
Update gomatrixserverlib to matrix-org/gomatrixserverlib#223 2020-09-23 12:47:16 +01:00
Neil Alexander
de8b39065e
Enforce valid key IDs (#1437)
* Enforce valid key IDs

* Don't use key_id from dendrite.yaml as it is in matrix_key.pem
2020-09-23 11:07:57 +01:00
Neil Alexander
f908f8baab
Update gomatrixserverlib 2020-09-22 16:41:46 +01:00
Neil Alexander
a854e3aa18
Fix backoff bug 2020-09-22 14:53:36 +01:00
Neil Alexander
a14b29b526
Initial notary support (#1436)
* Initial work on notary support

* Somewhat working (but not properly filtered) notary support, other tweaks

* Update gomatrixserverlib
2020-09-22 14:40:54 +01:00
Neil Alexander
a7563ede3d
Process federated joins in background context (#1434)
* Return early from federated room join

* Synchronous perform-join as long as possible

* Don't allow multiple federated joins to the same room by the same user
2020-09-22 11:05:45 +01:00
Neil Alexander
45de9dc1c0
Use room version cache in Events() 2020-09-21 16:49:37 +01:00
Neil Alexander
a06c18bb56
Soft-fail (#1364)
* Initial work on soft-fail

* Fix state block retrieval

* Copy-pasta QueryLatestEventsAndState code

* Fix state lookup

* Clean up

* Fix up failing sytest

* Linting

* Update previous events SQLite insert query

* Update SQLite InsertPreviousEvent properly

* Hopefully fix the event references updates

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-21 14:55:46 +01:00
Neil Alexander
880b164490
Refactor backoff again (#1431)
* Tweak backoffs

* Refactor backoff some more, remove BackoffIfRequired as it adds unnecessary complexity

* Ignore 404s
2020-09-21 13:30:37 +01:00
Kegsay
18231f25b4
Implement rejected events (#1426)
* WIP Event rejection

* Still send back errors for rejected events

Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.

* Implement rejected events

Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.

* Update test to match reality

* Modify InputRoomEvents to no longer return an error

Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.

* Remove redundant returns; linting

* Update blacklist
2020-09-16 13:00:52 +01:00
Kegan Dougal
ba6c7c4a5c Disable prometheus to unbreak tests 2020-09-15 16:15:34 +01:00
Neil Alexander
965f068d1a
Handle state with input event as new events (#1415)
* SendEventWithState events as new

* Use cumulative state IDs for final event

* Error wrapping in calculateAndSetState

* Handle overwriting same event type and state key

* Hacky way to spot historical events

* Don't exclude from sync

* Don't generate output events when rewriting forward extremities

* Update output event check

* Historical output events

* Define output room event type

* Notify key changes on state

* Don't send our membership event twice

* Deduplicate state entries

* Tweaks

* Remove unnecessary nolint

* Fix current state upsert in sync API

* Send auth events as outliers, state events as rewrite

* Sync API don't consume state events

* Process events actually

* Improve outlier check

* Fix local room check

* Remove extra room check, it seems to break the whole damn world

* Fix federated join check

* Fix nil pointer exception

* Better comments on DeduplicateStateEntries

* Reflow forced federated joins

* Don't force federated join for possibly even local invites

* Comment SendEventWithState better

* Rewrite room state in sync API storage

* Add TODO

* Clean up all room data when receiving create event

* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event

* Rename to PurgeRoom

* Exclude backfilled messages from /sync

* Split out rewriting state from updating state from state res

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-15 11:17:46 +01:00
Kegsay
8dc9506210
Don't use more than 999 variables in SQLite querys. (#1425)
* Don't use more than 999 variables in SQLite querys.

Solve this problem in a more general and reusable way.
Also fix #1369
Add some unit tests.

Signed-off-by: Henrik Sölver <henrik.solver@gmail.com>

* Don't rely on testify for basic assertions

* Readability improvements and linting

Co-authored-by: Henrik Sölver <henrik.solver@gmail.com>
2020-09-14 16:39:38 +01:00
Kegsay
913020e4b7
Add hardware requirements; fixes #324 (#1420) 2020-09-10 17:37:31 +01:00
Kegsay
95d7e2336d
Add support for database migrations (#1416)
* Add support for database migrations

Closes #1246

This PR does NOT add any migrations as an example. I have
manually tested that the library works with SQL and Go based
upgrades correctly. Documentation should be sufficient for
devs to add migrations.

* Clarifications

* Linting
2020-09-10 15:10:32 +01:00
Matthew Hodgson
39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Neil Alexander
35564dd73c
Process membership updates in writers (#1414) 2020-09-08 17:48:07 +01:00
Neil Alexander
a0f2a4510f
Exclude deleted keys from selectBatchDeviceKeysSQL (#1412) 2020-09-08 17:47:54 +01:00
Kegsay
b4bd0cc0f5
Track goids when running with tracing enabled (#1413)
* Track goids when running with tracing enabled

* Linting
2020-09-08 17:30:05 +01:00
Neil Alexander
668a722ee0
Backoff for 401s (#1410)
* Backoff for 401s

* Human-readable retry_after in logs
2020-09-08 13:41:08 +01:00
Neil Alexander
726ad6ce2e
Backoff ignore invalid signatures (#1408) 2020-09-08 10:28:13 +01:00
Neil Alexander
1602df8752
Ignore state events with invalid signatures when joining rooms (#1407)
* Use state from RespSendJoin post-check

* Don't create input events for invalid events

* Let's try this again

* Update gomatrixserverlib

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@38f437f
2020-09-07 16:54:51 +01:00
Kegsay
c992f4f1f4
Remove current state server (#1405)
* Remove current state server

Closes #1365 #1272 #1357

* Remove current state server from scripts/docs
2020-09-07 14:47:59 +01:00
Neil Alexander
8589f8373e
Update gomatrixserverlib to matrix-org/gomatrixserverlib@66753e2 2020-09-07 14:39:04 +01:00
Kegsay
7913759921
Remove QueryBulkStateContent from current state server (#1404)
* Remove QueryBulkStateContent from current state server

Expected fail due to db impl not existing

* Implement query bulk state content

* Fix up rejecting invites over federation

* Fix bulk content marshalling
2020-09-07 12:38:09 +01:00
Neil Alexander
895ead8048
Use background context when processing event with missing state (#1403)
* Use background context when processing event with missing state

* Five minute timeout

* Remove context from txnreq, thread through instead

* Fix unit tests
2020-09-07 12:32:40 +01:00
Neil Alexander
b9caccbce8
Update GetStateEvent behaviour (#1399) 2020-09-04 19:40:21 +01:00
Kegsay
088294ee65
Remove QueryRoomsForUser from current state server (#1398) 2020-09-04 15:58:30 +01:00