Commit graph

166 commits

Author SHA1 Message Date
Neil Alexander
68d92fd3b4
Update sytest-whitelist/blacklist 2020-12-04 14:09:21 +00:00
alexkursell
2b03d24358
Fix /joined_members API response (#1606)
* Fix /joined_members API response

* Fix golint issue
2020-12-03 11:01:49 +00:00
bodqhrohro
1cf9f20d2f
Allow = in user identifiers (#1590)
While I was breaking through all the TDD bureaucracy to finally push this fix,
it turned out that it already got fixed in #1578. Still I push the rest of the
changes (basically, everything except of actually fixing the bug ;)).

`=` is crucial for matrix-bifrost which turns XMPP @'s into =40.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
2020-11-18 22:56:06 +00:00
David Spenler
35ea55e70b
Implemented whois endpoint (#1573)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-17 10:07:03 +00:00
Neil Alexander
3034d8e805
Update sytest lists to blacklist flaky 'Forgotten room messages cannot be paginated' test 2020-11-16 10:51:55 +00:00
S7evinK
bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a046.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f64.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
S7evinK
eccd0d2c1b
Implement forgetting about rooms (#1572)
* Add basic storage methods

* Add internal api handler

* Add check for forgotten room

* Add /rooms/{roomID}/forget endpoint

* Add missing rsAPI method

* Remove unused parameters

* Add passing tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing file

* Add postgres migration

* Add sqlite migration

* Use Forgetter to forget room

* Remove empty line

* Update HTTP status codes

It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-05 10:19:23 +00:00
Neil Alexander
7ca89ef511
Update gomatrixserverlib 2020-10-20 17:10:37 +01:00
Neil Alexander
c7bf122a26
Update sytest lists 2020-10-19 15:38:42 +01:00
Neil Alexander
5d80ff11a0
Update sytest-whitelist 2020-10-19 15:17:28 +01:00
Neil Alexander
6e63df1d9a
KindOld (#1531)
* Add KindOld

* Don't process latest events/memberships for old events

* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events

* Signal to downstream components if an event has become a forward extremity

* Don't exclude from sync

* Soft-fail checks on KindNew

* Don't run the latest events updater at all for KindOld

* Don't make federation sender change after all

* Kind in federation sender join

* Don't send isForwardExtremity

* Fix syncapi

* Update comments

* Fix SendEventWithState

* Update sytest-whitelist

* Generate old output events

* Sync API consumes old room events

* Update comments
2020-10-19 14:59:13 +01:00
Neil Alexander
6f12b8f85c
Ignore typing events where sender doesn't match origin (#1523)
* Ignore typing notifications where the sender doesn't match the origin

* Update sytest-whitelist

* Fix formatting directives
2020-10-14 16:49:25 +01:00
Neil Alexander
e3a3908654
Update sytest-whitelist (closes #1315) 2020-10-14 16:07:15 +01:00
Kegsay
9096bfcee8
Validate m.room.create events in send_join responses (#1505)
* Validate m.room.create events in send_join responses

For sytest compliance, refs #1315 and #1317

Fixes #1317

* Linting
2020-10-10 00:21:15 +01:00
Pestdoktor
c4c8bfd027
reject invalid UTF-8 (#1472)
* reject invalid UTF-8

Signed-off-by: Jonas Fentker <jonas@fentker.eu>

* update sytest-whitelist

Signed-off-by: Jonas Fentker <jonas@fentker.eu>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-09 09:15:51 +01:00
Loïck Bonniot
f3e8ae01ef
Implement fully read markers (#1475)
See #653

Signed-off-by: Loïck Bonniot <git@lesterpig.com>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-09 09:15:35 +01:00
Neil Alexander
d821f9d3c9
Deep checking of forward extremities (#1491)
* Deep forward extremity calculation

* Use updater txn

* Update error

* Update error

* Create previous event references in StoreEvent

* Use latest events updater to row-lock prev events

* Fix unexpected fallthrough

* Fix deadlock

* Don't roll back

* Update comments in calculateLatest

* Don't include events that we can't find references for in the forward extremities

* Add another passing test
2020-10-07 14:05:33 +01:00
Neil Alexander
2e71d2708f
Resolve state after event against current room state when determining latest state changes (#1479)
* Resolve state after event against current room state when determining latest state changes

* Update sytest-whitelist

* Update sytest-whitelist, blacklist
2020-10-05 17:47:08 +01:00
Loïck Bonniot
4e8c484618
Implement account deactivation (#1455)
* Implement account deactivation

See #610
Signed-off-by: Loïck Bonniot <git@lesterpig.com>

* Rename 'is_active' to 'is_deactivated'

Signed-off-by: Loïck Bonniot <git@lesterpig.com>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-02 17:18:20 +01:00
Kegsay
279044cd90
Add history visibility guards (#1470)
* Add history visibility guards

Default to 'joined' visibility to avoid leaking events, until we get
around to implementing history visibility completely. Related #617

* Don't apply his vis checks on shared rooms

* Fix order of checks

* Linting and remove another misleading check

* Update whitelist
2020-10-02 17:08:13 +01:00
Neil Alexander
fb9a8f215b
Fix initial sync (#1465)
* Fix complete sync check

* Remove unnecessary 'since' copy

* Fix failing test

* Un-whitelist a couple of tests

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-10-02 12:50:58 +01:00
bn4t
da89f2bd1f
Finish implementing the capabilities endpoint (#1449)
Closes #1310

Signed-off-by: Benjamin Nater <me@bn4t.me>
2020-09-29 10:06:59 +01:00
Neil Alexander
6fbf89a166
Return the correct error codes for v6 invite JSON violations (#1440)
* Return the correct error codes for v6 invite JSON violations

* Update sytest-whitelist
2020-09-24 17:16:59 +01:00
Neil Alexander
3013ade84f
Reject make_join for empty rooms (#1439)
* Sanity-check room version on RS event input

* Update gomatrixserverlib

* Reject make_join when no room members are left

* Revert some changes from wrong branch

* Distinguish between room not existing and room being abandoned on this server

* nolint
2020-09-24 16:18:13 +01:00
Neil Alexander
a14b29b526
Initial notary support (#1436)
* Initial work on notary support

* Somewhat working (but not properly filtered) notary support, other tweaks

* Update gomatrixserverlib
2020-09-22 14:40:54 +01:00
Neil Alexander
a06c18bb56
Soft-fail (#1364)
* Initial work on soft-fail

* Fix state block retrieval

* Copy-pasta QueryLatestEventsAndState code

* Fix state lookup

* Clean up

* Fix up failing sytest

* Linting

* Update previous events SQLite insert query

* Update SQLite InsertPreviousEvent properly

* Hopefully fix the event references updates

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-21 14:55:46 +01:00
Kegsay
18231f25b4
Implement rejected events (#1426)
* WIP Event rejection

* Still send back errors for rejected events

Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.

* Implement rejected events

Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.

* Update test to match reality

* Modify InputRoomEvents to no longer return an error

Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.

* Remove redundant returns; linting

* Update blacklist
2020-09-16 13:00:52 +01:00
Matthew Hodgson
39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Neil Alexander
5076925c18
Password changes (#1397)
* User API support for password changes

* Password changes in client API

* Update sytest-whitelist

* Remove debug logging

* Default logout_devices to true

* Fix deleting devices by local part
2020-09-04 15:16:13 +01:00
Rohit Mohan
3f9b829bc5
Public room client API changes (#1368)
Signed-off-by: Rohit Mohan <rohitmohan96@gmail.com>
2020-09-01 10:26:34 +01:00
Kegsay
55498c8deb
Fix 'Invited user can reject invite over federation several times' (#1341) 2020-08-25 18:59:00 +01:00
Neil Alexander
a5a85c6a11
Make PerformJoin responsible for sending invite to RS input (#1277)
* Make PerformJoin send input membership event

* Invite input room events in separate goroutine

* Don't limit roomserver input events using request context

* Synchronous input room events

* Nope, that didn't work

* oops send state key to GetMembership

* Don't generate stripped state in client API more times than necessary, generate output events on receiving end of federated invite

* Commit membership updater changes

* Tweaks
2020-08-17 17:33:19 +01:00
Neil Alexander
6cb1a65809
Synchronous invites (#1273)
* Refactor invites to be synchronous

* Fix synchronous invites

* Fix client API return type for send invite error

* Linter

* Restore PerformError on rsAPI.PerformInvite

* Update sytest-whitelist

* Don't override PerformError with normal errors

* Fix error passing

* Un-whitelist a couple of tests

* Update sytest-whitelist

* Try to handle multiple invite rejections better

* nolint

* Update gomatrixserverlib

* Fix /v1/invite test

* Remove replace from go.mod
2020-08-17 11:40:49 +01:00
Kegsay
20c8f252a7
Make 'Device list doesn't change if remote server is down' pass (#1268)
- As a last resort, query the DB when exhausting all possible remote query
  endpoints, but keep the field in `failures` so clients can detect that this
  is stale data.
- Unblock `DeviceListUpdater.Update` on failures rather than timing out.
- Use a mutex when writing directly to `res`, not just for failures.
2020-08-13 16:43:27 +01:00
Kegsay
820c56c165
Fix more E2E sytests (#1265)
* WIP: Eagerly sync device lists on /user/keys/query requests

Also notify servers when a user's device display name changes. Few
caveats:
 - sytest `Device deletion propagates over federation` fails
 - `populateResponseWithDeviceKeysFromDatabase` is called from multiple
   goroutines and hence is unsafe.

* Handle deleted devices correctly over federation
2020-08-12 22:43:02 +01:00
Kegsay
d98ec12422
Add sync mechanism to block when updating device lists (#1264)
* Add sync mechanism to block when updating device lists

With a timeout, mainly for sytest to fix the test
"Server correctly handles incoming m.device_list_update"
which is flakey because it assumes that when `/send` 200 OKs
that the server has updated the device lists in prep for
`/keys/query` which is not always true when using workers.

* Fix UT

* Add new working test
2020-08-12 13:50:54 +01:00
Kegsay
b8b854d642
Bugfixes for 'If remote user leaves room we no longer receive device updates' (#1262)
* Bugfixes for 'If remote user leaves room we no longer receive device updates'

* Update whitelist and README
2020-08-12 10:50:52 +01:00
Neil Alexander
bcdf9577a3
Support for server ACLs (#1261)
* First pass at server ACLs (not efficient)

* Use transaction origin, update whitelist

* Fix federation API test

It's sufficient for us to return nothing in response to current state, so that the server ACL check returns no ACLs.

* More efficient server ACLs - hopefully

* Fix queries

* Fix queries

* Avoid panics by nil pointers

* Bug fixes

* Fix state event type

* Fix mutex

* Update logging

* Ignore port when matching servername

* Use read mutex

* Fix bugs

* Fix sync API test

* Comments

* Add tests, tweaks to behaviour

* Fix test output
2020-08-11 18:19:11 +01:00
Kegan Dougal
cce3678abe Update whitelist 2020-08-11 11:58:31 +01:00
Kegsay
befccd7d51
Reduce cooldown to make sure sytest doesn't give up (#1257)
* Reduce cooldown to make sure sytest doesn't give up

* More sytests pass weeeeeee
2020-08-11 10:44:59 +01:00
Kegsay
fdabba1851
bugfix: when a user's key changes, don't notify everyone on the server (#1253)
* bugfix: when a user's key changes, don't notify everyone on the server

Instead just notify the users you share a room with.

* Update whitelist
2020-08-10 12:38:33 +01:00
Kegsay
f371783da7
Finish inbound E2E device lists (#1243)
* Add tests for device list updates

* Add stale_device_lists table and use db before asking remote for device keys

* Fetch remote keys if all devices are requested

* Add display_name col to store remote device names

Few other tweaks to make `Server correctly handles incoming m.device_list_update`
pass.

* Fix sqlite otk bug

* Unbuffered channel to block /send causing sytest to not race anymore

* Linting and fix bug whereby we didn't send updated dl tokens to the client causing a tightloop on /sync sometimes

* No longer assert staleness as Update blocks on workers now

* Back out tweaks

* Bugfixes
2020-08-07 17:32:13 +01:00
Kegan Dougal
2197e54441 Update whitelist 2020-08-04 13:50:32 +01:00
Kegsay
fb56bbf0b7
Generate stream IDs for locally uploaded device keys (#1236)
* Breaking: add stream_id to keyserver_device_keys table

* Add tests for stream ID generation

* Fix whitelist
2020-08-03 17:07:06 +01:00
Kegsay
b5cb1d1534
Fix edge cases around device lists (#1234)
* Fix New users appear in /keys/changes

* Create blank device keys when logging in on a new device

* Add PerformDeviceUpdate and fix a few bugs

- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key

* Unbreak sqlite properly

* Use a real DB for currentstateserver integration tests

* Race fix
2020-07-31 14:40:45 +01:00
Kegsay
a7e67e65a8
Notify clients when devices are deleted (#1233)
* Recheck device lists when join/leave events come in

* Add PerformDeviceDeletion

* Notify clients when devices are deleted

* Unbreak things

* Remove debug logging
2020-07-30 18:00:56 +01:00
Kegsay
a2174d3294
Implement /keys/changes (#1232)
* Implement /keys/changes

And refactor QueryKeyChanges to accept a `to` offset.

* Unbreak tests

* Sort keys when serialising log tokens
2020-07-30 14:52:21 +01:00
Kegsay
9355fb5ac8
Hook up device list updates to the sync notifier (#1231)
* WIP hooking up key changes

* Fix import cycle, get tests passing and binary compiling

* Linting and update whitelist
2020-07-30 11:15:46 +01:00
Neil Alexander
acc8e80a51
User directory (#1225)
* User directory

* Fix syncapi unit test

* Make user directory only show remote users you know about from your joined rooms

* Update sytest-whitelist

* Review comments
2020-07-28 10:53:17 +01:00
Kegsay
c632867135
Modify /state/{eventType}/{stateKey} to return the event at the time the user left (#1222)
* Modify /state/{eventType}/{stateKey} to return the event at the time the user left

Or live, depending on their current state. Hopefully fixes some sytests!

* Linting

* Set HasBeenInRoom

* Fix cases for world-readable history visibility

* Fix bug in finding the requested state event

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-28 10:09:10 +01:00
Kegsay
c8d476a3cc
Return HTTP errors when trying to kick invalid users (#1221)
Room integrity was never compromised as GMSL does auth checks,
but we would incorrectly 200 OK the request instead of 403ing.
2020-07-27 09:20:09 +01:00
Kegsay
541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Kegsay
1d72ce8b7a
Implement claiming one-time keys locally (#1210)
* Add API shape for claiming keys

* Implement claiming one-time keys locally

Fairly boring, nothing too special going on.
2020-07-21 14:47:53 +01:00
Kegsay
f5e7e7513c
Implement /keys/query locally (#1204)
* Implement /keys/query locally

* Fix sqlite tests and close rows
2020-07-15 18:40:41 +01:00
Kegsay
9dd2ed7f65
Implement key uploads (#1202)
* Add storage layer for postgres/sqlite

* Return OTK counts when inserting new keys

* Hook up the key DB and make a test pass

* Convert postgres queries to be sqlite queries

* Blacklist test due to requiring rejected events

* Unbreak tests

* Update blacklist
2020-07-15 12:02:34 +01:00
Neil Alexander
72b3160776
Send-to-device messages over federation (#1198)
* Initial work to send send-to-device messages over federation

* Wire up send-to-device consumer, message formatting

* Generate random message ID

* Review comments, update sytest whitelist
2020-07-14 12:33:37 +01:00
Kegsay
abf26c12f1
Add User-Interactive Authentication (#1193)
* Add User-Interactive Authentication

And use it when deleting a device. With tests.

* Make remaining sytest pass

* Linting

* 403 not 401 on wrong user/pass
2020-07-10 00:39:44 +01:00
Kegsay
d9648b0615
Finish implementing redactions (#1189)
* Add a bit more logging to the fedsender

* bugfix: continue sending PDUs if ones are added whilst sending another PDU

Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.

* Break if no txn is sent

* WIP syncapi work

* More debugging

* Bump GMSL version to pull in working Event.Redact

* Remove logging

* Make redactions work on v3+

* Fix more tests
2020-07-08 17:45:39 +01:00
Kegsay
ea9df46c70
Implement local redaction (#1182)
* Create redaction events and apply checks (but do not send them)

* Send redactions to the roomserver

* Linting

* Slightly better wording
2020-07-03 17:24:51 +01:00
Kegsay
4c1e6597c0
Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver (#1174)
* Use content_value instead of membership

* Fix build

* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver

- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
  which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
    * Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
    * Apply pagination parameters to the slice.
    * Do a `QueryBulkStateContent` request to the currentstateserver to pull out
      required state event *content* (not entire events).
    * Aggregate and return the chunk.

Mostly but not fully implemented (DB queries on currentstateserver are missing)

* Fix pq query

* Make postgres work

* Make sqlite work

* Fix tests

* Unbreak pagination tests

* Linting
2020-07-02 15:41:18 +01:00
Kegsay
1ad7219e4b
Implement /sync limited and read timeline limit from stored filters (#1168)
* Move filter table to syncapi where it is used

* Implement /sync `limited` and read timeline limit from stored filters

We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.

* Update whitelist

* Default to the default timeline limit if it's unset, also strip the extra event correctly

* Update whitelist
2020-06-26 15:34:41 +01:00
Kegsay
164057a3be
Honour event size limits and return 413 (#1167) 2020-06-26 12:51:54 +01:00
Kegsay
4897beabee
Finish implementing retiring invites (#1166)
* Pass retired invites to the syncapi with the event ID of the invite

* Implement retire invite streaming

* Update whitelist
2020-06-26 11:07:52 +01:00
Kegan Dougal
c1d2382e6d Reject non-numeric ports (done in GMSL) 2020-06-25 18:05:41 +01:00
Kegsay
7a8282fccf
Handle invite v1 (#1165)
* Implement invite v1 for sytest mainly

* Bump gmsl version which falls back to invite v1 if v2 404s

* Update whitelist
2020-06-25 17:07:40 +01:00
Kegan Dougal
67f7a53f12 Add missing typing test 2020-06-25 15:06:46 +01:00
Kegsay
43cddfe00f
Return remote errors from FS.PerformJoin (#1164)
* Return remote errors from FS.PerformJoin

Follows the same pattern as PerformJoin on roomserver (no error return).

Also return the right format for incompatible room version errors.

Makes a bunch of tests pass!

* Handle network errors better when returning remote HTTP errors

* Linting

* Fix tests

* Update whitelist, pass network errors through in API=1 mode
2020-06-25 15:04:48 +01:00
Kegsay
e560619f76
Refactor SendMembership - make ban test pass (#1160)
* Refactor SendMembership - make ban test pass

* Only check invite auth events for local invites
2020-06-24 18:19:54 +01:00
Kegsay
a06d0921c9
Make same membership transitions 403, add sytests (#1161)
* Make same membership transitions 403, add sytests

* Update blacklist
2020-06-24 18:19:14 +01:00
Kegan Dougal
ebaaf65c54 This doesn't pass 2020-06-24 13:40:50 +01:00
Kegan Dougal
c72d23c8eb Update whitelist 2020-06-24 10:28:03 +01:00
Kegsay
0577bfca55
Pass join errors through internal API boundaries (#1157)
* Pass join errors through internal API boundaries

Required for certain invite sytests. We will need to think of a
better way of handling this going forwards.

* Include m.room.avatar in stripped state; handle trailing slashes when GETing state events

* Update whitelist

* Update whitelist
2020-06-24 09:59:59 +01:00
Kegsay
914f6cadce
Add /send restrictions and return correct error codes (#1156)
* Add /send restrictions and return correct error codes

- Max 50 PDUs / 100 EDUs
- Fail the transaction when PDUs contain bad JSON

* Update whitelist

* Unbreak test

* Linting
2020-06-23 13:15:15 +01:00
Kegsay
4220a374ca
Fix room checks for /state and /state_ids (#1155)
We would return a 403 first (as the server would not be allowed to
see this event) and only then return a 404 if the event is not in
the given room. We now invert those checks for /state and /state_ids
to make the tests pass.
2020-06-23 11:47:48 +01:00
Neil Alexander
7f26b0cd13
Bind build support, further Yggdrasil demo updates (#1152)
* Add gobind builds for Yggdrasil demo

* Massage client API a bit

* Fix build

* Fix gobind build

* Fix gobind client API setup

* Tweaks

* Tweaks

* Update sytest-whitelist, add comment

* Default to sending push rules on initial sync
2020-06-19 13:29:27 +01:00
Kegsay
84a7881468
Make account data sytests pass (#1147) 2020-06-17 17:01:03 +01:00
Kegan Dougal
9b408c19fb Missing sytests 2020-06-17 16:47:21 +01:00
Kegsay
8efeb8eb3b
Return the correct /joined_members response and allow ?format=event (#1146) 2020-06-17 16:21:42 +01:00
Kegsay
8e7c1eda05
Enable more sytests (#1145) 2020-06-17 15:40:37 +01:00
Neil Alexander
23bed196e6
Fallback parsing for Content-Disposition minefield (#1144) 2020-06-17 14:26:45 +01:00
Kegan Dougal
c4d9b37492 add test 2020-06-17 13:54:47 +01:00
Neil Alexander
5d5aa0a31d
Media filename handling improvements (#1140)
* Derive content ID from hash+filename but preserve dedupe, improve Content-Disposition handling and ASCII handling

* Linter fix

* Some more comments

* Update sytest-whitelist
2020-06-17 11:53:26 +01:00
Neil Alexander
04c99092a4
Update whitelist for sytest media fix (#1137)
* Update sytest-whitelist, are-we-synapse-yet.list

* Update gomatrixserverlib

* Update gomatrixserverlib

* Loop avoidance

* Return UTF-8 filenames

* Replace quotes only, instead of using strconv.Quote

* Update sytest-whitelist

* Update sytest-whitelist
2020-06-16 18:31:38 +01:00
Neil Alexander
45011579eb Update sytest-whitelist 2020-06-16 14:47:30 +01:00
Neil Alexander
8a6152ca70
Enable room version 6 (#1087)
* Return bad request on CS API /send if bad JSON

* Return some more M_BAD_JSON in the right places

* nolint because damnit gocyclo all I added was a type check for an error

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update sytest-whitelist

* Update gomatrixserverlib

* Update sytest-whitelist

* NotJSON -> BadJSON
2020-06-04 10:53:39 +01:00
Neil Alexander
a5d822004d
Send-to-device support (#1072)
* Groundwork for send-to-device messaging

* Update sample config

* Add unstable routing for now

* Send to device consumer in sync API

* Start the send-to-device consumer

* fix indentation in dendrite-config.yaml

* Create send-to-device database tables, other tweaks

* Add some logic for send-to-device messages, add them into sync stream

* Handle incoming send-to-device messages, count them with EDU stream pos

* Undo changes to test

* pq.Array

* Fix sync

* Logging

* Fix a couple of transaction things, fix client API

* Add send-to-device test, hopefully fix bugs

* Comments

* Refactor a bit

* Fix schema

* Fix queries

* Debug logging

* Fix storing and retrieving of send-to-device messages

* Try to avoid database locks

* Update sync position

* Use latest sync position

* Jiggle about sync a bit

* Fix tests

* Break out the retrieval from the update/delete behaviour

* Comments

* nolint on getResponseWithPDUsForCompleteSync

* Try to line up sync tokens again

* Implement wildcard

* Add all send-to-device tests to whitelist, what could possibly go wrong?

* Only care about wildcard when targeted locally

* Deduplicate transactions

* Handle tokens properly, return immediately if waiting send-to-device messages

* Fix sync

* Update sytest-whitelist

* Fix copyright notice (need to do more of this)

* Comments, copyrights

* Return errors from Do, fix dendritejs

* Review comments

* Comments

* Constructor for TransactionWriter

* defletions

* Update gomatrixserverlib, sytest-blacklist
2020-06-01 17:50:19 +01:00
Neil Alexander
6d50212f29
Miscellaneous fixes (#1060)
* Add missing routing for PerformDirectoryLookupRequest

* Tweak output

* Fix some bugs in devices

* Don't default to federated room joins in response to invite

* Update sytest-whitelist

* Update comments

* Return correct room ID from PerformJoin

* Fix appservice and EDU server API setup, update sytest-whitelist

* Update sytest-whitelist
2020-05-26 14:41:16 +01:00
Kegan Dougal
91902df95d Add tests due to https://github.com/matrix-org/sytest/pull/876 2020-05-21 11:51:15 +01:00
Kegan Dougal
7d91ef0616 This now passes on sytest/develop 2020-05-20 18:31:02 +01:00
Kegsay
6091bf044f
sytest: Add remaining backfill tests (#1052)
One failed because of `null` instead of `[]` in HTTP responses.

One failed because we hadn't implemented in-line filter limits!
2020-05-20 17:30:03 +01:00
Kegsay
1414922026
sytest: Make 'Inbound federation can backfill events' pass (#1051)
* sytest: Make 'Inbound federation can backfill events' pass

This breaks 'Outbound federation can backfill events' because now
we are returning the right number of events, which the previous
test was relying on.

Previously, /messages was backfilling the membership event, causing
the test to pass. Now we are no longer backfilling the membership
event due to the change in this commit, causing the test to fail.

The test should instead be returning the membership event locally
from synacpis database, but it doesn't do it fast enough, resulting
in a no-op /sync response with a next_batch=s0_0 which will never
pick up the local membership event when it rolls in. The test
does attempt to retry, but doesn't take the new next_batch=s1_0
resulting in it missing from the /messages response.

* Linting
2020-05-20 16:04:31 +01:00
Kegsay
260e69d138
Make "Outbound federation can backfill events" pass sytest (#1049)
- Use a backfill limit of 100 regardless of what was asked.
- Special case the create event for `StateIDsBeforeEvent`
- Trim to the limit in `syncapi`
2020-05-19 18:42:55 +01:00
Neil Alexander
106a335b7d Update sytest-whitelist 2020-05-12 17:49:04 +01:00
Kegsay
ce5dfbebf9
Implement /get_missing_events (#1022)
* WIP get_missing_events work

* More WIP get_missing_events work

* First working /get_missing_events implementation

Flakey currently due to racing between /sync and /send

* Final tweaks

* Remove log lines

* Linting

* go mod tidy

* Clamp min depth to 0

* sort events by depth because sytest makes me sad

Specifically I think it's
4172585c25/lib/SyTest/Federation/Client.pm (L265)
to blame here.
2020-05-12 16:24:28 +01:00
Neil Alexander
6e643860b1 Update sytest-whitelist 2020-05-11 14:01:15 +01:00
Neil Alexander
77fe509031
Enable v5 rooms (#992)
* Enable v5 roooms

* Update sytest-whitelist

* Enable v5 rooms by default, update gomatrixserverlib
2020-04-29 19:37:00 +01:00
Neil Alexander
3a858afca2
Loopback event from invite response (#982)
* Working invite v2 support

* Fix copyright notice

* Update gomatrixserverlib

* Add fallthrough

* Add missing continue

* Update sytest-whitelist, gomatrixserverlib

* Update gomatrixserverlib to test matrix-org/gomatrixserverlib#181

* Update gomatrixserverlib
2020-04-28 10:53:07 +01:00
Neil Alexander
9045b8e89f
Perspective key fetching, some federation room join fixes (#975)
* Update gomatrixserverlib

* Test matrix.org as perspective key server

* Base64 decode better

* Optional strict validity checking in gmsl

* Update gomatrixserverlib

* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)

* Update gomatrixserverlib, debug logging

* Remove debugging output

* More verbose debugging

* Print outliers

* Increase timeouts for testing, observe contexts before trying to join over more servers

* Don't block on roomserver (experimental)

* Don't block on roomserver

* Update gomatrixserverlib

* Update gomatrixserverlib

* Configurable perspective key fetchers

* Output number of configured keys for perspective

* Example perspective config included

* Undo debug stack trace

* Undo debug stack trace

* Restore original HTTP listener in monolith

* Fix lint

* Review comments

* Set default HTTP server timeout to 5 minutes now, block again when joining

* Don't use HTTP address for HTTPS whoops

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Actually add perspectives

* Actually add perspectives

* Update gomatrixserverlib
2020-04-20 17:42:34 +01:00
Kegsay
c62844234b
Add login sytests (#974) 2020-04-20 14:19:25 +01:00
Neil Alexander
895a72b6ee
Move /room/{roomID}/state endpoints into client API (#606) (#962)
* Move /room/{roomID}/state endpoints into client API (#606)

* Update sytest-whitelist

* Blacklist tests which rely on endpoints we don't implement
2020-04-14 18:36:08 +01:00