Commit graph

169 commits

Author SHA1 Message Date
Neil Alexander
89a16bdcd9
Version 0.4.0 2021-07-12 11:48:29 +01:00
kegsay
70e4bbda3b
Only log filename and not entire path (#1906) 2021-07-08 13:13:27 +01:00
Kegsay
f8d3a762c4
Add a per-room mutex to federationapi when processing transactions (#1810)
* Add a per-room mutex to federationapi when processing transactions

This has numerous benefits:
 - Prevents us doing lots of state resolutions in busy rooms. Previously, room forks would always result
   in a state resolution being performed immediately, without checking if we were already doing this in
   a different transaction. Now they will queue up, resulting in fewer calls to `/state_ids`, `/g_m_e`, etc.
 - Prevents memory usage from growing too large as a result and potentially OOMing.

And costs:
 - High traffic rooms will be slightly slower due to head-of-line blocking from other servers,
   though this has always been an issue as roomserver has a per-room mutex already.

* Fix unit tests

* Correct mutex lock ordering
2021-03-30 10:01:32 +01:00
Kegsay
af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Neil Alexander
6a35d9f1b1
Version 0.3.11 2021-03-02 11:46:35 +00:00
Neil Alexander
2fdc318f2c
Version 0.3.10 2021-02-17 16:05:04 +00:00
Neil Alexander
2386e0c7af
Gradually evict oldest cache entries (#1768)
* Gradually evict oldest cache entries

* Keep the remaining 10% of cached entries
2021-02-17 13:50:27 +00:00
Neil Alexander
397b158c05
Version 0.3.9 2021-02-04 12:39:43 +00:00
Neil Alexander
61090cbe35
Version 0.3.8 2021-01-28 13:37:50 +00:00
Neil Alexander
e14e9a1606
Version 0.3.7 2021-01-26 13:13:48 +00:00
Neil Alexander
9f443317bc
Graceful shutdowns (#1734)
* Initial graceful stop

* Fix dendritejs

* Use process context for outbound federation requests in destination queues

* Reduce logging

* Fix log level
2021-01-26 12:56:20 +00:00
Neil Alexander
67c89b3c5e
Version 0.3.6 2021-01-18 13:37:49 +00:00
Neil Alexander
ec1b017906
Version 0.3.5 2021-01-11 13:55:40 +00:00
Neil Alexander
4a0461378a
Version 0.3.4 2020-12-18 14:53:31 +00:00
Neil Alexander
b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander
851c02659a
Version 0.3.3 2020-12-09 14:13:02 +00:00
Neil Alexander
5d65a879a5
Federation sender event cache (#1614)
* Cache federation sender events

* Store in the correct cache

* Update federation event cache

* Fix Unset

* Give EDUs same caching treatment as PDUs

* Make federationsender_cache_size configurable

* Default caches configuration

* Fix unit tests

* Revert "Fix unit tests"

This reverts commit 24eb5d2252.

* Revert "Default caches configuration"

This reverts commit 464ecd1e64.

* Revert "Make federationsender_cache_size configurable"

This reverts commit 4631f53241.
2020-12-04 14:52:10 +00:00
Neil Alexander
1f3a498601
Version 0.3.2 2020-12-02 17:46:28 +00:00
Neil Alexander
b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander
bdf6490375
Add ability to disable federation (#1604)
* Allow disabling federation

* Don't start federation queues if disabled

* Fix for Go 1.13
2020-12-02 15:10:03 +00:00
Neil Alexander
9c52f82736
Version 0.3.1 2020-11-20 11:38:58 +00:00
Neil Alexander
c636be5070
Update last seen on sync requests (#1593)
* Update last seen on sync requests

* Fix MSC2836 unit tests

* Only update once per minute

* Remove debug logging

* Configurable option

* Simplify updateLastSeen/cleanLastSeen
2020-11-20 11:29:02 +00:00
Kegsay
6353b0b7e4
MSC2836: Threading - part one (#1589)
* Add mscs/hooks package, begin work for msc2836

* Flesh out hooks and add SQL schema

* Begin implementing core msc2836 logic

* Add test harness

* Linting

* Implement visibility checks; stub out APIs for tests

* Flesh out testing

* Flesh out walkThread a bit

* Persist the origin_server_ts as well

* Edges table instead of relationships

* Add nodes table for event metadata

* LEFT JOIN to extract origin_server_ts for children

* Add graph walking structs

* Implement walking algorithm

* Add more graph walking tests

* Add auto_join for local rooms

* Fix create table syntax on postgres

* Add relationship_room_id|servers to the unsigned section of events

* Persist the parent room_id/servers in edge metadata

Other events cannot assert the true room_id/servers for the
parent event, only make claims to them, hence why this is
edge metadata.

* guts to pass through room_id/servers

* Refactor msc2836 to allow handling from federation

* Add JoinedVia to PerformJoin responses

* Fix tests; review comments
2020-11-19 11:34:59 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
Neil Alexander
3856e547de
Version 0.3.0 2020-11-16 11:13:05 +00:00
S7evinK
cb41c992f7
Add -version flag (#1581)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-11-16 09:54:58 +00:00
S7evinK
bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a046.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f64.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
S7evinK
d5675feb96
Add possibilty to configure MaxMessageBytes for sarama (#1563)
* Add configuration for max_message_bytes for sarama

* Log all errors when sending multiple messages

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing config

* - Better comments on what MaxMessageBytes is used for
- Also sets the size the consumer may use
2020-10-27 14:11:37 +00:00
Neil Alexander
59428cdde3
Version 0.2.1 (#1558)
* Version 0.2.1

* Credit where credit due

* Update CHANGES.md

* Include PRs
2020-10-22 12:38:12 +01:00
Neil Alexander
24e38c4135
Internal HTTP APIs over H2C (#1541)
* H2C on internal HTTP because SCIENCE

* Update comments
2020-10-20 17:13:12 +01:00
Neil Alexander
6a16d46fba
Version 0.2.0 (#1551)
* v0.2.0-rc2

* Update CHANGES.md (also with some markdown lint suggestions)

* Update version number

* Update CHANGES.md

* Update CHANGES.md

* Update CHANGES.md

* Add known issue

* Update CHANGES.md
2020-10-20 16:46:53 +01:00
Kegsay
4a7fb9c045
Automatically upgrade databases on startup (#1529)
* Support auto-upgrading accounts DB

* Auto-upgrade device DB deltas

* Support up/downgrading from cmd/goose

* Linting

* Create tables then do migrations then prepare statements

To avoid failing due to some things not existing

* Linting
2020-10-15 18:09:41 +01:00
Neil Alexander
49abe359e6
Start Kafka connections for each component that needs them (#1527)
* Start Kafka connection for each component that needs one

* Fix roomserver unit tests

* Rename to naffkaInstance (@Kegsay review comment)

* Fix import cycle
2020-10-15 13:27:13 +01:00
Neil Alexander
286dd408ae
Better semver compliance for version reporting 2020-10-14 12:51:06 +01:00
Kegsay
d7ea814fa8
Wrap NewMembershipUpdater in a db writer (#1515) 2020-10-13 10:20:27 +01:00
Neil Alexander
fe5d1400bf
Update federation timeouts (#1504)
* Update to matrix-org/gomatrixserverlib#234

* Update gomatrixserverlib

* Update federation timeouts

* Fix dendritejs

* Increase /send context time in destination queue
2020-10-09 17:08:32 +01:00
Kegan Dougal
b12b7abcc0 v0.1.0
Squashed commit of the following:

commit 570528e5f19d76fe2fd7ea646246eea9650b1b60
Author: Kegan Dougal <kegan@matrix.org>
Date:   Thu Oct 8 14:52:10 2020 +0100

    v0.1.0

commit 02c020bced3be6d1a63e1001ab9c0ef6231ba087
Merge: db840f02 8035c50c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Oct 8 14:46:32 2020 +0100

    Merge branch 'master' into v0.1.0

commit db840f025b5d7b7c7e1ba905646571cb03dd7b22
Merge: adc19a3d 78f6e1a3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Oct 8 13:31:36 2020 +0100

    Merge branch 'master' into v0.1.0

commit adc19a3d5f0c9d5a85b3c1dd7d1772236fb8cc9b
Merge: c8fc6855 3e12f6e9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Oct 8 10:31:58 2020 +0100

    Merge branch 'master' into v0.1.0

commit c8fc68555c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Oct 7 18:41:04 2020 +0100

    Version 0.1.0rc3

commit 15bf385141
Merge: e7d9eea4 8bca7a83
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Oct 7 18:39:25 2020 +0100

    Merge branch 'master' into v0.1.0

commit e7d9eea4a0
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Oct 5 17:56:45 2020 +0100

    v0.1.0rc2

commit 3fa76370f2
Merge: f7cecdd9 52ddded7
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Oct 5 17:56:28 2020 +0100

    Merge branch 'master' into v0.1.0

commit f7cecdd9a8
Author: Kegan Dougal <kegan@matrix.org>
Date:   Fri Oct 2 17:25:59 2020 +0100

    Bump to 0.1.0rc1
2020-10-08 14:53:46 +01:00
Neil Alexander
8bca7a83a9
Update monolith -api behaviour (#1484)
* Update monolith -api mode listeners

* Fix check

* Fix another check

* Update HTTP API addr behaviour

* Redefine NoExternalListener

* NoListener
2020-10-07 16:59:22 +01:00
Kegsay
bf7e85848b
Rename serverkeyapi to signingkeyserver (#1492)
* Rename serverkeyapi to signingkeyserver

We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)

* Convert docker/scripts to use signing-key-server

* Rename missed bits
2020-10-07 16:23:18 +01:00
Neil Alexander
7048532bc4
Update version imprinting (#1462)
* Add version tag constant

* Update build imprinting
2020-10-01 16:09:11 +01:00
Neil Alexander
b471d4d77a
Set internal HTTP API timeout to 10 minutes (#1459) 2020-10-01 15:00:35 +01:00
Neil Alexander
b1d5360335
Update HTTP clients (#1457)
* Update gomatrixserverlib

* Use separate HTTP client for API calls, set User-Agent for outbound HTTP requests
2020-10-01 11:55:17 +01:00
Neil Alexander
f290e92a34
Remove TLS fingerprints, improve perspective unmarshal handling (#1452)
* Add prefer_direct_fetch option

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Don't deal in TLS fingerprints anymore
2020-09-29 17:08:18 +01:00
Neil Alexander
145db37d89
Allow configuring old verify keys (#1443)
* Allow configuring old verify keys

* Update sample config

* Update sample config

* Fix config population

* Key ID formatting validity of old_verify_keys

* Update comment
2020-09-25 10:58:53 +01:00
Sam
a6700331ce
Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)
* Replace all usages of txn.Stmt with sqlutil.TxStmt

Signed-off-by: Sam Day <me@samcday.com>

* Fix sign off link in PR template.

Signed-off-by: Sam Day <me@samcday.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-24 11:10:14 +01:00
Neil Alexander
de8b39065e
Enforce valid key IDs (#1437)
* Enforce valid key IDs

* Don't use key_id from dendrite.yaml as it is in matrix_key.pem
2020-09-23 11:07:57 +01:00
Kegsay
8dc9506210
Don't use more than 999 variables in SQLite querys. (#1425)
* Don't use more than 999 variables in SQLite querys.

Solve this problem in a more general and reusable way.
Also fix #1369
Add some unit tests.

Signed-off-by: Henrik Sölver <henrik.solver@gmail.com>

* Don't rely on testify for basic assertions

* Readability improvements and linting

Co-authored-by: Henrik Sölver <henrik.solver@gmail.com>
2020-09-14 16:39:38 +01:00
Matthew Hodgson
39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Kegsay
b4bd0cc0f5
Track goids when running with tracing enabled (#1413)
* Track goids when running with tracing enabled

* Linting
2020-09-08 17:30:05 +01:00
Kegsay
c992f4f1f4
Remove current state server (#1405)
* Remove current state server

Closes #1365 #1272 #1357

* Remove current state server from scripts/docs
2020-09-07 14:47:59 +01:00
Neil Alexander
74743ac8ae
Rate limiting (#1385)
* Initial rate limiting

* Move rate limiting to client API

* Update rate limits to hopefully be self-cleaning

* Use X-Forwarded-For, add comments

* Reduce rate limit threshold

* Tweak interval

* Configurable backoff

* Review comments, set cleanup interval to 30 seconds

* Allow generate-config to produce sane CI config

* Fix Complement dockerfile
2020-09-03 10:12:11 +01:00
Kegsay
9d9e854fe0
Add Queryer and Inputer and factor out more RSAPI stuff (#1382)
* Add Queryer and use embedded structs

* Add Inputer and factor out more RS API stuff

This neatly splits up the RS API based on the functionality it provides,
whilst providing a useful place for code sharing via the `helpers` package.
2020-09-02 17:13:15 +01:00
Neil Alexander
3b0774805c
Version imprint (#1383)
* Versions

* Update build.sh
2020-09-02 16:18:08 +01:00
Neil Alexander
89c772fb78
Report which component failed to consume (#1375) 2020-09-01 16:53:38 +01:00
Kegsay
9af2f5f1f2
Remove device DB from clientapi (#1352)
* Remove device DB from clientapi

* Remove device DB from startup configuration

It's all an impl detail now in user API
2020-08-27 18:53:40 +01:00
oliverpool
a4db43e096
Don't overwrite global err before return (#1293)
Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>
2020-08-25 13:11:52 +01:00
Neil Alexander
c8b873abc8
Roomserver NID caches (#1335)
* Initial work on roomserver NID caches

* Give caches to roomserver storage

* Populate caches

* Fix bugs

* Fix WASM build

* Don't hit cache twice in RoomNIDExcludingStubs

* Store reverse room ID-room NID mapping, consult caches when assigning NIDs
2020-08-25 12:32:29 +01:00
Neil Alexander
4b2db1dff5
Update Naffka (#1295)
* Update Naffka

* Fix Naffka setup

* Update Naffka

* Update Naffka
2020-08-24 13:49:20 +01:00
Neil Alexander
9d53351dc2
Component-wide TransactionWriters (#1290)
* Offset updates take place using TransactionWriter

* Refactor TransactionWriter in current state server

* Refactor TransactionWriter in federation sender

* Refactor TransactionWriter in key server

* Refactor TransactionWriter in media API

* Refactor TransactionWriter in server key API

* Refactor TransactionWriter in sync API

* Refactor TransactionWriter in user API

* Fix deadlocking Sync API tests

* Un-deadlock device database

* Fix appservice API

* Rename TransactionWriters to Writers

* Move writers up a layer in sync API

* Document sqlutil.Writer interface

* Add note to Writer documentation
2020-08-21 10:42:08 +01:00
Neil Alexander
b24747b305
Transaction writer changes, move roomserver writers (#1285)
* Updated TransactionWriters, moved locks in roomserver, various other tweaks

* Fix redaction deadlocks

* Fix lint issue

* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter

* Fix us not sending transactions through in latest events updater
2020-08-19 15:38:27 +01:00
Neil Alexander
e7d450adb8
Add SkipClean and UseEncodedPaths to intermediate routers (#1275) 2020-08-17 12:28:20 +01:00
Neil Alexander
c1f4faf308
Fix Docker builds, polylith components (#1269) 2020-08-13 18:27:19 +01:00
Neil Alexander
9677a95afc
API setup refactoring (#1266)
* Start HTTP endpoint refactoring

* Update SetupAndServeHTTP

* Fix builds

* Don't set up external listener if no address configured

* TLS HTTP setup

* Break apart client/federation/key/media muxes

* Tweaks

* Fix P2P demos

* Fix media API routing

* Review comments @Kegsay

* Update sample config

* Fix gobind build

* Fix External -> Public in federation API test
2020-08-13 12:16:37 +01:00
Neil Alexander
8b6ab272fb
New sample config (#1258)
* Config tweaks

* Tweaks to config

* Add new sample dendrite-config.yaml file

* Review comments @Kegsay
2020-08-11 13:21:26 +01:00
Neil Alexander
52eeeb1627
Prefix-defined Kafka topics (#1254)
* Prefix-defined Kafka topics

* Fix current state server test
2020-08-10 15:18:37 +01:00
Neil Alexander
4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Neil Alexander
30c2325eaf
Allow enforcing X.509 certificate validity (MSC1711) (#1249)
* Configurable X.509 certificate validation

* Fix dendritejs

* Update go.mod/go.sum for matrix-org/gomatrixserverlib#214

* Update sample config
2020-08-07 17:25:31 +01:00
Kegsay
9355fb5ac8
Hook up device list updates to the sync notifier (#1231)
* WIP hooking up key changes

* Fix import cycle, get tests passing and binary compiling

* Linting and update whitelist
2020-07-30 11:15:46 +01:00
Kegsay
61963a74ae
Add logic for determining when device lists have changed due to membership changes (#1220)
* Flesh out structure for handling device list updates for room membership changes

* First cut untested algorithm

* Add tests for determining changed/left device lists

* Linting

* Unbreak tests

* Sigh.. linting
2020-07-27 09:19:55 +01:00
Kegsay
98f2f09bb4
keyserver: produce key change events (#1218)
* Produce kafka events when keys are added

* Consume key changes in syncapi with TODO markers for handling them and catching up

* unbreak tests

* Linting
2020-07-23 16:41:36 +01:00
Kegsay
541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Neil Alexander
1e71fd645e
Persistent federation sender blacklist (#1214)
* Initial persistence of blacklists

* Move statistics folder

* Make MaxFederationRetries configurable

* Set lower failure thresholds for Yggdrasil demos

* Still write events into database for blacklisted hosts (they can be tidied up later)

* Review comments
2020-07-22 17:01:29 +01:00
Neil Alexander
11a39fe3b5
Deduplicate FS database, EDU persistence table (#1207)
* Deduplicate FS database, add some EDU persistence groundwork

* Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes

* Fix build due to bad keyserver import

* Working EDU persistence

* gocyclo, unsurprisingly

* Remove unused

* Update copyright notices
2020-07-20 16:55:20 +01:00
Kegsay
9dd2ed7f65
Implement key uploads (#1202)
* Add storage layer for postgres/sqlite

* Return OTK counts when inserting new keys

* Hook up the key DB and make a test pass

* Convert postgres queries to be sqlite queries

* Blacklist test due to requiring rejected events

* Unbreak tests

* Update blacklist
2020-07-15 12:02:34 +01:00
Kegsay
396219ef53
Add boilerplate for key server APIs (#1196)
Also add a README which outilnes how things will work.
2020-07-13 16:02:35 +01:00
Kegsay
d9648b0615
Finish implementing redactions (#1189)
* Add a bit more logging to the fedsender

* bugfix: continue sending PDUs if ones are added whilst sending another PDU

Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.

* Break if no txn is sent

* WIP syncapi work

* More debugging

* Bump GMSL version to pull in working Event.Redact

* Remove logging

* Make redactions work on v3+

* Fix more tests
2020-07-08 17:45:39 +01:00
Kegsay
99ea1f9b48
Emit redacted_event from the roomserver when redactions are validated (#1186)
* Emit redacted_event from the roomserver when redactions are validated

- Consume them in the currentstateserver and act accordingly.
- Add integration test for the roomserver to check that injecting
  `m.room.redaction` events result in `redacted_event` being emitted.

* Linting

* Ignore events that redact themselves
2020-07-07 12:51:55 +01:00
Kegsay
ea9df46c70
Implement local redaction (#1182)
* Create redaction events and apply checks (but do not send them)

* Send redactions to the roomserver

* Linting

* Slightly better wording
2020-07-03 17:24:51 +01:00
Kegsay
6c4b8185d7
Implement ExtraPublicRoomsProvider for p2p demos (#1180)
* Change API and rename to ExtraPublicRoomsProvider

* Make dendritejs work again

* Maybe make libp2p demo work again

* Linting
2020-07-03 12:59:00 +01:00
Kegsay
9c1f38621c
Remove publicroomsapi (#1176)
* Remove all of publicroomsapi

* Remove references to publicroomsapi

* Remove doc references to publicroomsapi
2020-07-02 17:11:33 +01:00
Kegsay
4c1e6597c0
Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver (#1174)
* Use content_value instead of membership

* Fix build

* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver

- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
  which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
    * Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
    * Apply pagination parameters to the slice.
    * Do a `QueryBulkStateContent` request to the currentstateserver to pull out
      required state event *content* (not entire events).
    * Aggregate and return the chunk.

Mostly but not fully implemented (DB queries on currentstateserver are missing)

* Fix pq query

* Make postgres work

* Make sqlite work

* Fix tests

* Unbreak pagination tests

* Linting
2020-07-02 15:41:18 +01:00
Kegsay
6f49758b90
Remove membership table from account DB (#1172)
* Remove membership table from account DB

And make code which needs that data use the currentstate server

* Unbreak tests; use a membership enum for space
2020-06-30 13:34:59 +01:00
Kegsay
ca5bbffd8d
Add a new component: currentstateserver (#1171)
* Add a new component: currentstateserver

- Add a skeleton for it, with databases and a single query method.
- Add integration tests for it.
- Add listen/address fields in the config (breaking as this will force people to specify this to validate)

Not currently hooked up to anything yet.

* Unbreak config tests

* Add current_state to sample config

* comments
2020-06-30 10:37:21 +01:00
Neil Alexander
72444e4a4f
User API polylith component (#1151)
* User API polylith component

* Add docker-pull.sh
2020-06-19 09:37:19 +01:00
Kegsay
e09d24e732
Move account/device DBs to userapi (#1141) 2020-06-17 12:05:56 +01:00
Kegsay
83391da0e0
Make syncapi use userapi (#1136)
* Make syncapi use userapi

* Unbreak things

* Fix tests

* Lint
2020-06-16 17:05:38 +01:00
Kegsay
1942928ee5
Make federationapi use userapi (#1135)
Removes dependencies on account DB, device DB and ASAPI.
2020-06-16 14:53:19 +01:00
Neil Alexander
fc0e74ae0f
Fix media API for demos and possibly Synapse (#1134)
* Fix media API for demos and possibly Synapse

* User API

* goimports
2020-06-16 14:29:11 +01:00
Kegsay
9c77022513
Make userapi responsible for checking access tokens (#1133)
* Make userapi responsible for checking access tokens

There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.

* Cleanup

* Review comments and test fix
2020-06-16 14:10:55 +01:00
Neil Alexander
57b7fa3db8
More server key updates, tests (#1129)
* More key tweaks

* Start testing stuff

* Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests

* Don't store our own keys in the database

* Don't store our own keys in the database

* Don't run tests for now

* Tweak caching behaviour, update tests

* Update comments, add fixes from forward-merge

* Debug logging

* Debug logging

* Perform final comparison against original set of requests

* oops

* Fetcher timeouts

* Fetcher timeouts

* missing func

* Tweaks

* Update gomatrixserverlib

* Fix Federation API test

* Break up FetchKeys

* Add comments to caching

* Add URL check in test

* Partially revert "Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests"

This reverts commit d7eb54c5b3.

* Fix federation API test

* Fix internal cache stuff again

* Fix server key API test

* Update comments

* Update comments from review

* Fix lint
2020-06-16 13:11:20 +01:00
Kegsay
7c36fb78a7
Fix rooms v3 url paths for good - with tests (#1130)
* Fix rooms v3 url paths for good - with tests

- Add a test rig around `federationapi` to test routing.
- Use `JSONVerifier` over `KeyRing` so we can stub things out more easily.
- Add `test.NopJSONVerifier` which verifies nothing.
- Add `base.BaseMux` which is the original `mux.Router` used to spawn public/internal routers.
- Listen on `base.BaseMux` and not the default serve mux as it cleans paths which we don't want.
- Factor out `ListenAndServe` to `test.ListenAndServe` and add flag for listening on TLS.

* Fix comments

* Linting
2020-06-15 16:57:59 +01:00
Kegsay
0ba1245a46
Current wiring (#1125)
* Current wiring

* Add ServerKeyAPI lines
2020-06-15 10:13:57 +01:00
Kegsay
ecd7accbad
Rehuffle where things are in the internal package (#1122)
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
2020-06-12 14:55:57 +01:00
Neil Alexander
079d8fe8fb
More key tweaks (#1116) 2020-06-12 11:07:26 +01:00
Kegsay
b7187a9a35
Remove clientapi producers which aren't actually producers (#1111)
* Remove clientapi producers which aren't actually producers

They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.

* Remove TODO

* unbreak ygg
2020-06-10 12:17:54 +01:00
Kegsay
98cb0705ea
Remove unused UserUpdates producer (#1109) 2020-06-10 10:34:22 +01:00
Kegsay
85ac8a3f5b
Factor out how monolith routes get added (#1107)
Previously we had 3 monoliths:
 - dendrite-monolith-server
 - dendrite-demo-libp2p
 - dendritejs

which all had their own of setting up public routes. Factor this
out into a new `setup.Monolith` struct which gets all dependencies
set as fields. This is different to `basecomponent.Base` which
doesn't provide any way to set configured deps (e.g public rooms db)

Part of a larger process to clean up how we initialise Dendrite.
2020-06-09 12:07:33 +01:00
Neil Alexander
e7b19d2c70
More flexible caching (#1101) 2020-06-05 16:42:01 +01:00
Kegsay
29a20d1da7
General cleanup when making components (#1098)
* Remove ParseMonolith/LoadMonolith

* cleanup which components need to be made
2020-06-05 09:28:15 +01:00
Kegsay
2bd12f635c
Convert serverkeys to inthttp (#1097) 2020-06-04 16:26:35 +01:00