Commit graph

24 commits

Author SHA1 Message Date
Till
9fe509b18d
Fix syncapi shared users query & device lists (#2614)
* Fix query issue, only add "changed" users if we actually share a room

* Avoid log spam if context is done

* Undo changes to filterSharedUsers

* Add logging again..

* Fix SQLite shared users query

* Change query to include invited users
2022-08-03 18:35:17 +02:00
Till
081f5e7226
Update database migrations, remove goose (#2264)
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Till
a7e92f8cb9
History visibility database changes (#2533)
* Add new history_visibility column

* Update SQL queries to include history_visibility

* Store the history visibilty calculated by the roomserver

* Update GMSL

* Update migrations

* Fix migration

* Update GMSL

* Fix `go.sum`

* Update GMSL to use sql.Scanner & sql.Valuer

* Re-order migration/table creation

* Update gomatrixserverlib

* Add history_visibility column to current_room_state

* Fix migrations

* Return error instead of Fatal log

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-18 14:46:15 +02:00
Neil Alexander
90bf01d8b1
Use sync API database in filterSharedUsers (#2572)
* Add function to the sync API storage package for filtering shared users

* Use the database instead of asking the RS API

* Fix unit tests

* Fix map handling in `filterSharedUsers`
2022-07-15 16:25:26 +01:00
Till
2a5b8e0306
Only load members of newly joined rooms (#2389)
* Only load members of newly joined rooms

* Comment that the query is prepared at runtime

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-28 18:53:28 +02:00
Till
69f2ff7c82
Correctly use provided filters (#2339)
* Apply filters correctly

* Fix issues; Use prepareWithFilters

* Update gmsl & tests

* go.mod..

* PR comments
2022-04-11 09:05:23 +02:00
S7evinK
d8facd6308
Fix SQL statement for PurgeRoomState (#2280) 2022-03-16 11:25:50 +01:00
Neil Alexander
507a8e6773
Don't range entire state for /sync (#2270)
* Don't range entire state for rooms the user has no reason to care about

* Remove unnecessary db field in postgresql
2022-03-11 12:48:45 +00:00
Neil Alexander
6e44450cc9
Don't re-request state events that are already in the timeline (#1739)
* Don't request state events if we already have the timeline events (Postgres only)

* Rename variable

* nocyclo

* Add SQLite

* Tweaks

* Revert query change

* Don't dedupe if asking for full state

* Update query
2021-02-04 12:20:37 +00:00
Neil Alexander
e1ace7e44a
Add event ID index on current state table (helps performance) (#1649) 2020-12-16 18:16:39 +00:00
Neil Alexander
bad81c028f
Don't recalculate event ID so often in sync (#1624)
* Don't bail so quickly in fetchMissingStateEvents

* Don't recalculate event IDs so often in sync API

* Add comments

* Fix comments

* Update to matrix-org/gomatrixserverlib@eb6a890
2020-12-09 18:07:17 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
Neil Alexander
965f068d1a
Handle state with input event as new events (#1415)
* SendEventWithState events as new

* Use cumulative state IDs for final event

* Error wrapping in calculateAndSetState

* Handle overwriting same event type and state key

* Hacky way to spot historical events

* Don't exclude from sync

* Don't generate output events when rewriting forward extremities

* Update output event check

* Historical output events

* Define output room event type

* Notify key changes on state

* Don't send our membership event twice

* Deduplicate state entries

* Tweaks

* Remove unnecessary nolint

* Fix current state upsert in sync API

* Send auth events as outliers, state events as rewrite

* Sync API don't consume state events

* Process events actually

* Improve outlier check

* Fix local room check

* Remove extra room check, it seems to break the whole damn world

* Fix federated join check

* Fix nil pointer exception

* Better comments on DeduplicateStateEntries

* Reflow forced federated joins

* Don't force federated join for possibly even local invites

* Comment SendEventWithState better

* Rewrite room state in sync API storage

* Add TODO

* Clean up all room data when receiving create event

* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event

* Rename to PurgeRoom

* Exclude backfilled messages from /sync

* Split out rewriting state from updating state from state res

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-15 11:17:46 +01:00
Neil Alexander
5aaf32bbed
Select distinct on room memberships in sync API (#1292) 2020-08-21 09:57:52 +01:00
Kegsay
ecd7accbad
Rehuffle where things are in the internal package (#1122)
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
2020-06-12 14:55:57 +01:00
Kegsay
24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Kegsay
1b34130a5b
Finish merging syncserver.go (#1033)
* Refactor all postgres tables; start work on sqlite

* wip sqlite merges; database is locked errors to investigate and failing tests

* Revert "wip sqlite merges; database is locked errors to investigate and failing tests"

This reverts commit 26cbfc5b75.

* convert current room state table

* port over sqlite topology table

* remove a few functions

* remove more functions

* Share more code

* factor out completesync and a bit more

* Remove remaining code
2020-05-14 16:11:37 +01:00
Neil Alexander
ad5849d222
HeaderedEvents in sync API (#922)
* Use HeaderedEvent in syncapi

* Update notifier test

* Fix persisting headered event

* Clean up unused API function

* Fix overshadowed err from linter

* Write headered JSON to invites table too

* Rename event_json to headered_event_json in syncapi database schemae

* Fix invites_table queries

* Update QueryRoomVersionCapabilitiesResponse comment

* Fix syncapi SQLite
2020-03-19 12:07:01 +00:00
Prateek Sachan
c019ad7086
Log errors from rows.Close (#920)
* Log errors from rows.Close

* fixed imports

* Added contextual messages

* fixed review changes
2020-03-18 10:17:18 +00:00
Thibaut CHARLES
6942ee1de0
Upgrade gomatrixserverlib dependency (#808)
* Upgrade gomatrixserverlib dependency

Signed-off-by: Thibaut CHARLES cromfr@gmail.com

* Added missing passing sytest

Signed-off-by: Thibaut CHARLES cromfr@gmail.com

* Fix login using identifier key

Not a full fix, it only really supports logging in with
the localpart of an mxid.

Signed-off-by: Serra Allgood <serra@allgood.dev>

* Replace deprecated prometheus.InstrumentHandler and unsafe time.Ticker

* goimports

* re-add temporarily missing deps?

* Refactor InstrumentHandlerCounter definition

* URL decode args

* Return server names  (#833)

* Remove unnecessary map->array processing

* Return server names in room federation directory query

* Knock off a TODO

* Fix /send_join and /send_leave (#821)

Fix the /send_join and /send_leave endpoints, so that they use the v2 endpoints as mandated by MSC1802. Also comment out the SyTest tests that are failing because of lack of support for the v1 endpoints.

* Refuse /send_join without m.room.create (#824)

Signed-off-by: Abhishek Kumar <abhishekkumar2718@gmail.com>

* AS should use the v1 endpoint, rather than r0 (#827)

* docker: Passthrough parameters to dendrite-monolith-server

* Fix copy & paste error (#812)

* Use gomatrixserverlib.Transaction instead of local type (#590) (#811)

* Move files back if linting fails (#810)

* replaced gometalinter description with golangci-lint (#837)

* Amend syncapi SQL queries to return missing columns (#840)

* This commit updates a couple of the syncapi SQL queries to return additional columns that are required/expected by rowsToStreamEvents in output_room_events_table.go.

It's not exactly clear to me yet what transaction_id and session_id do, but these being added n #367 results in state events breaking the /sync endpoint.

This is a temporary fix. We need to come up with a better solution.

* gomatrix to gomatrixserverlib on some weird line change

* Tweaks from @babolivier review comments

* Implement storage interfaces (#841)

* Implement interfaces for federationsender storage

* Implement interfaces for mediaapi storage

* Implement interfaces for publicroomsapi storage

* Implement interfaces for roomserver storage

* Implement interfaces for syncapi storage

* Implement interfaces for keydb storage

* common.PartitionStorer in publicroomsapi interface

* Update copyright notices

* make cmd directory path absolute in build.sh (#830)

* Resync testfile with current sytest pass/fail (#832)

* Resync testfile with current sytest pass/fail

* Add displayname test

* Fall back to postgres when database connection string parsing fails (#842)

* Fall back to postgres when parsing the database connection string for a URI schema fails

* Fix behaviour so that it really tries postgres when URL parsing fails and it complains about unknown schema if it succeeds

* Fix #842

* Fix #842 - again...

* Federation fixes (#845)

* Update gomatrixserverlib to p2p commit 92c0338, other tweaks

* Update gomatrixserverlib to p2p commit e5dcc65

* Rewrite getAuthChain

* Update gomatrixserverlib in go.mod/go.sum

* Correct a couple of package refs for updated gmsl/gomatrix

* Update gomatrixserverlib ref in go.mod/go.sum

* Update getAuthChain comments following @babolivier review

* Add a Sytest blacklist file (#849)

* Add more passing tests to the testfile, add test blacklist file (#848)

* CS API: Support for /messages, fixes for /sync (#847)

* Merge forward

* Tidy up a bit

* TODO: What to do with NextBatch here?

* Replace SyncPosition with PaginationToken throughout syncapi

* Fix PaginationTokens

* Fix lint errors

* Add a couple of missing functions into the syncapi external storage interface

* Some updates based on review comments from @babolivier

* Some updates based on review comments from @babolivier

* argh whitespacing

* Fix opentracing span

* Remove dead code

* Don't overshadow err (fix lint issue)

* Handle extremities after inserting event into topology

* Try insert event topology as ON CONFLICT DO NOTHING

* Prevent OOB error in addRoomDeltaToResponse

* Thwarted by gocyclo again

* Fix NewPaginationTokenFromString, define unit test for it

* Update pagination token test

* Update sytest-whitelist

* Hopefully fix some of the sync batch tokens

* Remove extraneous sync position func

* Revert to topology tokens in addRoomDeltaToResponse etc

* Fix typo

* Remove prevPDUPos as dead now that backwardTopologyPos is used instead

* Fix selectEventsWithEventIDsSQL

* Update sytest-blacklist

* Update sytest-whitelist

* Some fixes for #847 (#850)

* Fix a couple of cases where backfilling events we already had causes panics, hopefully fix ordering of events, update GMSL dependency for backfill URL fixes

* Remove commented out lines from output_room_events_table schema

* Wire up publicroomsapi for roomserver events (#851)

* Wire up publicroomsapi to roomserver events

* Remove parameter that was incorrectly brought over from p2p work

* nolint containsBackwardExtremity for now

* Store our own keys in the keydb (#853)

* Store our own keys in the keydb

The DirectKeyFetcher makes the assumption that you can always reach the key/v2/server endpoint of any server, including our own. We previously haven't bothered to store our own keys in the keydb so this would mean we end up making key requests to ourselves.

In the libp2p world as an example, self-dialling is not possible, therefore this would render it impossible to get our own keys.

This commit adds our own keys into the keydb so that we don't create unnecessarily (and maybe impossible) requests.

* Use golang.org/x/crypto/ed25519 instead of crypto/ed25519 for pre-Go 1.13

* More sync fixes (#854)

* Further sync tweaks

* Remove unnecessary blank line

* getBackwardTopologyPos always returns a usable value

* Revert order fixing

* Implement GET endpoints for account_data in clientapi (#861)

* Implement GET endpoints for account_data in clientapi

* Fix accountDB parameter

* Remove fmt.Println

* Add empty push rules into account data on account creation (#862)

* Handle kind=guest query parameter on /register (#860)

* Handle kind=guest query parameter on /register

* Reorganized imports

* Pass device_id as nil

* Added tests to systest-whitelist

* Update sytest-whitelist

* Blacklist 'displayname updates affect room member events' (#859)

* Room version abstractions (#865)

* Rough first pass at adding room version abstractions

* Define newer room versions

* Update room version metadata

* Fix roomserver/versions

* Try to fix whitespace in roomsSchema

* Implement room version capabilities in CS API (#866)

* Add wiring for querying the roomserver for the default room version

* Try to implement /capabilities for room versions

* Update copyright notices

* Update sytests, add /capabilities endpoint into CS API

* Update sytest-whitelist

* Add GetDefaultRoomVersion

* Fix cases where state package was shadowed

* Fix version formatting

* Update Dockerfile to Go 1.13.6

* oh yes types I remember

* And fix the default too

* Update documentation for Go 1.13 (#867)

* Pass cfg by reference around the codebase (#819)

* Pass cfg by reference around the codebase

* Merge branch 'master' into pass-cfg-by-ref

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Implement missing device management features (#835)

* Implement missing device management features

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add a little more documentation

* Undo changes

* Use non-anonymous struct to decode devices list

* Update sytest-whitelist

* Update sytest-whitelist

* Update sytest-blacklist

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Adding sslmode: disable to sytest server config (#813)

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Fix AppService bind addrs in test (#805)

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Always defer *sql.Rows.Close and consult with Err (#844)

* Always defer *sql.Rows.Close and consult with Err

database/sql.Rows.Next() makes sure to call Close only after exhausting
result rows which would NOT happen when returning early from a bad Scan.
Close being idempotent makes it a great candidate to get always deferred
regardless of what happens later on the result set.

This change also makes sure call Err() after exhausting Next() and
propagate non-nil results from it as the documentation advises.

Closes #764

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

* Override named result parameters in last returns

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

* Do the same over new changes that got merged

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Clean up

Co-authored-by: Serra Allgood <serra@allgood.dev>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
Co-authored-by: Abhishek Kumar <31231064+abhishekkumar2718@users.noreply.github.com>
Co-authored-by: Will Hunt <will@half-shot.uk>
Co-authored-by: S7evinK <tfaelligen@gmail.com>
Co-authored-by: Arshpreet <30545756+arsh-7@users.noreply.github.com>
Co-authored-by: Prateek Sachan <42961174+prateek2211@users.noreply.github.com>
Co-authored-by: Behouba Manassé <behouba@gmail.com>
Co-authored-by: aditsachde <23707194+aditsachde@users.noreply.github.com>
Co-authored-by: Kiril Vladimiroff <kiril@vladimiroff.org>
2020-02-11 15:46:51 +00:00
Kiril Vladimiroff
d5dbe546e4
Always defer *sql.Rows.Close and consult with Err (#844)
* Always defer *sql.Rows.Close and consult with Err

database/sql.Rows.Next() makes sure to call Close only after exhausting
result rows which would NOT happen when returning early from a bad Scan.
Close being idempotent makes it a great candidate to get always deferred
regardless of what happens later on the result set.

This change also makes sure call Err() after exhausting Next() and
propagate non-nil results from it as the documentation advises.

Closes #764

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

* Override named result parameters in last returns

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

* Do the same over new changes that got merged

Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11 14:12:21 +00:00
Neil Alexander
49f760a30b
CS API: Support for /messages, fixes for /sync (#847)
* Merge forward

* Tidy up a bit

* TODO: What to do with NextBatch here?

* Replace SyncPosition with PaginationToken throughout syncapi

* Fix PaginationTokens

* Fix lint errors

* Add a couple of missing functions into the syncapi external storage interface

* Some updates based on review comments from @babolivier

* Some updates based on review comments from @babolivier

* argh whitespacing

* Fix opentracing span

* Remove dead code

* Don't overshadow err (fix lint issue)

* Handle extremities after inserting event into topology

* Try insert event topology as ON CONFLICT DO NOTHING

* Prevent OOB error in addRoomDeltaToResponse

* Thwarted by gocyclo again

* Fix NewPaginationTokenFromString, define unit test for it

* Update pagination token test

* Update sytest-whitelist

* Hopefully fix some of the sync batch tokens

* Remove extraneous sync position func

* Revert to topology tokens in addRoomDeltaToResponse etc

* Fix typo

* Remove prevPDUPos as dead now that backwardTopologyPos is used instead

* Fix selectEventsWithEventIDsSQL

* Update sytest-blacklist

* Update sytest-whitelist
2020-01-23 17:51:10 +00:00
Neil Alexander
3bd023e74d
Federation fixes (#845)
* Update gomatrixserverlib to p2p commit 92c0338, other tweaks

* Update gomatrixserverlib to p2p commit e5dcc65

* Rewrite getAuthChain

* Update gomatrixserverlib in go.mod/go.sum

* Correct a couple of package refs for updated gmsl/gomatrix

* Update gomatrixserverlib ref in go.mod/go.sum

* Update getAuthChain comments following @babolivier review
2020-01-10 12:11:44 +00:00
Neil Alexander
c28577ea25
Implement storage interfaces (#841)
* Implement interfaces for federationsender storage

* Implement interfaces for mediaapi storage

* Implement interfaces for publicroomsapi storage

* Implement interfaces for roomserver storage

* Implement interfaces for syncapi storage

* Implement interfaces for keydb storage

* common.PartitionStorer in publicroomsapi interface

* Update copyright notices
2020-01-03 14:07:05 +00:00
Renamed from syncapi/storage/current_room_state_table.go (Browse further)