If we returned multiple distinct ranges, we always assumed that
the history visibility was "joined", so we would never return
events in the invite/shared state. This would be fine if the client
had a way to fetch those events sent before they were joined, but
they did not have a way as the prev_batch token would not be set
correctly. We now only return a single range of events and the prev
batch for /that/ range only, and defer to the upstream HS for
history visibility calculations.
Add end-to-end test to assert this new behaviour works.
Specifically this is targetting invite rejections, where the leave
event is inside the leave block of the sync v2 response.
Previously, we would make a snapshot with this leave event. If the
proxy wasn't in this room, it would mean the room state would just
be the leave event, which is wrong. If the proxy was in the room,
then state would correctly be rolled forward.
When the proxy is run with large DBs (10m+ events), the
startup queries are very slow (around 30min to load the initial snapshot.
After much EXPLAIN ANALYZEing, the cause is due to Postgres' query planner
not making good decisions when the the tables are that large. Specifically,
the startup queries need to pull all joined members in all rooms, which
ends up being nearly 50% of the entire events table of 10m rows. When this
query is embedded in a subselect, the query planner assumes that the subselect
will return only a few rows, and decides to pull those rows via an index. In this
particular case, indexes are the wrong choice, as there are SO MANY rows a Seq Scan
is often more appropriate. By using an index (which is a btree), this ends up doing
log(n) operations _per row_ or `O(0.5 * n * log(n))` assuming we pull 50% of the
table of n rows. As n increases, this is increasingly the wrong call over a basic
O(n) seq scan. When n=10m, a seq scan has a cost of 10m, but using indexes has a
cost of 16.6m. By dumping the result of the subselect to a temporary table, this
allows the query planner to notice that using an index is the wrong thing to do,
resulting in better performance. On large DBs, this decreases the startup time
from 30m to ~5m.
The previous query would:
- Map room IDs to snapshot NIDs
- UNNEST(events) on all those state snapshots
- Compare if the type/state_key match the filter
This was very slow under the following circumstances:
- The rooms have lots of members (e.g Matrix HQ)
- The required_state has no filter on m.room.member
This is what Element X does.
To improve this, we now have _two_ columns per state snapshot:
- membership_events : only the m.room.member events
- events : everything else
Now if a query comes in which doesn't need m.room.member events, we just need
to look in the everything-else bucket of events which is significantly smaller.
This reduces these queries to about 50ms, from 500ms.
We already extracted the joined users in all rooms, but then
this function would do another query to pull out the join counts.
This query was particularly inefficient, clocking in at 4s (!) on
my test server. Removed it entirely and instead do len(joinedUsers)
by calling AllJoinedMembers first.
Features:
- Add `typing` extension.
- Add `receipts` extension.
- Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`.
- Add `SYNCV3_PPROF` support.
- Add `by_notification_level` sort order.
- Add `include_old_rooms` support.
- Add support for `$ME` and `$LAZY`.
- Add correct filtering when `*,*` is used as `required_state`.
- Add `num_live` to each room response to indicate how many timeline entries are live.
Bug fixes:
- Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm.
- Send back an `errcode` on HTTP errors (e.g expired sessions).
- Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :(
- Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic.
- Send HTTP 400 for invalid range requests.
- Don't publish no-op unread counts which just adds extra noise.
- Fix leaking DB connections which could eventually consume all available connections.
- Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever.
Improvements:
- Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s.
- Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s.
- Add `SlidingSyncUntil...` in tests to reduce races.
- Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s.
- Add trace task for initial syncs.
- Include the proxy version in UA strings.
- HTTP errors now wait 1s before returning to stop clients tight-looping on error.
- Pending event buffer is now 2000.
- Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8.
- Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
A fundamental assumption in the proxy has been that the order of events
in `timeline` in v2 will be the same all the time. There's some evidence
to suggest this isn't true in the wild. This commit refactors the proxy
to not assume this. It does this by:
- Not relying on the number of newly inserted rows and slicing the events
to figure out _which_ events are new. Now the INSERT has `RETURNING event_id, event_nid`
and we return a map from event ID to event NID to explicitly say which
events are new.
- Add more paranoia when calculating new state snapshots: if we see the
same (type, state key) tuple more than once in a snapshot we error out.
- Add regression tests which try to insert events out of order to trip the
proxy up.
- Replace `PrevBatch string` in user room data with `PrevBatches lru.Cache`.
This allows us to persist prev batch tokens in-memory rather than doing
N sequential DB lookups which would take ~4s for ~150 rooms on the postgres
instance running the database. The tokens are keyed off a tuple of the
event ID being searched and the latest event in the room, to allow prev
batches to be assigned when new sync v2 responses arrive.
- Thread through context to complex storage functions for profiling
Previously, we would only optimise pulling out event types i.e. if you want
state events with types A and B we only pull out all current state with event
type A or B. This falls down when the client wants their own member event,
as m.room.member is the bulk of the current state. This commit optimises the
SQL queries to also take into account the state key asked for, whilst still
supporting wildcards '*' when they are requested.
Fixes https://github.com/matrix-org/sliding-sync/issues/23
- Added InvitesTable
- Allow invites to be sorted/searched the same as any other room by
implementing RoomMetadata for the invite (though this is best effort
as we don't have heroes)
RoomMetadata stores the current invite/join count, heroes for the
room, most recent timestamp, name event content, canonical alias, etc
This information is consistent across all users so can be globally
cached for future use. Make ConnState call CalculateRoomName with
RoomMetadata to run the name algorithm.
This is *almost* complete but as there are no Heroes yet in the
metadata, things don't quite render correctly yet.
