Features:
- Add `typing` extension.
- Add `receipts` extension.
- Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`.
- Add `SYNCV3_PPROF` support.
- Add `by_notification_level` sort order.
- Add `include_old_rooms` support.
- Add support for `$ME` and `$LAZY`.
- Add correct filtering when `*,*` is used as `required_state`.
- Add `num_live` to each room response to indicate how many timeline entries are live.
Bug fixes:
- Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm.
- Send back an `errcode` on HTTP errors (e.g expired sessions).
- Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :(
- Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic.
- Send HTTP 400 for invalid range requests.
- Don't publish no-op unread counts which just adds extra noise.
- Fix leaking DB connections which could eventually consume all available connections.
- Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever.
Improvements:
- Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s.
- Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s.
- Add `SlidingSyncUntil...` in tests to reduce races.
- Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s.
- Add trace task for initial syncs.
- Include the proxy version in UA strings.
- HTTP errors now wait 1s before returning to stop clients tight-looping on error.
- Pending event buffer is now 2000.
- Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8.
- Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
A fundamental assumption in the proxy has been that the order of events
in `timeline` in v2 will be the same all the time. There's some evidence
to suggest this isn't true in the wild. This commit refactors the proxy
to not assume this. It does this by:
- Not relying on the number of newly inserted rows and slicing the events
to figure out _which_ events are new. Now the INSERT has `RETURNING event_id, event_nid`
and we return a map from event ID to event NID to explicitly say which
events are new.
- Add more paranoia when calculating new state snapshots: if we see the
same (type, state key) tuple more than once in a snapshot we error out.
- Add regression tests which try to insert events out of order to trip the
proxy up.
Whilst this goes against the spec and is likely a Synapse bug, we still need to
work in the real world so handle it, log it, and add a regression test for it.
Using ANY allows us to give a single parameter containing many many
entries, which bypasses the postgres parameter limit of 65535. Without
this, large rooms like Matrix HQ which have current state >65535 events
will not be stored correctly.
Add torture test to the events table to assert that we can query >65535
events.
- Fix a few issues around pulling current state for a room.
- Add a function to extract the most recent message in a room, for
pre-populating the last timestamp.
It's easier to roll forward than roll backwards. Add 'replaces_nid' field
on the events table which tells which nid in the snapshot gets replaced, if any.
This stream uses the event NID, not its own position. Add a snapshot ID to events
in the events table in preparation for loading state snapshots from an event NID.
