Previously we would _only_ do this when live streaming, and not
for initially loaded rooms. This just seems to be an oversight
when this feature was landed. We now defer lazy loading members
until after the response has been fully calculated.
We also need to add a distinction between "strict" matching
on `required_state` and not, as the tests only care about the
LL typing member, but we can also see LL _timeline_ members, which
then sometimes flake the test. Since we don't care about the LL
timeline member, just use a looser check.
Suppose I build `MatchRoomTimelineMostRecent(1, []Event{C})`.
this matcher was given a list `[A, B, C]` of three events.
The matcher should allow that, but before this patch it wouldn't.
I don't think this is the intention?
Features:
- Add `typing` extension.
- Add `receipts` extension.
- Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`.
- Add `SYNCV3_PPROF` support.
- Add `by_notification_level` sort order.
- Add `include_old_rooms` support.
- Add support for `$ME` and `$LAZY`.
- Add correct filtering when `*,*` is used as `required_state`.
- Add `num_live` to each room response to indicate how many timeline entries are live.
Bug fixes:
- Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm.
- Send back an `errcode` on HTTP errors (e.g expired sessions).
- Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :(
- Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic.
- Send HTTP 400 for invalid range requests.
- Don't publish no-op unread counts which just adds extra noise.
- Fix leaking DB connections which could eventually consume all available connections.
- Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever.
Improvements:
- Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s.
- Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s.
- Add `SlidingSyncUntil...` in tests to reduce races.
- Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s.
- Add trace task for initial syncs.
- Include the proxy version in UA strings.
- HTTP errors now wait 1s before returning to stop clients tight-looping on error.
- Pending event buffer is now 2000.
- Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8.
- Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
With regression tests. Comments explain the edge case, but basically
previously we were not calling `builder.AddRoomsToSubscription` with
the new room because we didn't know it was brand new, as it had a
valid swap operation. It only had a valid swap op because we "inserted"
(read: pretended) that the room has always been there at `len(list)`
so the from index was outside the known range. This works great most
of the time, but failed in the case where you use a large window size
e.g `[[0,20]]` for 3 rooms, then the 4th room is still "inside the range"
and hence is merely an update, not a brand new room, so we wouldn't add
the room to the builder.
Fixed by decoupling adding rooms to the builder and expecting swap/insert
ops; they aren't mutually exclusive.
