sliding-sync/state/receipt_table_test.go

391 lines
8.1 KiB
Go
Raw Normal View History

add extensions for typing and receipts; bugfixes and additional perf improvements Features: - Add `typing` extension. - Add `receipts` extension. - Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`. - Add `SYNCV3_PPROF` support. - Add `by_notification_level` sort order. - Add `include_old_rooms` support. - Add support for `$ME` and `$LAZY`. - Add correct filtering when `*,*` is used as `required_state`. - Add `num_live` to each room response to indicate how many timeline entries are live. Bug fixes: - Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm. - Send back an `errcode` on HTTP errors (e.g expired sessions). - Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :( - Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic. - Send HTTP 400 for invalid range requests. - Don't publish no-op unread counts which just adds extra noise. - Fix leaking DB connections which could eventually consume all available connections. - Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever. Improvements: - Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s. - Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s. - Add `SlidingSyncUntil...` in tests to reduce races. - Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s. - Add trace task for initial syncs. - Include the proxy version in UA strings. - HTTP errors now wait 1s before returning to stop clients tight-looping on error. - Pending event buffer is now 2000. - Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8. - Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
2022-12-14 18:53:55 +00:00
package state
import (
"encoding/json"
"reflect"
"sort"
"testing"
"github.com/matrix-org/sliding-sync/internal"
add extensions for typing and receipts; bugfixes and additional perf improvements Features: - Add `typing` extension. - Add `receipts` extension. - Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`. - Add `SYNCV3_PPROF` support. - Add `by_notification_level` sort order. - Add `include_old_rooms` support. - Add support for `$ME` and `$LAZY`. - Add correct filtering when `*,*` is used as `required_state`. - Add `num_live` to each room response to indicate how many timeline entries are live. Bug fixes: - Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm. - Send back an `errcode` on HTTP errors (e.g expired sessions). - Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :( - Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic. - Send HTTP 400 for invalid range requests. - Don't publish no-op unread counts which just adds extra noise. - Fix leaking DB connections which could eventually consume all available connections. - Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever. Improvements: - Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s. - Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s. - Add `SlidingSyncUntil...` in tests to reduce races. - Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s. - Add trace task for initial syncs. - Include the proxy version in UA strings. - HTTP errors now wait 1s before returning to stop clients tight-looping on error. - Pending event buffer is now 2000. - Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8. - Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
2022-12-14 18:53:55 +00:00
)
func sortReceipts(receipts []internal.Receipt) {
sort.Slice(receipts, func(i, j int) bool {
keyi := receipts[i].EventID + receipts[i].RoomID + receipts[i].UserID + receipts[i].ThreadID
keyj := receipts[j].EventID + receipts[j].RoomID + receipts[j].UserID + receipts[j].ThreadID
return keyi < keyj
})
}
func parsedReceiptsEqual(t *testing.T, got, want []internal.Receipt) {
t.Helper()
sortReceipts(got)
sortReceipts(want)
if len(got) != len(want) {
t.Fatalf("got %d, want %d, got: %+v want %+v", len(got), len(want), got, want)
}
for i := range want {
if !reflect.DeepEqual(got[i], want[i]) {
t.Errorf("i=%d got %+v want %+v", i, got[i], want[i])
}
}
}
2024-02-15 18:02:40 +00:00
func TestReceiptPacking(t *testing.T) {
testCases := []struct {
receipts []internal.Receipt
wantEDU receiptEDU
name string
}{
{
name: "single receipt",
receipts: []internal.Receipt{
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 42,
},
},
wantEDU: receiptEDU{
Type: "m.receipt",
Content: map[string]receiptContent{
"$bar": {
Read: map[string]receiptInfo{
"@baz": {
TS: 42,
},
},
},
},
},
},
{
name: "two distinct receipt",
receipts: []internal.Receipt{
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 42,
},
{
RoomID: "!foo2",
EventID: "$bar2",
UserID: "@baz2",
TS: 422,
},
},
wantEDU: receiptEDU{
Type: "m.receipt",
Content: map[string]receiptContent{
"$bar": {
Read: map[string]receiptInfo{
"@baz": {
TS: 42,
},
},
},
"$bar2": {
Read: map[string]receiptInfo{
"@baz2": {
TS: 422,
},
},
},
},
},
},
{
name: "MSC4102: unthreaded wins when threaded first",
receipts: []internal.Receipt{
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 42,
ThreadID: "thread_id",
},
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 420,
},
},
wantEDU: receiptEDU{
Type: "m.receipt",
Content: map[string]receiptContent{
"$bar": {
Read: map[string]receiptInfo{
"@baz": {
TS: 420,
},
},
},
},
},
},
{
name: "MSC4102: unthreaded wins when unthreaded first",
receipts: []internal.Receipt{
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 420,
},
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 42,
ThreadID: "thread_id",
},
},
wantEDU: receiptEDU{
Type: "m.receipt",
Content: map[string]receiptContent{
"$bar": {
Read: map[string]receiptInfo{
"@baz": {
TS: 420,
},
},
},
},
},
},
{
name: "MSC4102: unthreaded wins in private receipts when unthreaded first",
receipts: []internal.Receipt{
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 420,
IsPrivate: true,
},
{
RoomID: "!foo",
EventID: "$bar",
UserID: "@baz",
TS: 42,
ThreadID: "thread_id",
IsPrivate: true,
},
},
wantEDU: receiptEDU{
Type: "m.receipt",
Content: map[string]receiptContent{
"$bar": {
ReadPrivate: map[string]receiptInfo{
"@baz": {
TS: 420,
},
},
},
},
},
},
}
for _, tc := range testCases {
edu, err := PackReceiptsIntoEDU(tc.receipts)
if err != nil {
t.Fatalf("%s: PackReceiptsIntoEDU: %s", tc.name, err)
}
gotEDU := receiptEDU{
Type: "m.receipt",
Content: make(map[string]receiptContent),
}
if err := json.Unmarshal(edu, &gotEDU); err != nil {
t.Fatalf("%s: json.Unmarshal: %s", tc.name, err)
}
if !reflect.DeepEqual(gotEDU, tc.wantEDU) {
t.Errorf("%s: EDU mismatch, got %+v\n want %+v", tc.name, gotEDU, tc.wantEDU)
}
}
}
add extensions for typing and receipts; bugfixes and additional perf improvements Features: - Add `typing` extension. - Add `receipts` extension. - Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`. - Add `SYNCV3_PPROF` support. - Add `by_notification_level` sort order. - Add `include_old_rooms` support. - Add support for `$ME` and `$LAZY`. - Add correct filtering when `*,*` is used as `required_state`. - Add `num_live` to each room response to indicate how many timeline entries are live. Bug fixes: - Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm. - Send back an `errcode` on HTTP errors (e.g expired sessions). - Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :( - Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic. - Send HTTP 400 for invalid range requests. - Don't publish no-op unread counts which just adds extra noise. - Fix leaking DB connections which could eventually consume all available connections. - Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever. Improvements: - Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s. - Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s. - Add `SlidingSyncUntil...` in tests to reduce races. - Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s. - Add trace task for initial syncs. - Include the proxy version in UA strings. - HTTP errors now wait 1s before returning to stop clients tight-looping on error. - Pending event buffer is now 2000. - Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8. - Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
2022-12-14 18:53:55 +00:00
func TestReceiptTable(t *testing.T) {
db, close := connectToDB(t)
defer close()
add extensions for typing and receipts; bugfixes and additional perf improvements Features: - Add `typing` extension. - Add `receipts` extension. - Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`. - Add `SYNCV3_PPROF` support. - Add `by_notification_level` sort order. - Add `include_old_rooms` support. - Add support for `$ME` and `$LAZY`. - Add correct filtering when `*,*` is used as `required_state`. - Add `num_live` to each room response to indicate how many timeline entries are live. Bug fixes: - Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm. - Send back an `errcode` on HTTP errors (e.g expired sessions). - Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :( - Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic. - Send HTTP 400 for invalid range requests. - Don't publish no-op unread counts which just adds extra noise. - Fix leaking DB connections which could eventually consume all available connections. - Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever. Improvements: - Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s. - Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s. - Add `SlidingSyncUntil...` in tests to reduce races. - Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s. - Add trace task for initial syncs. - Include the proxy version in UA strings. - HTTP errors now wait 1s before returning to stop clients tight-looping on error. - Pending event buffer is now 2000. - Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8. - Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
2022-12-14 18:53:55 +00:00
roomA := "!A:ReceiptTable"
roomB := "!B:ReceiptTable"
edu := json.RawMessage(`{
"content": {
"$1435641916114394fHBLK:matrix.org": {
"m.read": {
"@rikj:jki.re": {
"ts": 1436451550453
}
},
"m.read.private": {
"@self:example.org": {
"ts": 1661384801651
}
}
}
},
"type": "m.receipt"
}`)
table := NewReceiptTable(db)
// inserting same receipts for different rooms should work - compound key should include the room ID
for _, roomID := range []string{roomA, roomB} {
newReceipts, err := table.Insert(roomID, edu)
if err != nil {
t.Fatalf("Insert: %s", err)
}
parsedReceiptsEqual(t, newReceipts, []internal.Receipt{
{
RoomID: roomID,
EventID: "$1435641916114394fHBLK:matrix.org",
UserID: "@rikj:jki.re",
TS: 1436451550453,
ThreadID: "",
},
{
RoomID: roomID,
EventID: "$1435641916114394fHBLK:matrix.org",
UserID: "@self:example.org",
TS: 1661384801651,
ThreadID: "",
IsPrivate: true,
},
})
}
// dupe receipts = no delta
newReceipts, err := table.Insert(roomA, edu)
assertNoError(t, err)
parsedReceiptsEqual(t, newReceipts, nil)
// selecting receipts -> ignores private receipt
got, err := table.SelectReceiptsForEvents(roomA, []string{"$1435641916114394fHBLK:matrix.org"})
assertNoError(t, err)
parsedReceiptsEqual(t, got, []internal.Receipt{
{
RoomID: roomA,
EventID: "$1435641916114394fHBLK:matrix.org",
UserID: "@rikj:jki.re",
TS: 1436451550453,
ThreadID: "",
},
})
// new receipt with old receipt -> 1 delta, also check thread_id is saved.
newReceipts, err = table.Insert(roomA, json.RawMessage(`{
"content": {
"$1435641916114394fHBLK:matrix.org": {
"m.read": {
"@rikj:jki.re": {
"ts": 1436451550453
},
"@alice:bar": {
"ts": 123456,
"thread_id": "yep"
}
}
}
},
"type": "m.receipt"
}`))
assertNoError(t, err)
parsedReceiptsEqual(t, newReceipts, []internal.Receipt{
{
RoomID: roomA,
EventID: "$1435641916114394fHBLK:matrix.org",
UserID: "@alice:bar",
TS: 123456,
ThreadID: "yep",
},
})
// updated receipt for user -> 1 delta
newReceipts, err = table.Insert(roomA, json.RawMessage(`{
"content": {
"$aaaaaaaa:matrix.org": {
"m.read": {
"@rikj:jki.re": {
"ts": 1436499990453
}
}
}
},
"type": "m.receipt"
}`))
assertNoError(t, err)
parsedReceiptsEqual(t, newReceipts, []internal.Receipt{
{
RoomID: roomA,
EventID: "$aaaaaaaa:matrix.org",
UserID: "@rikj:jki.re",
TS: 1436499990453,
ThreadID: "",
},
})
// selecting multiple receipts
table.Insert(roomA, json.RawMessage(`{
"content": {
"$aaaaaaaa:matrix.org": {
"m.read": {
"@bob:bar": {
"ts": 5555
},
"@self:example.org": {
"ts": 6666,
"thread_id": "yup"
}
}
}
},
"type": "m.receipt"
}`))
got, err = table.SelectReceiptsForEvents(roomA, []string{"$aaaaaaaa:matrix.org"})
assertNoError(t, err)
parsedReceiptsEqual(t, got, []internal.Receipt{
{
RoomID: roomA,
EventID: "$aaaaaaaa:matrix.org",
UserID: "@rikj:jki.re",
TS: 1436499990453,
ThreadID: "",
},
{
RoomID: roomA,
EventID: "$aaaaaaaa:matrix.org",
UserID: "@bob:bar",
TS: 5555,
ThreadID: "",
},
{
RoomID: roomA,
EventID: "$aaaaaaaa:matrix.org",
UserID: "@self:example.org",
TS: 6666,
ThreadID: "yup",
},
})
gotMap, err := table.SelectReceiptsForUser([]string{roomA}, "@self:example.org")
add extensions for typing and receipts; bugfixes and additional perf improvements Features: - Add `typing` extension. - Add `receipts` extension. - Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`. - Add `SYNCV3_PPROF` support. - Add `by_notification_level` sort order. - Add `include_old_rooms` support. - Add support for `$ME` and `$LAZY`. - Add correct filtering when `*,*` is used as `required_state`. - Add `num_live` to each room response to indicate how many timeline entries are live. Bug fixes: - Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm. - Send back an `errcode` on HTTP errors (e.g expired sessions). - Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :( - Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic. - Send HTTP 400 for invalid range requests. - Don't publish no-op unread counts which just adds extra noise. - Fix leaking DB connections which could eventually consume all available connections. - Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever. Improvements: - Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s. - Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s. - Add `SlidingSyncUntil...` in tests to reduce races. - Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s. - Add trace task for initial syncs. - Include the proxy version in UA strings. - HTTP errors now wait 1s before returning to stop clients tight-looping on error. - Pending event buffer is now 2000. - Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8. - Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
2022-12-14 18:53:55 +00:00
assertNoError(t, err)
parsedReceiptsEqual(t, gotMap[roomA], []internal.Receipt{
add extensions for typing and receipts; bugfixes and additional perf improvements Features: - Add `typing` extension. - Add `receipts` extension. - Add comprehensive prometheus `/metrics` activated via `SYNCV3_PROM`. - Add `SYNCV3_PPROF` support. - Add `by_notification_level` sort order. - Add `include_old_rooms` support. - Add support for `$ME` and `$LAZY`. - Add correct filtering when `*,*` is used as `required_state`. - Add `num_live` to each room response to indicate how many timeline entries are live. Bug fixes: - Use a stricter comparison function on ranges: fixes an issue whereby UTs fail on go1.19 due to change in sorting algorithm. - Send back an `errcode` on HTTP errors (e.g expired sessions). - Remove `unsigned.txn_id` on insertion into the DB. Otherwise other users would see other users txn IDs :( - Improve range delta algorithm: previously it didn't handle cases like `[0,20] -> [20,30]` and would panic. - Send HTTP 400 for invalid range requests. - Don't publish no-op unread counts which just adds extra noise. - Fix leaking DB connections which could eventually consume all available connections. - Ensure we always unblock WaitUntilInitialSync even on invalid access tokens. Other code relies on WaitUntilInitialSync() actually returning at _some_ point e.g on startup we have N workers which bound the number of concurrent pollers made at any one time, we need to not just hog a worker forever. Improvements: - Greatly improve startup times of sync3 handlers by improving `JoinedRoomsTracker`: a modest amount of data would take ~28s to create the handler, now it takes 4s. - Massively improve initial initial v3 sync times, by refactoring `JoinedRoomsTracker`, from ~47s to <1s. - Add `SlidingSyncUntil...` in tests to reduce races. - Tweak the API shape of JoinedUsersForRoom to reduce state block processing time for large rooms from 63s to 39s. - Add trace task for initial syncs. - Include the proxy version in UA strings. - HTTP errors now wait 1s before returning to stop clients tight-looping on error. - Pending event buffer is now 2000. - Index the room ID first to cull the most events when returning timeline entries. Speeds up `SelectLatestEventsBetween` by a factor of 8. - Remove cancelled `m.room_key_requests` from the to-device inbox. Cuts down the amount of events in the inbox by ~94% for very large (20k+) inboxes, ~50% for moderate sized (200 events) inboxes. Adds book-keeping to remember the unacked to-device position for each client.
2022-12-14 18:53:55 +00:00
{
RoomID: roomA,
EventID: "$1435641916114394fHBLK:matrix.org",
UserID: "@self:example.org",
TS: 1661384801651,
ThreadID: "",
IsPrivate: true,
},
{
RoomID: roomA,
EventID: "$aaaaaaaa:matrix.org",
UserID: "@self:example.org",
TS: 6666,
ThreadID: "yup",
},
})
}