sliding-sync/README.md

83 lines
4.9 KiB
Markdown
Raw Normal View History

2022-04-13 11:35:14 +01:00
![GitHub branch checks state](https://img.shields.io/github/checks-status/matrix-org/sliding-sync/main)
2021-05-13 16:25:42 +01:00
# sync-v3
2021-05-14 16:49:33 +01:00
Run an experimental sync v3 server using an existing Matrix account. This is possible because, for the most part,
v3 sync is a strict subset of v2 sync.
2021-12-23 16:36:12 +00:00
An implementation of [MSC3575](https://github.com/matrix-org/matrix-doc/blob/kegan/sync-v3/proposals/3575-sync.md).
2021-08-05 17:28:38 +01:00
2022-06-08 18:32:19 +01:00
Proxy version to MSC API specification:
- Version 0.1.x: [2022/04/01](https://github.com/matrix-org/matrix-spec-proposals/blob/615e8f5a7bfe4da813bc2db661ed0bd00bccac20/proposals/3575-sync.md)
- First release
- Version 0.2.x: [2022/06/09](https://github.com/matrix-org/matrix-spec-proposals/blob/3b2b3d547b41e4aeebbde2ad6e89606dd684a86c/proposals/3575-sync.md)
2022-07-14 15:48:30 +01:00
- Reworked where lists and ops are situated in the response JSON. Added new filters like `room_name_like`. Added `slow_get_all_rooms`. Standardised on env vars for configuring the proxy. Persist access tokens, encrypted with `SYNCV3_SECRET`.
2022-08-05 13:10:48 +01:00
- Version 0.3.x: [2022/08/05](https://github.com/matrix-org/matrix-spec-proposals/blob/61decae837b5448b073fc5c718172f9b4d1e5e18/proposals/3575-sync.md)
- Spaces support, `txn_id` support.
2022-08-23 16:15:28 +01:00
- Version 0.4.x [2022/08/23](https://github.com/matrix-org/matrix-spec-proposals/blob/59c83a857b4cf3cf6aca593c34efb44709b10d17/proposals/3575-sync.md)
- Support for `tags` and `not_tags`.
2022-06-08 18:32:19 +01:00
2021-05-14 16:49:33 +01:00
## Usage
2021-11-03 10:24:28 +00:00
Requires Postgres 13+.
2021-07-21 10:30:04 +01:00
```bash
2021-05-14 16:49:33 +01:00
$ createdb syncv3
$ echo -n "$(openssl rand -hex 32)" > .secret # this MUST remain the same throughout the lifetime of the database created above.
2021-05-14 16:49:33 +01:00
$ go build ./cmd/syncv3
$ SYNCV3_SECRET=$(cat .secret) SYNCV3_SERVER="https://matrix-client.matrix.org" SYNCV3_DB="user=$(whoami) dbname=syncv3 sslmode=disable" SYNCV3_BINDADDR=0.0.0.0:8008 ./syncv3
2021-05-14 16:49:33 +01:00
```
2021-07-21 10:30:04 +01:00
2021-09-30 17:02:53 +01:00
Then visit http://localhost:8008/client/ (with trailing slash) and paste in the `access_token` for any account on `-server`.
2021-07-21 10:30:04 +01:00
When you hit the Sync button nothing will happen initially, but you should see:
2022-04-01 13:44:09 +01:00
```
INF Poller: v2 poll loop started ip=::1 since= user_id=@kegan:matrix.org
```
2022-04-01 13:44:09 +01:00
Wait for the first initial v2 sync to be processed (this can take minutes!) and then v3 APIs will be responsive.
### Prometheus
To enable metrics, pass `SYNCV3_PROM=:2112` to listen on that port and expose a scraping endpoint `GET /metrics`.
If you want to hook this up to a prometheus, you can just define `prometheus.yml`:
```yaml
global:
scrape_interval: 30s
scrape_timeout: 10s
scrape_configs:
- job_name: ss
static_configs:
- targets: ["host.docker.internal:2112"]
```
then run Prometheus in a docker container:
```bash
docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
```
to play with the data, use PromLens and point it at http://localhost:9090:
```bash
docker run -p 8080:8080 prom/promlens
```
Useful queries include:
- `rate(sliding_sync_poller_num_payloads{job="ss"}[5m])` : This shows the payload rate from pollers to API processes,
broken down by type. A stacked graph display is especially useful as the height then represents the total payload
rate. This can be used to highlight abnormal incoming data, such as excessive payload rates. It can also be used
to gauge how costly certain payload types are. In general, receipts and device data tend to be the most frequent
background noise. A full list of payload types are defined in the [pubsub directory](https://github.com/matrix-org/sliding-sync/blob/main/pubsub/v2.go).
- `sliding_sync_poller_num_pollers` : Absolute count of the number of /sync v2 pollers over time. Useful either as a single value,
or display over time. The higher this value, the more pressure is put on the upstream Homeserver.
- `sliding_sync_api_num_active_conns` : Absolute count of the number of active sliding sync connections. Useful either as a single value,
or display over time. The higher this value, the more pressure is put on the proxy API processes.
- `sum(increase(sliding_sync_poller_process_duration_secs_bucket[1m])) by (le)` : Useful heatmap to show how long /sync v2 responses take to process.
This can highlight database pressure as processing responses involves database writes and notifications over pubsub.
- `sum(increase(sliding_sync_api_process_duration_secs_bucket[1m])) by (le)` : Useful heatmap to show how long sliding sync responses take to calculate,
which excludes all long-polling requests. This can highlight slow sorting/database performance, as these requests should always be fast.
2021-10-21 15:57:50 +01:00
### How can I help?
At present, the best way to help would be to run a local v3 server pointed at a busy account and just leave it and a client running in the background. Look at it occasionally and submit any issues you notice. You can save console logs by right-clicking -> Save As.
Please run the server with `SYNCV3_DEBUG=1` set. This will force the server to panic when assertions fail rather than just log them.