A rich toolkit for collecting information about the execution of decentralized and distributed systems is the only way to react in time and notice failures
- Distributed design requires continuous collection of information from all nodes of the system, regardless of its integrity (ex. connection issues or desync)
- A single solution regardless of the final configuration (a local network of developers from several nodes, a distributed business network with different administrators, or even a public network)
- Isolation of public and private information without disclosure
Accordingly, with point #1, nodes must transmit their telemetry data to the configured endpoint. Aggregation and caching for future analysis is the responsibility of the end service.
Based on the nature of data node selects public or private endpoint as the destination point. All transmission is done in a push model from the node to the end service.
As shown in the picture, participation in the collection of telemetry data is optional and Company C avoids it. To respect this choice, the publicly available data should not contain any details about company C nodes.
To support various monitoring and analytics tools, the end service can run in three different ways:
1) As a simple proxy of incoming data to support tools with push-model. The service will wrap incoming data with node id and push forward.
2) As a queue. The service will cache all incoming data until a future request from the consumer(s).
3) As an aggregation layer. The service will store all incoming data and calculate summaries for future requests.
Raw messages from nodes
type: proposal | vote | commit | finalized
type: discovered | connected | disconnected
- Peer based:
- Avg bandwidth
- Avg ping
- State based:
- Finalized block
- Average block time
- Time since last block
- Block propagation time
- Avg number of transactions in blocks
- Number of pending transaction
- Number of active users (signatories)
- Minimize middleware count
- Split private data (used for administration purposes) and public data
Prometheus metric types https://prometheus.io/docs/concepts/metric_types/
Substrate telemetry types and open-source monitor: https://github.com/paritytech/substrate-telemetry/blob/master/backend/src/types.rs