Observability is key thing in every application development, but apart from logs it can be achived with other tools like telemetry.
Main question for now is what telemetry we want to collect? Another problem is what metric we can expose at prometheus level and which of them cannot be presented by prometheus types.
There are 2 types of telemetry, peer local and global info. Here is all info about them and prometheus types which can be used for them
- Peer name
- Peer location
- Iroha version (counter?)
- Peer uptime (counter?)
- Peer current role (gauge with integer as role?)
- Its networking speed (gauge)
- Its latency (gauge)
- Last available block on peer (gauge)
- Last finalized block by peer (gauge)
- Block time (gauge)
- Block propagation time (gauge)
- Number of pending transactions (gauge)
- Finalized block (gauge)
- Average block time (histogram/summary)
- Time since last block (gauge)
- Block propagation time (histogram/summary)
- Number of transactions in block (gauge)
- Info about gas
A rich toolkit for collecting information about the execution of decentralized and distributed systems is the only way to react in time and notice failures
- Distributed design requires continuous collection of information from all nodes of the system, regardless of its integrity (ex. connection issues or desync)
- A single solution regardless of the final configuration (a local network of developers from several nodes, a distributed business network with different administrators, or even a public network)
- Isolation of public and private information without disclosure
Accordingly, with point #1, nodes must transmit their telemetry data to the configured endpoint. Aggregation and caching for future analysis is the responsibility of the end service.
Based on the nature of data node selects public or private endpoint as the destination point. All transmission is done in a push model from the node to the end service.
As shown in the picture, participation in the collection of telemetry data is optional and Company C avoids it. To respect this choice, the publicly available data should not contain any details about company C nodes.
To support various monitoring and analytics tools, the end service can run in three different ways:
1) As a simple proxy of incoming data to support tools with push-model. The service will wrap incoming data with node id and push forward.
2) As a queue. The service will cache all incoming data until a future request from the consumer(s).
3) As an aggregation layer. The service will store all incoming data and calculate summaries for future requests.
Raw messages from nodes
type: proposal | vote | commit | finalized
type: discovered | connected | disconnected
- Peer based:
- Avg bandwidth
- Avg ping
- State based:
- Finalized block
- Average block time
- Time since last block
- Block propagation time
- Avg number of transactions in blocks
- Number of pending transaction
- Number of active users (signatories)
- Minimize middleware count
- Split private data (used for administration purposes) and public data
Prometheus metric types https://prometheus.io/docs/concepts/metric_types/
Substrate telemetry types and open-source monitor: https://github.com/paritytech/substrate-telemetry/blob/master/backend/src/types.rs