Libp2p GossipSub IDONTWANT Message Performance Impact

October 28, 2024

Umar Farooq

4 min read

This post provides quick insights into the IDONTWANT message performance and highlights minor tweaks that can further contribute to performance gains.

Overview

IDONTWANT messages are introduced to curtail redundant transmissions without compromising resilience. Cutting down on duplicates can potentially render two significant advantages:

Reducing bandwidth requirements
Reducing message dissemination time (latency)

For IDONTWANTs to be effective, they must be received and processed by the sender before the sender starts relaying the respective message.

Duplicates investigation reveals that the average time difference between the first message arrival and the first duplicate arrival is higher than the average round trip time in Ethereum's GossipSub network.

This allows for timely IDONTWANT reception and canceling of many duplicate transmissions, showing a potential for a significant drop in bandwidth utilization. On the other hand, lowering message dissemination time is only possible by minimizing queuing delays at busy peers.

Experiments

We conducted a series of experiments with different arrangements (changing heartbeat_interval and message size) to precisely identify the impact of IDONTWANT messages on bandwidth utilization and message dissemination time.

The experiments are performed on nim-libp2p using the shadow simulator. The peer bandwidth and link latency are uniformly set between 50-150 Mbps and 40-130 milliseconds in five stages.

In all experiments, ten messages are transmitted to the network, i.e., ten peers (publishers) are selected as the message transmitters. Every publisher transmits exactly one message, and inter-packet spacing (delay) is set to four seconds for each published message. For a fair assessment, we ensure that the publishers are uniformly selected from each bandwidth class.

At the start of each experiment, two additional messages are transmitted to increase the TCP $C_{wnd}$ . These messages are not included in latency computations.

The simulation details are presented in the table below.

Parameter	Value	Parameter	Value
Peers	2000	Publishers	10
Peer bandwidth	50-150 Mbps	Link latency	40-130 ms
Message size	1KB, 50KB, 500KB, 1MB	$D$	8
Heartbeat interval	700ms, 1000ms, 1500ms	$D_{low}$	6
FloodPublish	False	$D_{high}$	12
Gossip factor	0.05	Muxer	yamux

Findings

We use bandwidth utilization and latency as evaluation metrics. Bandwidth utilization represents total network-wide traffic (including gossip and other control messages). Latency refers to network-wide message dissemination time. The total number of IWANT requests and the number of message transmissions saved by IDONTWANT messages are also presented for detailed insights.

Experiments reveal that IDONTWANT messages yield a noticeable (up to 21%) drop in bandwidth utilization. A higher drop is seen with a higher heartbeat interval. Interestingly, a relatively low bandwidth reduction (12-20%) is seen for 1MB messages, compared to 500KB messages (18-21%).

This is because downloading a large message may consume several hundred milliseconds. During this time, a receiver will likely generate multiple IWANT requests for the same message, increasing bandwidth utilization.

Moreover, a peer can generate IDONTWANTs only after it has finished downloading the message. A longer download time will result in simultaneous reception of the same message from other mesh members.

These IWANT requests mainly overwhelm early message receivers, which can negatively impact message dissemination time on some occasions. Therefore, a similar message dissemination time is seen with and without IDONTWANT messages.

Similar results are seen on our large-scale deployment runs (running Waku nodes in Kubernetes).

Please feel free to join the discussion and leave feedback regarding this post in the VAC forum.

Overview​

Experiments​

Findings​

References​

Overview

Experiments

Findings

References