A local comparison of bandwidth profiles showing significantly improved scalability in Waku v2 over Waku v1.
Background
The original plan for Waku v2 suggested theoretical improvements in resource usage over Waku v1, mainly as a result of the improved amplification factors provided by GossipSub. In its turn, Waku v1 proposed improvements over its predecessor, Whisper.
Given that Waku v2 is aimed at resource restricted environments, we are specifically interested in its scalability and resource usage characteristics. However, the theoretical performance improvements of Waku v2 over Waku v1, has never been properly benchmarked and tested.
Although we're working towards a full performance evaluation of Waku v2,
this would require significant planning and resources,
if it were to simulate "real world" conditions faithfully and measure bandwidth and resource usage across different network connections,
robustness against attacks/losses, message latencies, etc.
(There already exists a fairly comprehensive evaluation of GossipSub v1.1,
on which 11/WAKU2-RELAY is based.)
As a starting point, this post contains a limited and local comparison of the bandwidth profile (only) between Waku v1 and Waku v2. It reuses and adapts existing network simulations for Waku v1 and Waku v2 and compares bandwidth usage for similar message propagation scenarios.
Theoretical improvements in Waku v2
Messages are propagated in Waku v1 using flood routing. This means that every peer will forward every new incoming message to all its connected peers (except the one it received the message from). This necessarily leads to unnecessary duplication (termed amplification factor), wasting bandwidth and resources. What's more, we expect this effect to worsen the larger the network becomes, as each connection will receive a copy of each message, rather than a single copy per peer.
Message routing in Waku v2 follows the libp2p GossipSub protocol,
which lowers amplification factors by only sending full message contents to a subset of connected peers.
As a Waku v2 network grows, each peer will limit its number of full-message ("mesh") peerings -
libp2p suggests a maximum of 12 such connections per peer.
This allows much better scalability than a flood-routed network.
From time to time, a Waku v2 peer will send metadata about the messages it has seen to other peers ("gossip" peers).
See this explainer for a more detailed discussion.
Methodology
The results below contain only some scenarios that provide an interesting contrast between Waku v1 and Waku v2. For example, star network topologies do not show a substantial difference between Waku v1 and Waku v2. This is because each peer relies on a single connection to the central node for every message, which barely requires any routing: each connection receives a copy of every message for both Waku v1 and Waku v2. Hybrid topologies similarly show only a difference between Waku v1 and Waku v2 for network segments with mesh-like connections, where routing decisions need to be made.
For this reason, the following approach applies to all iterations:
- Simulations are run locally. This limits the size of possible scenarios due to local resource constraints, but is a way to quickly get an approximate comparison.
- Nodes are treated as a blackbox for which we only measure bandwidth, using an external bandwidth monitoring tool. In other words, we do not consider differences in the size of the envelope (for v1) or the message (for v2).
- Messages are published at a rate of 50 new messages per second to each network, except where explicitly stated otherwise.
- Each message propagated in the network carries 8 bytes of random payload, which is encrypted. The same symmetric key cryptographic algorithm (with the same keys) are used in both Waku v1 and v2.
- Traffic in each network is generated from 10 nodes (randomly-selected) and published in a round-robin fashion to 10 topics (content topics for Waku v2). In practice, we found no significant difference in average bandwidth usage when tweaking these two parameters (the number of traffic generating nodes and the number of topics).
- Peers are connected in a decentralized full mesh topology, i.e. each peer is connected to every other peer in the network. Waku v1 is expected to flood all messages across all existing connections. Waku v2 gossipsub will GRAFT some of these connections for full-message peerings, with the rest being gossip-only peerings.
- After running each iteration, we verify that messages propagated to all peers (comparing the number of published messages to the metrics logged by each peer).
For Waku v1, nodes are configured as "full" nodes (i.e. with full bloom filter),
while Waku v2 nodes are relay nodes, all subscribing and publishing to the same PubSub topic.
Network size comparison
Iteration 1: 10 nodes
Let's start with a small network of 10 nodes only and see how Waku v1 bandwidth usage compares to that of Waku v2. At this small scale we don't expect to see improved bandwidth usage in Waku v2 over Waku v1, since all connections, for both Waku v1 and Waku v2, will be full-message connections. The number of connections is low enough that Waku v2 nodes will likely GRAFT all connections to full-message peerings, essentially flooding every message on every connection in a similar fashion to Waku v1. If our expectations are confirmed, it helps validate our methodology, showing that it gives more or less equivalent results between Waku v1 and Waku v2 networks.
Sure enough, the figure shows that in this small-scale setup, Waku v1 actually has a lower per-peer bandwidth usage than Waku v2. One reason for this may be the larger overall proportion of control messages in a gossipsub-routed network such as Waku v2. These play a larger role when the total network traffic is comparatively low, as in this iteration. Also note that the average bandwidth remains more or less constant as long as the rate of published messages remains stable.
Iteration 2: 30 nodes
Now, let's run the same scenario for a larger network of highly-connected nodes, this time consisting of 30 nodes.
At this point, the Waku v2 nodes will start pruning some connections to limit the number of full-message peerings (to a maximum of 12),
while the Waku v1 nodes will continue flooding messages to all connected peers.
We therefore expect to see a somewhat improved bandwidth usage in Waku v2 over Waku v1.
Bandwidth usage in Waku v2 has increased only slightly from the smaller network of 10 nodes (hovering between 2000 and 3000 kbps). This is because there are only a few more full-message peerings than before. Compare this to the much higher increase in bandwidth usage for Waku v1, which now requires more than 4000 kbps on average.
Iteration 3: 50 nodes
For an even larger network of 50 highly connected nodes, the divergence between Waku v1 and Waku v2 is even larger. The following figure shows comparative average bandwidth usage for a throughput of 50 messages per second.
Average bandwidth usage (for the same message rate) has remained roughly the same for Waku v2 as it was for 30 nodes, indicating that the number of full-message peerings per node has not increased.
Iteration 4: 85 nodes
We already see a clear trend in the bandwidth comparisons above, so let's confirm by running the test once more for a network of 85 nodes. Due to local resource constraints, the effective throughput for Waku v1 falls to below 50 messages per second, so the v1 results below have been normalized and are therefore approximate. The local Waku v2 simulation maintains the message throughput rate without any problems.
Iteration 5: 150 nodes
Finally, we simulate message propagation in a network of 150 nodes. Due to local resource constraints, we run this simulation at a lower rate - 35 messages per second - and for a shorter amount of time.
Notice how the Waku v1 bandwidth usage is now more than 10 times worse than that of Waku v2. This is to be expected, as each Waku v1 node will try to flood each new message to 149 other peers, while the Waku v2 nodes limit their full-message peerings to no more than 12.
Discussion
Let's summarize average bandwidth growth against network growth for a constant message propagation rate. Since we are particularly interested in how Waku v1 compares to Waku v2 in terms of bandwidth usage, the results are normalised to the Waku v2 average bandwidth usage for each network size.
Extrapolation is a dangerous game, but it's safe to deduce that the divergence will only grow for even larger network topologies. Although control signalling contributes more towards overall bandwidth for Waku v2 networks, this effect becomes less noticeable for larger networks. For network segments with more than ~18 densely connected nodes, the advantage of using Waku v2 above Waku v1 becomes clear.
Network traffic comparison
The analysis above controls the average message rate while network size grows. In reality, however, active users (and therefore message rates) are likely to grow in conjunction with the network. This will have an effect on bandwidth for both Waku v1 and Waku v2, though not in equal measure. Consider the impact of an increasing rate of messages in a network of constant size:
The rate of increase in bandwidth for Waku v2 is slower than that for Waku v1 for a corresponding increase in message propagation rate. In fact, for a network of 30 densely-connected nodes, if the message propagation rate increases by 1 per second, Waku v1 requires an increased average bandwidth of almost 70kbps at each node. A similar traffic increase in Waku v2 requires on average 40kbps more bandwidth per peer, just over half that of Waku v1.
Conclusions
- Waku v2 scales significantly better than Waku v1 in terms of average bandwidth usage, especially for densely connected networks.
- E.g. for a network consisting of 150 or more densely connected nodes, Waku v2 provides more than 10x better average bandwidth usage rates than Waku v1.
- As the network continues to scale, both in absolute terms (number of nodes) and in network traffic (message rates) the disparity between Waku v2 and Waku v1 becomes even larger.
Future work
Now that we've confirmed that Waku v2's bandwidth improvements over its predecessor matches theory, we can proceed to a more in-depth characterisation of Waku v2's resource usage. Some questions that we want to answer include:
- What proportion of Waku v2's bandwidth usage is used to propagate payload versus bandwidth spent on control messaging to maintain the mesh?
- To what extent is message latency (time until a message is delivered to its destination) affected by network size and message rate?
- How reliable is message delivery in Waku v2 for different network sizes and message rates?
- What are the resource usage profiles of other Waku v2 protocols (e.g.
12/WAKU2-FILTERand19/WAKU2-LIGHTPUSH)?
Our aim is to get ever closer to a "real world" understanding of Waku v2's performance characteristics, identify and fix vulnerabilities and continually improve the efficiency of our suite of protocols.