The Brochain infrastructure

A visual overview

This is a graphical representation of what our chain infrastructure looks like. Not all chains we support get the full or exact services depicted here, but this is the core mainnet infrastructure design we run.

Infrastructure diagram. Inside the purple boxes are private networks. Outside are public.

Horcrux

Horcrux lets us do two things:

1. We can use remote signers, so that the private key of the signer is never present on the public servers. Even if a sentry node is compromised, there are no keys to be found or stolen.
2. We can scale the validator horizontally. The sentries and signers can be in different datacenters, even on different continents. If one node goes down, the others will still be able to sign the chain. We can lose entire datacenters with zero downtime.

Seed node

The seed nodes are important for the health of a PoS chain. Many operators still use persistent peers to ensure connectivity, but for a number of reasons this is bad practice compared to using seed nodes. Seed node only job is to ensure that there is an up-to-date list of peers for the chain.

Thanks to the seed node having a current list of reachable peers, we can use that to collect data for our network maps. The data isn't complete or guarranteed to be correct, but is as good of a basis as can be gathered from a single node in a chain.

Snapshots

We provide daily snapshots for most of our chains. Our snapshoting method is the following:

1. Spin up a completely fresh node
2. Use state sync to catch up
3. Stop the node
4. Prune the database to reduce the size
5. Compress and upload the database to the cloud
6. Clean up by removing th node again

This helps us produce snapshots that are as current and as small as possible. Normally our snapshots are in the order of 250 MB to 1.5 GB. Small snapshots makes deploying new nodes based on snapshots much faster.