One of the hardest problems in operating RPC infrastructure at scale is data quality. Returning incorrect data can have severe downstream consequences, as blockchains underpin financial systems that require strong correctness guarantees. The challenge is not simply detecting incorrect data, but defining what “correct” even means in a decentralized system. An RPC node is just a participant in a consensus-based distributed network. Any response it returns is only meaningful relative to other nodes on the same chain. There is no absolute ground truth available to an individual RPC consumer in isolation. Verifying correctness requires cross-checking responses against multiple independent nodes, ideally distributed across providers and regions. For most developers, doing this reliably is impractical. You would need access to a large, diverse inventory of nodes, continuously sampling and comparing results at global scale. That is exactly the problem an RPC router or aggregator must solve. Over the past few months, while operating RouteMesh at scale, we repeatedly encountered issues such as:
- Not knowing which nodes were synced to the tip of the chain versus which were lagging
- Nodes advertised as archive nodes that did not actually serve archive data
- Uncertainty around whether a null transaction result was valid or required retries, increasing latency
- Maintaining confidence that our aggregation layer was returning correct data when routing across thousands of nodes from dozens of providers
Design Considerations
When designing Sentinel, we focused on two core questions: At the moment a user request is served, is the data correct relative to the network? Is the node serving that data sufficiently close to the tip of the chain? At the same time, we imposed several constraints: No additional latency on user-facing requests Cost efficiency, given large price variance across RPC providers Dynamic behavior that adapts to each chain’s block cadence (e.g. 250ms vs 10s block times) Robust handling of edge cases in consensus sampling Accurate comparison across differing response formats and encodingsThe Sentinel System
Sentinel is built on two complementary classes of checks: Replay Checks and Lag Checks. Replay Checks For every n-th request on a given chain–method combination (for example, eth_call on Ethereum), the original request parameters are asynchronously replayed in parallel against a sample of nodes from our inventory. The result originally served to the user is compared against the consensus of these sampled nodes. Each node is scored on how consistently it agrees with consensus. Nodes that deviate are given lower scores, and a node whose score falls below the routing threshold is removed from production routing until it recovers. This removal is enforced by the routing layer — Sentinel measures and scores so the router knows who to trust. Lag Checks Correctness is meaningless if data is stale. Sentinel also runs lag checks, which issueeth_blockNumber calls against a sample of nodes and measure how far each node’s reported block lags behind the observed chain tip.
Lag is normalized using the average block production time for each chain, allowing us to reason in terms of time rather than raw block counts.
- Nodes slightly behind the tip receive lower scores
- Nodes significantly out of sync are removed from production routing, as there is no acceptable reason for severe lag in production RPC traffic
Node Rehabilitation (“Staging Arena”)
Nodes that fail replay or lag checks are not permanently removed by default. Instead, they are moved into a staging arena, where they must pass consecutive validation rounds before being reintroduced to production traffic. This allows providers to recover from transient failures while preventing unreliable nodes from degrading user experience. The exact thresholds and rules are continuously refined as we gather more empirical data. Implementation Details and Edge Cases Some additional details for completeness:- Validation frequency adapts to traffic: high-traffic routes are checked more often, while low-traffic routes are checked less frequently
- If there are not enough nodes to establish meaningful consensus, no penalties are applied
- If two nodes return errors and one returns a valid result, the erroring nodes are considered out of consensus
- Transport-level failures (non-200 responses) do not penalize a node, as production routing will simply retry elsewhere