Summary

A bug causing Ethereum mainnet nodes running the Reth execution client to stall was discovered. Later in the day the Reth team updated that the root cause was still being worked on and provided mitigating steps to recover notes. A root cause was identified later in the week and then an updated client release to resolve the issue was published on September 8. Post mortem public on September 11.

Recommended actions

As of Sept 8, the recommendation is to update to the latest version of Reth v1.7.0:

https://github.com/paradigmxyz/reth/releases/tag/v1.7.0

Initially, the Sep 2 tweet from Georgios and post from Alexey, recover with the following:

0. Shut down your node
1. reth stage drop --datadir DATADIR merkle
2. reth stage unwind --datadir DATADIR to-block 23272426
3. reth node --datadir DATADIR --debug.tip 0x2eb1fcafd864aafe21f2cb66310a869b8945231330f0da80c9e9b77861b56fca
4. Restart reth with your normal arguments when previous command finished syncing

If you were running with --datadir argument, provide it accordingly.

Latest updates

11 Sept 17:11 UTC

Post mortem from Reth team:

https://laced-king-de5.notion.site/Incident-Post-Mortem-Reth-Mainnet-State-Root-Mismatch-26732f2c348480dea8b8c2a8753696dc

08 Sep 16:00 UTC

Reth released a version that fixed the bug: