Final update

Hotfix released! Update now:

https://github.com/hyperledger/besu/releases/tag/23.10.3-hotfix

Post-mortem:

Jan 6th 2024 Mainnet Halting Event - Hyperledger Besu - Hyperledger Foundation

Summary

Besu validators suddenly went offline at block 18,947,893. Besu is aware of the issue and investigating to be able to deploy a fix ASAP

Recommended actions

• Use the Rescue Node to get back online while waiting for Besu to deploy a fix
  • Rescue Node docs here
  • Rescue Node discussion here
• Or switch to another execution layer client (Nethermind recommended)

Ethdocker users

For users of Eth Docker, here are the commands to implement each of the Besu workarounds. After trying a workaround, check logs with ./ethd logs -f execution. If you see that a backward sync started, then it worked and Besu is repairing

1. Updating Besu: ./ethd update then ./ethd up. If this did not work, try step 2.
2. Resyncing CL: ./ethd resync-consensus. If this did not work, try step 3.
3. Stop CL, set head, resync CL: nano .env, set EL_EXTRAS=--rpc-http-api WEB3,ETH,NET,DEBUG, save and close. ./ethd up, then ./ethd stop consensus. Now do the setHead: docker compose exec -it -u root execution bash then

apt update && apt install -y curl
curl -X POST --data '{"jsonrpc":"2.0","method":"debug_setHead","params":["0x1211F34"],"id":1}' <http://127.0.0.1:8545>

And resync CL once more: ./ethd resync-consensus

If this did not work, try step 4.

1. Force world state resync - takes 4+ hours: docker compose exec -it -u root execution bash then