Ethereum’s ‘unannounced hard fork’ was trying to avoid the disruption it caused. For Infura, the ConsenSys-run servers that synchronise the Ethereum network with the majority of decentralised finance (DeFi) applications, this was initially an obvious issue.
Infura went down around 8:00 UTC Wednesday, and with it, some of Ethereum’s most popular applications like Metamask, MakerDAO, Uniswap, Compound and MyCrypto, among others. Shortly after, Binance halted Ethereum trading after noticing conflicting transactions on its Ethereum node. As other exchanges suspended trading as well, the real issue became clear: A bug in the Go Ethereum (Geth) client, whose code underpins 80% of Ethereum’s applications, had split the Ethereum blockchain in two.
The two contrasting transaction histories indicated that users of Etheruem communicated briefly with various versions of the Ethereum blockchain. This put user funds at risk by knocking out most of the DeFi applications from Ethereum for a few hours, rather than causing delays.
Infura has resolved the problem by upgrading their nodes, as have other service providers that were affected by the snafu. An older version of Geth was run by these stakeholders, which included a bug that was silently patched by Ethereum developers in the new update, an update that Infura and Blockchair, among others, ignored.
Besides these two service providers, other Ethereum users and wallet providers were also affected because they didn’t update their code. The fiasco has critics challenging Ethereum’s perceived decentralization, while stakeholders are wondering why the change was pushed in secret without coordination between Geth and other development teams. To some, the split is the most pressing challenge for Ethereum since the infamous DAO hack of 2016.
In a just-published post-mortem, Péter Szilágyi, a team lead at Ethereum, wrote that a fork “was (deliberately) triggered on the Ethereum network.”
A representative from Optimism, an Ethereum scaling project, recently posted that the project purposefully triggered the bug the Geth update fixed, setting off the fork. When the fork occurred, it created two versions of the Ethereum transaction ledger: one with transactions from updated Geth clients, and one with transactions from older Geth clients (like Infura).
“The fix was deployed several months ago and only today a transaction that caused that split came in,” Nikitia Zhavoronkov, the lead developer at Blockchair, an Ethereum block explorer who was affected by the fork, told CoinDesk in a direct message.
Thinking the update was “a minor change to the code,” Blockchair didn’t bother with the update because it wouldn’t be worth the downtime for their services. But more than minor, developers apparently made a quiet change to Geth’s consensus mechanism in the update, as well.
“The Geth team indeed changed the consensus implementation in the v1.9.17 release, however the team did not create any new rules that the Ethereum community didn’t know about or agree to,” Szilágyi writes in the post, saying these rules were laid out in an Ethereum Improvement Proposal three years ago.
“If you don’t consider accidentally introducing a bug a ‘consensus upgrade,’ then you should also not consider fixing the said bug a few months later a ‘consensus upgrade’,” he argued.
Ironically, the fork was caused by the very consensus bug that Geth’s update sought to address. The Ethereum bounty program recently recognized John Yang, a newcomer to Ethereum’s open-source community, for discovering this and another vulnerability. Geth developer and Ethereum security expert Martin Swende tweeted the changes in the update fix the disclosed issues, intimating that the debacle is a “reminder to keep your node(s) up to date!”
Swende continues to say in the tweet thread that developers did not announce the big change to avoid drawing attention to the flaw. In his own explanation, Szilágyi said that “silently” fixing the bug invited less “disruption.”
“Each major project that the dev team is in close contact with should have a security contact that can help manage and coordinate a smooth upgrade, and we should work together,” Matt Luongo, the founder of Thesis. “When forks are surprises, anyone who has built atop Ethereum like we have could lose money,” he continued.
Thesis builds the Keep Network, which issues tBTC, a form of tokenized bitcoin for the Ethereum blockchain. Luongo said the fork put tBTC users’ funds at risk, but not because of the chain split, which has been resolved after Infura and others updated their Geth clients.
It’s because the downtime meant that users staking Ethereum in Keep Network couldn’t coordinate with the Ethereum mainchain; as a result, they risked having part of their stakes “slashed” for not meeting their fiduciary requirements.
Despite the problems the split caused, prices for ether, the Ethereum blockchain’s native cryptocurrency, rose 4.6% Wednesday after the news emerged, suggesting that traders see little systemic or long-term threat from the snafu.
