On March 23rd, Sky Mavis’s Ronin validator nodes and Axie DAO validator nodes were compromised resulting in 173,600 Ethereum and 25.5M USDC drained from the Ronin bridge. A full timeline of the event can be found here. Now that the hackers have been identified, and all user funds are in the process of being restored, we would like to share a postmortem with details about the hack, what security measures are being implemented, and key learnings.

This security breach served as a reminder that no company is immune to external threats. Cryptocurrency theft is rising quickly and the purpose of this postmortem is to play a small part in keeping the entire industry more secure. We hope this information can serve as a guide to other companies that could be the target of such an attack.

Timeline

The hack occurred on March 23rd 2022, and was discovered on March 29th by the Sky Mavis team. We didn’t have a proper tracking system for monitoring large outflows from the bridge, which is why the breach wasn’t discovered immediately. Once the new bridge is deployed it will not be possible to withdraw transactions at this size without human interaction. The attacker managed to get control over five of the nine validator private keys — 4 Sky Mavis validators and 1 Axie DAO — in order to forge fake withdrawals. This resulted in 173,600 Ethereum and 25.5M USDC drained from the Ronin bridge in two transactions (1 and 2).

Attack details

Sky Mavis employees are under constant advanced spear-phishing attacks on various social channels and one employee was compromised. This employee no longer works at Sky Mavis. The attacker managed to leverage that access to penetrate Sky Mavis IT infrastructure and gain access to the validator nodes.

At the time, Sky Mavis controlled 4/9 validators, which would not be enough to forge withdrawals. The validator key scheme is set up to be decentralized so that it limits an attack vector, similar to this one, but the attacker found a backdoor through our gas-free RPC node, which they abused to get the signature for the Axie DAO validator.

This traces back to November 2021 when Sky Mavis requested help from the Axie DAO to distribute free transactions due to an immense user load. The Axie DAO allowlisted Sky Mavis to sign various transactions on its behalf. This was discontinued in December 2021, but the allowlist access was not revoked.

Once the attacker got access to Sky Mavis systems they were able to get the signature from the Axie DAO validator by using the gas-free RPC.

The vulnerability was fixed by adding additional validator nodes. However, to ensure this does not happen again we’ve deployed a comprehensive security roadmap.

Security Roadmap

Sky Mavis is taking the following steps to bolster its security now and in the future.

• Continuously working with top tier security experts to avoid lingering threats.

  • Shortly after the breach Sky Mavis engaged CrowdStrike and Polaris Infosec to handle internal surveillance and forensics. Sky Mavis is also working with other firms that will not be named to ensure hostile actors are unable to get an overview of our defenses.  

• Increasing the amount of Validating Nodes on Ronin Network

  • At the time of the security breach, Sky Mavis had nine validator nodes. We have increased this to 11 and are onboarding three more validator nodes soon. In the next three months, our target is 21 validator nodes, with the long-term goal of having over 100.

• Implement Stricter Internal Procedures

  • We are inspecting every area of security, including our internal procedures. We are putting a strong emphasis on security for all employees which includes more robust training courses to combat external threats and the use of work-only devices to further mitigate risks.

• Conduct Audits

  • Ronin is now the gold standard when it comes to security. All code is being fully reviewed and optimized, with security experts looking at the entire architecture.

• Create a Zero-Trust Organization

  • Our goal is to become a fully antifragile, zero-trust organization. Zero-trust is a framework that assumes that Sky Mavis is always at risk to external and internal threats. A zero-trust security model verifies and authorizes every connection, such as when a user connects to an application or software to a data set via an application programming interface. It ensures the interaction meets the conditional requirements of our security policies.

• Launch Bug Bounty

  • We recognize the importance and value of security researchers’ efforts in helping keep our community safe. Sky Mavis is offering bounties of up to $1 million to encourage responsible disclosure of security vulnerabilities. You can find more information about the Bug Bounty program here. 

• ISO27001 and other security related certifications. 

  • Over time Sky Mavis will go through various certification processes.

The Hackers

We now know that the FBI has attributed North Korea-based Lazarus Group, highly skilled hackers, to the Ronin Validator Security Breach. The US Government, specifically the Treasury Department, has sanctioned the address that received the stolen funds. Lazarus Group is a state-sponsored cybercrime organization that is extremely resourceful and sophisticated and has been connected to many high-profile hacks.

We are so grateful for every law enforcement official we have engaged with to help us identify the hackers and our focus now is to ensure this never happens again by implementing the strongest security measures.

Ronin Bridge Re-opening

The Ronin Network bridge is currently being redesigned and will open once we are confident that it can stand the test of time. We initially expected to be able to deploy the upgrade by the end of April, but this is not a process that we can afford to rush. The bridge will secure billions of dollars in assets, and it needs to be done right. If all goes as planned the bridge will reopen in mid/late May. In the meantime, Binance is supporting Ronin Network for both wETH and USDC withdrawals and deposits for Axie Infinity users.

Rest assured that all user funds in the bridge are guaranteed by the recent Sky Mavis funding round, Axie Infinity and Sky Mavis balance sheet assets, and personal funds from the core team.

In order to open the bridge we need to:

1. Upgrade the Ronin bridge contracts. We’re 80% done with this.

2. Rework the bridge’s backend: This is In progress

3. Deploy a validator dashboard that allows for approving large transactions and adding/removing new validators. This is being designed now.

4. Migrate all pending withdrawals: This will happen after the deployment of the new bridge

Conclusion

We are grateful for the community, our incredible board members, and law enforcement officials who have all come along aside us to help us navigate this uncharted territory and ensure we come out stronger than ever. You have all been a source of strength and inspiration for us. Our commitment to building a community-led digital nation that invites gamers to learn Web3 technology while building a better, decentralized world together has not wavered. We are ready to get back to building.