Testnet Halt 95

March 2020 Testnet Halt 95

  • Chain id: enigma-testnet

  • Date: 16/03/2020 3am UTC

  • Related issues: https://github.com/scrtlabs/SecretNetwork/issues/95

Description

On the 15 Mar 2020, around 9pm UTC the following param-change proposal was submitted:

{
  "title": "Rwd Change",
  "description": "This is tansaction to test a parameter-change for rewards.",
  "changes": [
    {
      "subspace": "distribution",
      "key": "baseproposerreward",
      "value": "0.999000000000000000"
    }
  ],
  "deposit": [
    {
      "denom": "uscrt",
      "amount": "10000000"
    }
  ]
}

At around 3am UTC of the following night the proposal got accepted, and as a result the network halted, with following error:

When the vote passed, the distribution module parameters changed to:

  • The problem occurred because the sum of baseproposerreward and bonusproposerreward can't be grater than 1 i.e. 0.999 + 0.04 > 1. This results in miscalculations of the rewards and fees.

  • The cause is a bug in Cosmos SDK in the parameter value validation, causing the proposal to pass despite being invalid. More on that here: https://github.com/cosmos/cosmos-sdk/issues/5808

Additional Notes

Another invalid proposal was on voting period, and by itself would have caused the network to halt as well:

Action Items

  • https://github.com/scrtlabs/SecretNetwork/issues/95

  • https://github.com/scrtlabs/SecretNetwork/issues/97

  • https://github.com/scrtlabs/SecretNetwork/issues/104

Recovery Process

  1. Logged in to the testnet bootstrap machine.

  2. Exported state from the last "rounded" block height:

  1. Removed all references to proposal ids 4 and 5 in:

  1. Made sure the distribution parameters still make sense:

  1. Erased the coins in possession of the gov ModuleAccount:

  1. "Refund" coins to the account that deposited to these proposals on the first place i.e. added to account's balance in:

  1. A problem occurred with staking, described at: https://github.com/cosmos/cosmos-sdk/issues/5818 Changed the following:

To this:

  1. Then a problem occured with the compute module:

This one got fixed when deleted the .enigmad/.compute directory.

  1. Reset state:

  1. Restarted the node.

Last updated

Was this helpful?