Testnet Halt 95

March 2020 Testnet Halt 95

  • Chain id: enigma-testnet
  • Date: 16/03/2020 3am UTC
  • Related issues: https://github.com/scrtlabs/SecretNetwork/issues/95

Description

On the 15 Mar 2020, around 9pm UTC the following param-change proposal was submitted:
{
"title": "Rwd Change",
"description": "This is tansaction to test a parameter-change for rewards.",
"changes": [
{
"subspace": "distribution",
"key": "baseproposerreward",
"value": "0.999000000000000000"
}
],
"deposit": [
{
"denom": "uscrt",
"amount": "10000000"
}
]
}
At around 3am UTC of the following night the proposal got accepted, and as a result the network halted, with following error:
Mar 16 05:02:52 ip-172-31-44-28 enigmad[20612]: I[2020-03-16|05:02:52.767] Executed block module=state height=171146 validTxs=0 invalidTxs=0
Mar 16 05:02:52 ip-172-31-44-28 enigmad[20612]: I[2020-03-16|05:02:52.780] Committed state module=state height=171146 txs=0 appHash=FB4739B6F0D4FED77D431922E340B95B8144BF37483D3C1225431311A5BB229D
Mar 16 05:02:58 ip-172-31-44-28 enigmad[20612]: E[2020-03-16|05:02:58.075] CONSENSUS FAILURE!!! module=consensus err="negative coin amount" stack="goroutine 1419 [running]:\nruntime/debug.Stack(0xc0012c2ce0, 0x1115f40, 0x16671b0)\n\t/usr/lib/go-1.13/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2(0xc002741c00, 0x149b478)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:613 +0x57\npanic(0x1115f40, 0x16671b0)\n\t/usr/lib/go-1.13/src/runtime/panic.go:679 +0x1b2\ngithub.com/cosmos/cosmos-sdk/types.DecCoins.Sub(...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/types/dec_coin.go:307\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens(0x169cf60, 0xc000eab5b0, 0xc000ab8460, 0xc000ab8460, 0x169cf60, 0xc000eab5e0, 0x169cfa0, 0xc000eab630, 0xc000e9fbe0, 0xc, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:66 +0x1589\ngithub.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker(0x16ad9a0, 0xc000034048, 0x16c2100, 0xc005382800, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/abci.go:26 +0x2e6\ngithub.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/x/distribution/module.go:147\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(0xc000ab9030, 0x16ad9a0, 0xc000034048, 0x16c2100, 0xc005382800, 0xa, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:297 +0x1ca\ngithub.com/enigmampc/SecretNetwork.(*EnigmaChainApp).BeginBlocker(...)\n\t/home/assafmo/workspace/enigmachain/app.go:391\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc000d197c0, 0xc004071c00, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:136 +0x469\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc000e7cd20, 0xc004071c00, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:231 +0x101\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc000eab1b0, 0xc004071c00, 0x20, 0x20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:69 +0x6b\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp(0x16ae3a0, 0xc0027fa9a0, 0x16bb520, 0xc000eab1b0, 0xc003dc3a40, 0x16c4080, 0xc000011138, 0x6, 0xc00272a9a0, 0xe)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:280 +0x3e1\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc0000fce00, 0xa, 0x0, 0xc00272a980, 0x6, 0xc00272a9a0, 0xe, 0x29c8a, 0xc0038a1660, 0x20, ...)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:131 +0x17a\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc002741c00, 0x29c8b)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1431 +0x8f5\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc002741c00, 0x29c8b)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1350 +0x383\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1(0xc002741c00, 0x0, 0x29c8b)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1285 +0x90\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc002741c00, 0x29c8b, 0x0)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1322 +0x61a\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc002741c00, 0xc005087c20, 0xc003fd0b70, 0x28, 0xc0012c9a38, 0xd31f92, 0x0)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1819 +0xa39\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc002741c00, 0xc005087c20, 0xc003fd0b70, 0x28, 0xf136b9f2d600ff82, 0x108, 0x100)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1642 +0x59\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc002741c00, 0x168b640, 0xc0028f4538, 0xc003fd0b70, 0x28)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:709 +0x252\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc002741c00, 0x0)\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:644 +0x6eb\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/home/assafmo/workspace/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:335 +0x13a\n"
Mar 16 05:36:48 ip-172-31-44-28 enigmad[20612]: E[2020-03-16|05:36:48.395] Connection failed @ sendRoutine module=p2p [email protected]149.248.55.89:60440 conn=MConn{149.248.55.89:60440} err="pong timeout"
Mar 16 05:36:48 ip-172-31-44-28 enigmad[20612]: E[2020-03-16|05:36:48.402] Stopping peer for error module=p2p peer="Peer{MConn{149.248.55.89:60440} 842822b88cb5762ba99cee50a37156c6d0a6c452 in}" err="pong timeout"
When the vote passed, the distribution module parameters changed to:
community_tax: "0.020000000000000000"
base_proposer_reward: "0.999000000000000000"
bonus_proposer_reward: "0.040000000000000000"
withdraw_addr_enabled: true
  • The problem occurred because the sum of baseproposerreward and bonusproposerreward can't be grater than 1 i.e. 0.999 + 0.04 > 1. This results in miscalculations of the rewards and fees.
  • The cause is a bug in Cosmos SDK in the parameter value validation, causing the proposal to pass despite being invalid. More on that here: https://github.com/cosmos/cosmos-sdk/issues/5808

Additional Notes

Another invalid proposal was on voting period, and by itself would have caused the network to halt as well:
"changes": [
{
"subspace": "distribution",
"key": "bonusproposerreward",
"value": "\"0.999000000000000000\""
}
]

Action Items

  • https://github.com/scrtlabs/SecretNetwork/issues/95
  • https://github.com/scrtlabs/SecretNetwork/issues/97
  • https://github.com/scrtlabs/SecretNetwork/issues/104

Recovery Process

  1. 1.
    Logged in to the testnet bootstrap machine.
  2. 2.
    Exported state from the last "rounded" block height:
enigmad export --for-zero-height --height=170000 > state_export.json
  1. 1.
    Removed all references to proposal ids 4 and 5 in:
"gov":{
"deposits":[...],
"proposals":[...],
"votes":[...],
...
}
  1. 1.
    Made sure the distribution parameters still make sense:
"distribution":{
"params":{
"base_proposer_reward":"0.010000000000000000",
"bonus_proposer_reward":"0.040000000000000000",
"community_tax":"0.020000000000000000",
"withdraw_addr_enabled":true
},
...
}
  1. 1.
    Erased the coins in possession of the gov ModuleAccount:
"auth":{
"accounts":[
{
"type":"cosmos-sdk/ModuleAccount",
"value":{
"coins": [
<subtracted amount here>
]
"name":"gov"
}
}, ...
}, ...
  1. 1.
    "Refund" coins to the account that deposited to these proposals on the first place i.e. added to account's balance in:
"app_state":{
"auth":{
"accounts":[
{
"value":{
"coins":[
{
"amount":"<added to this amount>"
}
]
}
}
]
}
}
  1. 1.
    A problem occurred with staking, described at: https://github.com/cosmos/cosmos-sdk/issues/5818 Changed the following:
"distribution":{
"delegator_starting_infos":[
{
...,
"starting_info":{
"...,
"stake":"999990000.000000000000000000"
},
...
},
...
],
...
}
To this:
"distribution":{
"delegator_starting_infos":[
{
...,
"starting_info":{
"...,
"stake":"990000000.000000000000000000"
},
...
},
...
],
...
}
  1. 1.
    Then a problem occured with the compute module:
panic: create wasm contract failed: Wasm Error: Filesystem error: File exists (os error 17)
This one got fixed when deleted the .enigmad/.compute directory.
  1. 1.
    Reset state:
enigmad unsafe-reset-all
  1. 1.
    Restarted the node.
Export as PDF
Copy link
Edit on GitHub
On this page
March 2020 Testnet Halt 95
Description
Additional Notes
Action Items
Recovery Process