Secret Network
WebsiteDiscordGithub
  • 👋INTRODUCTION
    • Secret Network Introduction
    • Secret Network Techstack
      • Private transactions - A quick summary
      • Blockchain Technology
        • Cosmos Basics
        • Tendermint
        • Cosmos SDK
        • IBC
        • CosmWasm
      • Privacy Technology
        • Encryption - Key Management
          • Overview
          • Key Derivation & Encryption Techniques
          • The Initialization Of Secret Network
          • Full Node Boostrap
          • Contract State Encryption
          • Transaction Encryption
          • Consensus seed rotation
        • Trusted Execution Environments  (TEE) —  Intel SGX
          • How Secret Network Uses SGX
          • SGX-SPS Security & Reliabillity
          • Remote Attestation
          • Trusted & Untrusted Core
          • Sealing
        • Private smart contracts - overview
  • 💻Development
    • 🏠Getting Started
      • Setting Up Your Environment
        • Cargo.toml Best Practices (Crates vs Dependencies)
      • Compile and Deploy
      • Running the Application
      • Compile and Deploy on Secret testnet (best for Javascript devs)
      • Fullstack dApp Integration
    • 🤓Secret Contracts
      • Secret Contracts & CosmWasm
        • Framework overview
        • Secret Contract Components
          • Instantiation Message
          • Execution Message
          • Query Message
          • Deps/DepsMut
          • Storage
            • Prefixed Storage
            • Singleton
            • Keymap
            • Append Store
            • Best practices
        • CosmWasm vs Secret CosmWasm
      • Secret Tokens (SNIP-20)
      • Contract - module call
      • Secret contract - Design Space/Features
        • Secret Contracts introduction
        • Gas/Fee usage
        • TPS and scalability
        • Privacy Essentials
        • Access Control
          • Viewing Keys
          • Permits
        • Trusted and untrusted data
        • Secret-VRF - on-chain Randomness
        • Privacy design
          • Mitigate privacy risks - full guide
          • Gas Evaporation & Tracking
        • Confidential Computing Layer
        • Fiat/Crypto Onboarding
        • Account abstraction
        • Fee abstraction
        • Wallet support
        • Bridge (messaging/tokens)
        • IBC (Hooks, PFM, Wasm)
        • Price Oracles
        • Auto Restaking
      • Permissioned Viewing
        • Viewing Keys
        • Permits
      • Cross Contract Communication
      • Submessages
        • get_contract_code_hash
      • Randomness API - Secret VRF
        • Native On-chain randomness
        • Randomness over IBC
      • Execution Finalization
      • Factory Contracts
      • Contract Migration
        • Manual - < v1.11
        • Native - from v1.11
      • Cross-deploy Vanilla CW and Secret Contracts
      • Testing Secret Contracts
        • Unit Tests
        • Continuous Integration
        • Datatype Handling - Uint, floats etc.
    • 👀Secret Contract - Reference, Guides, Examples
      • Starter guide - Millionaire's Problem
      • Reference Contracts
      • Open source dApps
      • Tools & Libraries
        • Network interaction SDKs
          • Secret.js (JavaScript)
          • SecretPy (Python)
          • SecretK (Kotlin)
          • Secret.NET
            • Snippets
              • Deploying a Contract
              • Send Native Coin
              • Query a Contract
              • Create a new Wallet
              • Permits
              • SNIP20
              • SNIP721
          • Shade.Js
        • LocalSecret - Devnet docker
        • Smart contract - Tools/Utils
          • Secret Toolkit
          • CW-Plus
          • Fadroma - SC framework
          • Hidden Gems
          • Other
            • Secret IDE
            • Polar
    • 🖼️Frontend Development
      • Getting Started with SecretJS
        • Usage Examples
          • Sending Queries
          • Sending Messages
          • Contract Migration
          • Wallet Integrations
          • Query Permits
          • SNIP20 (SCRT Tokens)
          • SNIP721 (Secret NFTs)
      • Feegrant
        • Understanding Feegrant allowances
        • Grant allowances
        • Using grant allowances to execute transactions
        • Using the Fee Grant Faucet
    • 📬Deployment Addresses/ API Endpoints
      • Secret (SNIP 20) token contracts list
        • SNIP Asset naming guidelines
      • Connecting to the Network
        • API Endpoints Mainnet (Secret-4)
        • API Endpoints Testnet (Pulsar-3)
        • Usage examples
        • Comparison of endpoint types
      • Create your own SNIP-25 IBC Token
  • Secret AI
    • 🤝Introduction
    • 🏗️Architecture
    • 👩‍💻Secret AI SDK
      • Setting Up Your Environment
      • Running the Application
    • 🪙Economics
    • Smart Contract Reference
      • SubscriptionManager
      • WorkerManager
      • RewardsManager
  • 🌐SecretVM - Confidential Virtual Machines
    • 🤝Introduction
    • 🏗️Architecture
    • ☑️Attestation
      • What is Attestation
      • Obtaining Attestation Data
      • Attestation Report - Key Fields
      • Chain of Trust
    • 🏁Launching a SecretVM
    • 🛠️Managing SecretVM Lifecycle
    • ✅Verifying a SecretVM
      • 🐢Full Verification
      • 🐰Quick Verification
    • 👨‍💻SecretVM CLI
      • Getting started
      • Authentication Commands
      • Virtual Machine Commands
    • 💡Best Practices for Developers
    • 🔐Verifiable Message Signing
    • 📖Glossary
  • 🔓Confidential Computing Layer
    • 🧑‍🚀IBC Developer Toolkit
      • Basics
        • Overview
        • Cross-chain Messaging with IBC Hooks
          • Functions, Methods, and Data Structures
          • Typescript SDK
          • IBC-Hooks
        • IBC Relaying with Go Relayer
      • Usecases
        • Storing Encrypted Data on Secret Network
          • Key-Value store Developer Tutorial
        • Secret VRF for IBC with IBC-Hooks
        • Confidential Voting
        • Sealed Bid Auctions
      • Supported Networks
        • Mainnet
        • Testnet
    • 🤝Ethereum (EVM) Developer Toolkit
      • Basics
        • Overview
        • Connecting Metamask to Secret Network
        • SecretPath + Reown integration
        • Cross-chain Messaging
          • SecretPath
            • Architecture Overview
            • Detailed Architecture
            • SecretPath Developer Tutorials
            • Public EVM Gateway Architecture
            • How to deploy SecretPath on your chain
          • Axelar GMP
            • Architecture Overview
            • Axelar GMP Developer Tutorial
      • Usecases
        • Storing Encrypted Data on Secret Network
          • Key-Value store Developer Tutorial
        • Sealed Bid Auction
          • Sealed Bid Auction Developer Tutorial
        • Confidential Voting
          • Confidential Voting Developer Tutorial with SecretPath
        • VRF
          • Implementing VRF into any EVM Contract
          • VRF Developer Tutorial
          • Performance figures of SecretVRF vs competitors
          • Using encrypted payloads for VRF
          • Converting from Chainlink VRF to Secret VRF in four steps
        • Confidential Document Sharing
        • Tokens
          • From EVM to Secret
      • Supported Networks
        • EVM
          • EVM Mainnet
          • EVM Testnet
          • Gateway Contract ABI
        • Secret Gateway
          • SecretPath mainnet (secret-4) contracts
          • SecretPath testnet (pulsar-3) contracts
    • 🙌Solana Developer Toolkit
      • Usecases
        • Storing Encrypted Data on Secret Network
          • Key-value Store Developer Tutorial
        • VRF
          • VRF Developer Tutorial
      • Program IDs
        • Solana Mainnet & Testnet
        • Gateway Contract IDL
  • 🤫Overview, Ecosystem and Technology
    • 🚀Secret Network Overview
      • The SCRT coin
      • Private Tokens
      • Use Cases
        • Decentralized Finance (DeFi)
        • Art And Digital Media
        • Gaming
        • Data Privacy
        • Payments And Transactions
        • Communication
      • The technology
      • History
      • Roadmap (Core development)
        • Secret 2.0
      • Where To Buy SCRT?
      • Using the Testnet
    • 🐸Ecosystem Overview
      • Wallets
      • Applications
      • Explorers & tools
      • Funding
        • SCRT Labs Grants
        • Dilutive funding/VC raise
        • Community Pool
        • Developer bounties
          • SCRT Labs bounties [on-hold]
          • CCBL [on-hold]
          • CCR [On-hold]
        • Application specific
          • Shade Grants
      • Contributors & Entities
        • Validators
        • SCRT Labs
        • Secret Foundation
        • Secret Committees
          • Support
          • Governance
      • Secret Network Dictionary
  • 🔧Infrastructure
    • 🔓Use SecretCLI
      • Secretcli vs. Secretd
      • Install
      • Configuration
      • Address Types
      • Key Types
      • Generating Keys
      • Viewing Keys
      • Query Transactions
      • Send Tokens
      • Multisig Keys
      • Multisig Transactions
      • Transaction Broadcasting
      • Fees & Gas
      • Fee Distribution
      • Secret Contracts
      • Slashing
      • Minting
      • Delegating
      • Restake
      • Nodes
      • Governance
        • Creating Governance Proposals
        • Query Proposals
        • Deposits
        • Voting
    • 🔐Use Ledger hardware wallet
      • 🔐Ledger with SecretCLI
    • 🖥️Running a node/validator
      • Setting up a node/validator
        • Hardware setup
          • Hardware Compliance
          • VPS/Bare-Metal Compliance
            • Leaseweb Setup
            • PhoenixNAP Setup
            • Psychz Setup
            • nForce Setup
            • Vultr Setup
            • OVHCloud Setup
            • Microsoft Azure Setup
          • Patching your Node
          • Enclave verification
          • Registration troubleshooting
        • Testnet Setup
          • Install SGX
          • Install secretd
          • Setup Full Node
          • Testnet State Sync
          • Becoming a Testnet Validator
          • Installing CLI & Creating A New Address
        • Mainnet Setup
          • Install SGX
          • Install secretd
          • Setup Full Node
          • Quicksync / Snapshot
          • Statesync
          • Becoming A Validator
          • Installing CLI & Creating A New Address
      • Maintaining a node/validator
        • Slashing information
        • Migrating a Validator
        • Troubleshooting
        • Validator Backup
        • Server security
          • SSH authentication
          • Server configuration
          • Uncomplicated-Firewall (UFW)
          • Local CLI
        • Node Monitoring
          • Prometheus
            • Environment Preperation
            • Install Node Exporter
            • Install Prometheus
            • Configuring Prometheus
          • Grafana
            • Install Grafana
            • Grafana Dashboard
            • Next Steps
          • Docker
            • Install Docker
            • Configuration
            • Start Containers
            • Grafana Dashboard
            • Application Ports
            • Stop Containers
          • Goaccess
            • Install Goaccess
            • Setup Goaccess
        • Helpful commands
          • Query Validators
          • Bond Tokens
          • Withdraw Rewards
          • Query Delegations
          • Unbond Tokens
          • Query Unbonding-Delegations
          • Redelegate Tokens
          • Query Redelegations
          • Query Parameters
          • Query Pool
          • Query Delegations To Validator
      • API Noderunning
        • Running Multiple Nodes on the Same Server
        • Node Loadbalancing using Nginx
          • Setup Nginx
          • Example Nginx config
        • Using Auto heal to improve cluster uptime for Nginx
      • Sentry and Archive nodes
        • Mantlemint
        • Sentry Nodes
        • Archive Nodes
    • ⛓️IBC Relayers
      • Hermes
      • RLY
      • IBC channel database
    • 🆙Upgrade Instructions
      • v1.13
      • v1.12
      • v1.11
      • v1.10
      • v1.9
      • v1.8
      • v1.7
      • Shockwave Omega v1.6
      • v1.5
      • Shockwave Delta v1.4
      • Shockwave Delta v1.4 (Testnet)
      • Shockwave Alpha v1.3
      • Cosmovisor
      • Vulcan Network Upgrade (OLD)
    • ☠️Postmortems
      • SNIP-20 leaks
      • xApic
      • Secpk-Verifications Bloat
      • Earn Contract Exploit
      • Testnet Halt 95
    • ✍️Contribute to the documentation
      • Report Bugs
      • Suggest Enhancements
      • First Contribution Guide
      • Pull Request Templates
        • Update Documentation
        • Bug Fix
        • Improve Performance
        • Change Functionality
      • Style Guide
    • 🌊Versioning & Changelog
      • Secret Network v1.13
      • Secret Network v1.12
      • Secret Network v1.11
      • Secret Network v1.10
      • Secret network v1.9
      • Secret Network v1.7/1.8
      • Secret Network v1.6
      • Secret Network v1.5
      • Secret Network v1.4 (CosmWasm 1.0)
Powered by GitBook
On this page
  • What happened?
  • Why Did This Happen?

Was this helpful?

Edit on GitHub
Export as PDF
  1. Infrastructure
  2. Postmortems

Secpk-Verifications Bloat

PreviousxApicNextEarn Contract Exploit

Last updated 1 year ago

Was this helpful?

On the afternoon of Feb 21, 2022 the community started seeing impacted network performance stemming from the launch of the Shade airdrop. The performance degradation had multiple reasons and a lot of lessons were learned from this stress test. Some of these findings are documented here so to educate network participants.

The community came together quite quickly to solve as many problems as possible, the incident response can be found here:

What happened?

On Feb 21, 2022 at around 11pm UTC the Shade protocol airdrop was launched, drawing a lot of attention to Secret Network.

As a part of their airdrop mechanism, Shade heavily utilized secp256k1 signature verification in their contracts, which is very computationally expensive.

These transactions are causing blocks to slow down due to the time required to compute each block, causing the mempool to fill up which delays the execution of transactions. A further effect of blocks that take a long time to compute is that queries were slowed down as well, as at that time a node could not both compute a block and serve a request.

The network was clogged with transactions for multiple days on end because of the high demand and slow block times, after rate limiting the shade airdrop application the network picked back up again so that the rest of the applications were no longer affected.

Why Did This Happen?

The reason for the abnormal behavior is mostly due to nodes running an outdated WebAssembly engine, which does not handle long computations very efficiently. Also, gas calculations do not account for this inefficiency, which further compounds the issue.

To put it simply, these Secpk verifications took more computation power than they were paying for in Gas. A full block would therefore be several magnitudes more complex to compute which made it so that validators were not done in the normal 6 second time frame causing block time to become longer and longer. Longer block times means less space for transactions per second causing the network to be too slow to handle transactions.

A secondary reason as to why validator nodes were not able to meet the computational demand was not related to hardware, but to peering. Validators speak to their peers so to pass information about consensus along. Validators can set persistent peers but the network can decide to completely cut off certain validators when the network is being stressed. What happened is that during the chain congestion only certain groups of validators were talking to each other. When 67% of voting power was found the block would be signed leaving the rest of the validators to not sign at all. This created very spotty patterns in the nodes who signed blocks, some were signing every one and others were signing none even though they were both done before the block was committed.

Quick note: The shinobi protocol testnet launched slightly before the shade airdrop also caused significant slowdown of the network, blocks were full just from a few of their transactions at the same time. it was later recognised that this was because of the same problem with Secpk verifications which Shinobi uses for light client verification's of cross-chain bitcoin transfers.

What has been done

  1. Firstly, a small upgrade was released which significantly improved the query node performance. This upgrade allows nodes to both serve many more requests, and lessen the impact of long block computations. This will help services like Keplr stay available during network-wide events. Reference:

  2. The execution performance for computationally expensive functions like secp256k1 verification are changed to being exposed to contracts (instead of being executed inside the contract) which made them much more efficient. New APIs were released during Shockwave Alpha which brought 500x improvements to these transactions. Reference:

  3. Introduced Seeds for solving the validator peering issues. One can reference the to add these seeds to their peering list.

  4. We are also replacing our WASM engine with a newer, more performant one. This item is still on the and will help with long term scalability of Secret Network.

  5. Lastly, we will also be re-evaluating gas calculation and pricing and try to adjust the gas to more accurately reflect the computational cost of each contract. This was a huge lesson learned, gas needs to equal the computational cost or nodes will not be able to handle the load.

For some extra information on lessons learned from this event you can read this blog:

🔧
☠️
https://forum.scrt.network/t/discuss-network-issues-w-shade-airdrop-2-21-22/5475
https://github.com/scrtlabs/SecretNetwork/releases/tag/v1.2.5
https://github.com/scrtlabs/SecretNetwork/releases/tag/v1.3.0
validator documentation
roadmap
https://scrt.network/blog/scrt-labs-update-scaling-secrets