Secret Network
WebsiteDiscordGithub
  • 👋INTRODUCTION
    • Secret Network Introduction
    • Secret Network Techstack
      • Private transactions - A quick summary
      • Blockchain Technology
        • Cosmos Basics
        • Tendermint
        • Cosmos SDK
        • IBC
        • CosmWasm
      • Privacy Technology
        • Encryption - Key Management
          • Overview
          • Key Derivation & Encryption Techniques
          • The Initialization Of Secret Network
          • Full Node Boostrap
          • Contract State Encryption
          • Transaction Encryption
          • Consensus seed rotation
        • Trusted Execution Environments  (TEE) —  Intel SGX
          • How Secret Network Uses SGX
          • SGX-SPS Security & Reliabillity
          • Remote Attestation
          • Trusted & Untrusted Core
          • Sealing
        • Private smart contracts - overview
  • 💻Development
    • 🏠Getting Started
      • Setting Up Your Environment
        • Cargo.toml Best Practices (Crates vs Dependencies)
      • Compile and Deploy
      • Running the Application
      • Compile and Deploy on Secret testnet (best for Javascript devs)
      • Fullstack dApp Integration
    • 🤓Secret Contracts
      • Secret Contracts & CosmWasm
        • Framework overview
        • Secret Contract Components
          • Instantiation Message
          • Execution Message
          • Query Message
          • Deps/DepsMut
          • Storage
            • Prefixed Storage
            • Singleton
            • Keymap
            • Append Store
            • Best practices
        • CosmWasm vs Secret CosmWasm
      • Secret Tokens (SNIP-20)
      • Contract - module call
      • Secret contract - Design Space/Features
        • Secret Contracts introduction
        • Gas/Fee usage
        • TPS and scalability
        • Privacy Essentials
        • Access Control
          • Viewing Keys
          • Permits
        • Trusted and untrusted data
        • Secret-VRF - on-chain Randomness
        • Privacy design
          • Mitigate privacy risks - full guide
          • Gas Evaporation & Tracking
        • Confidential Computing Layer
        • Fiat/Crypto Onboarding
        • Account abstraction
        • Fee abstraction
        • Wallet support
        • Bridge (messaging/tokens)
        • IBC (Hooks, PFM, Wasm)
        • Price Oracles
        • Auto Restaking
      • Permissioned Viewing
        • Viewing Keys
        • Permits
      • Cross Contract Communication
      • Submessages
        • get_contract_code_hash
      • Randomness API - Secret VRF
        • Native On-chain randomness
        • Randomness over IBC
      • Execution Finalization
      • Factory Contracts
      • Contract Migration
        • Manual - < v1.11
        • Native - from v1.11
      • Cross-deploy Vanilla CW and Secret Contracts
      • Testing Secret Contracts
        • Unit Tests
        • Continuous Integration
        • Datatype Handling - Uint, floats etc.
    • 👀Secret Contract - Reference, Guides, Examples
      • Starter guide - Millionaire's Problem
      • Reference Contracts
      • Open source dApps
      • Tools & Libraries
        • Network interaction SDKs
          • Secret.js (JavaScript)
          • SecretPy (Python)
          • SecretK (Kotlin)
          • Secret.NET
            • Snippets
              • Deploying a Contract
              • Send Native Coin
              • Query a Contract
              • Create a new Wallet
              • Permits
              • SNIP20
              • SNIP721
          • Shade.Js
        • LocalSecret - Devnet docker
        • Smart contract - Tools/Utils
          • Secret Toolkit
          • CW-Plus
          • Fadroma - SC framework
          • Hidden Gems
          • Other
            • Secret IDE
            • Polar
    • 🖼️Frontend Development
      • Getting Started with SecretJS
        • Usage Examples
          • Sending Queries
          • Sending Messages
          • Contract Migration
          • Wallet Integrations
          • Query Permits
          • SNIP20 (SCRT Tokens)
          • SNIP721 (Secret NFTs)
      • Feegrant
        • Understanding Feegrant allowances
        • Grant allowances
        • Using grant allowances to execute transactions
        • Using the Fee Grant Faucet
    • 📬Deployment Addresses/ API Endpoints
      • Secret (SNIP 20) token contracts list
        • SNIP Asset naming guidelines
      • Connecting to the Network
        • API Endpoints Mainnet (Secret-4)
        • API Endpoints Testnet (Pulsar-3)
        • Usage examples
        • Comparison of endpoint types
      • Create your own SNIP-25 IBC Token
  • Secret AI
    • 🤝Introduction
    • 🏗️Architecture
    • 👩‍💻Secret AI SDK
      • Setting Up Your Environment
      • Running the Application
    • 🪙Economics
    • Smart Contract Reference
      • SubscriptionManager
      • WorkerManager
      • RewardsManager
  • 🌐SecretVM - Confidential Virtual Machines
    • 🤝Introduction
    • 🏗️Architecture
    • ☑️Attestation
      • What is Attestation
      • Obtaining Attestation Data
      • Attestation Report - Key Fields
      • Chain of Trust
    • 🏁Launching a SecretVM
    • 🛠️Managing SecretVM Lifecycle
    • ✅Verifying a SecretVM
    • 💡Best Practices for Developers
    • 📖Glossary
  • 🔓Confidential Computing Layer
    • 🧑‍🚀IBC Developer Toolkit
      • Basics
        • Overview
        • Cross-chain Messaging with IBC Hooks
          • Functions, Methods, and Data Structures
          • Typescript SDK
          • IBC-Hooks
        • IBC Relaying with Go Relayer
      • Usecases
        • Storing Encrypted Data on Secret Network
          • Key-Value store Developer Tutorial
        • Secret VRF for IBC with IBC-Hooks
        • Confidential Voting
        • Sealed Bid Auctions
      • Supported Networks
        • Mainnet
        • Testnet
    • 🤝Ethereum (EVM) Developer Toolkit
      • Basics
        • Overview
        • Connecting Metamask to Secret Network
        • SecretPath + Reown integration
        • Cross-chain Messaging
          • SecretPath
            • Architecture Overview
            • Detailed Architecture
            • SecretPath Developer Tutorials
            • Public EVM Gateway Architecture
            • How to deploy SecretPath on your chain
          • Axelar GMP
            • Architecture Overview
            • Axelar GMP Developer Tutorial
      • Usecases
        • Storing Encrypted Data on Secret Network
          • Key-Value store Developer Tutorial
        • Sealed Bid Auction
          • Sealed Bid Auction Developer Tutorial
        • Confidential Voting
          • Confidential Voting Developer Tutorial with SecretPath
        • VRF
          • Implementing VRF into any EVM Contract
          • VRF Developer Tutorial
          • Performance figures of SecretVRF vs competitors
          • Using encrypted payloads for VRF
          • Converting from Chainlink VRF to Secret VRF in four steps
        • Confidential Document Sharing
        • Tokens
          • From EVM to Secret
      • Supported Networks
        • EVM
          • EVM Mainnet
          • EVM Testnet
          • Gateway Contract ABI
        • Secret Gateway
          • SecretPath mainnet (secret-4) contracts
          • SecretPath testnet (pulsar-3) contracts
    • 🙌Solana Developer Toolkit
      • Usecases
        • Storing Encrypted Data on Secret Network
          • Key-value Store Developer Tutorial
        • VRF
          • VRF Developer Tutorial
      • Program IDs
        • Solana Mainnet & Testnet
        • Gateway Contract IDL
  • 🤫Overview, Ecosystem and Technology
    • 🚀Secret Network Overview
      • The SCRT coin
      • Private Tokens
      • Use Cases
        • Decentralized Finance (DeFi)
        • Art And Digital Media
        • Gaming
        • Data Privacy
        • Payments And Transactions
        • Communication
      • The technology
      • History
      • Roadmap (Core development)
        • Secret 2.0
      • Where To Buy SCRT?
      • Using the Testnet
    • 🐸Ecosystem Overview
      • Wallets
      • Applications
      • Explorers & tools
      • Funding
        • SCRT Labs Grants
        • Dilutive funding/VC raise
        • Community Pool
        • Developer bounties
          • SCRT Labs bounties [on-hold]
          • CCBL [on-hold]
          • CCR [On-hold]
        • Application specific
          • Shade Grants
      • Contributors & Entities
        • Validators
        • SCRT Labs
        • Secret Foundation
        • Secret Committees
          • Support
          • Governance
      • Secret Network Dictionary
  • 🔧Infrastructure
    • 🔓Use SecretCLI
      • Secretcli vs. Secretd
      • Install
      • Configuration
      • Address Types
      • Key Types
      • Generating Keys
      • Viewing Keys
      • Query Transactions
      • Send Tokens
      • Multisig Keys
      • Multisig Transactions
      • Transaction Broadcasting
      • Fees & Gas
      • Fee Distribution
      • Secret Contracts
      • Slashing
      • Minting
      • Delegating
      • Restake
      • Nodes
      • Governance
        • Creating Governance Proposals
        • Query Proposals
        • Deposits
        • Voting
    • 🔐Use Ledger hardware wallet
      • 🔐Ledger with SecretCLI
    • 🖥️Running a node/validator
      • Setting up a node/validator
        • Hardware setup
          • Hardware Compliance
          • VPS/Bare-Metal Compliance
            • Leaseweb Setup
            • PhoenixNAP Setup
            • Psychz Setup
            • nForce Setup
            • Vultr Setup
            • OVHCloud Setup
            • Microsoft Azure Setup
          • Patching your Node
          • Enclave verification
          • Registration troubleshooting
        • Testnet Setup
          • Install SGX
          • Install secretd
          • Setup Full Node
          • Testnet State Sync
          • Becoming a Testnet Validator
          • Installing CLI & Creating A New Address
        • Mainnet Setup
          • Install SGX
          • Install secretd
          • Setup Full Node
          • Quicksync / Snapshot
          • Statesync
          • Becoming A Validator
          • Installing CLI & Creating A New Address
      • Maintaining a node/validator
        • Slashing information
        • Migrating a Validator
        • Troubleshooting
        • Validator Backup
        • Server security
          • SSH authentication
          • Server configuration
          • Uncomplicated-Firewall (UFW)
          • Local CLI
        • Node Monitoring
          • Prometheus
            • Environment Preperation
            • Install Node Exporter
            • Install Prometheus
            • Configuring Prometheus
          • Grafana
            • Install Grafana
            • Grafana Dashboard
            • Next Steps
          • Docker
            • Install Docker
            • Configuration
            • Start Containers
            • Grafana Dashboard
            • Application Ports
            • Stop Containers
          • Goaccess
            • Install Goaccess
            • Setup Goaccess
        • Helpful commands
          • Query Validators
          • Bond Tokens
          • Withdraw Rewards
          • Query Delegations
          • Unbond Tokens
          • Query Unbonding-Delegations
          • Redelegate Tokens
          • Query Redelegations
          • Query Parameters
          • Query Pool
          • Query Delegations To Validator
      • API Noderunning
        • Running Multiple Nodes on the Same Server
        • Node Loadbalancing using Nginx
          • Setup Nginx
          • Example Nginx config
        • Using Auto heal to improve cluster uptime for Nginx
      • Sentry and Archive nodes
        • Mantlemint
        • Sentry Nodes
        • Archive Nodes
    • ⛓️IBC Relayers
      • Hermes
      • RLY
      • IBC channel database
    • 🆙Upgrade Instructions
      • v1.13
      • v1.12
      • v1.11
      • v1.10
      • v1.9
      • v1.8
      • v1.7
      • Shockwave Omega v1.6
      • v1.5
      • Shockwave Delta v1.4
      • Shockwave Delta v1.4 (Testnet)
      • Shockwave Alpha v1.3
      • Cosmovisor
      • Vulcan Network Upgrade (OLD)
    • ☠️Postmortems
      • SNIP-20 leaks
      • xApic
      • Secpk-Verifications Bloat
      • Earn Contract Exploit
      • Testnet Halt 95
    • ✍️Contribute to the documentation
      • Report Bugs
      • Suggest Enhancements
      • First Contribution Guide
      • Pull Request Templates
        • Update Documentation
        • Bug Fix
        • Improve Performance
        • Change Functionality
      • Style Guide
    • 🌊Versioning & Changelog
      • Secret Network v1.13
      • Secret Network v1.12
      • Secret Network v1.11
      • Secret Network v1.10
      • Secret network v1.9
      • Secret Network v1.7/1.8
      • Secret Network v1.6
      • Secret Network v1.5
      • Secret Network v1.4 (CosmWasm 1.0)
Powered by GitBook
On this page
  • Prerequisites
  • Clone Repository
  • Install Required Libraries for Python
  • Setup the enviroment file
  • Create a File with Node URLs
  • Execute the Script

Was this helpful?

Edit on GitHub
Export as PDF
  1. Infrastructure
  2. Running a node/validator
  3. API Noderunning

Using Auto heal to improve cluster uptime for Nginx

PreviousExample Nginx configNextSentry and Archive nodes

Last updated 1 year ago

Was this helpful?

Most noderunners have the common problem that stale nodes, meaning nodes that are stuck at a certain block height and will not continue keeping up the chain tip, provide horrible UX to users. To mitigate this, we can add some extra monitoring tools to dynamically add or remove nodes from the cluster.

The provided software gives you an entry level solution for this. As everything is written in Python, you can adjust this to your API setup.

To install, please do the following:

Prerequisites

Ensure you have Python installed. If not, download and install Python from . You'll also need pip, Python's package manager, to install required libraries. If you're using a Linux-based system, ensure you have NGINX installed and properly configured.

Since we edit the nginx config directly, we need to give pyhton3 sudo rights.

Be aware when you follow this tutorial.

Clone Repository

To clone the AutoHealBot repository, use the following steps:

  1. Install Git: Ensure Git is installed on your system. If not, install it from .

  2. Clone the Repository: Open a terminal and run:

    git clone https://github.com/SecretSaturn/AutoHealBot.git
  3. Navigate to the Directory: After cloning, change to the repository directory:

    cd AutoHealBot

This clones the entire repository to your local machine, allowing you to access all files and resources. To proceed with the tutorial, follow additional setup instructions provided in the repository's README or other documentation.

Install Required Libraries for Python

When installing, make sure to install this under root with sudo, otherwise the script will later not find the libraries later on.

sudo pip install aiohttp python-dotenv dateutil

Setup the enviroment file

To configure your environment variables, copy over the .env.example in the repository.

# Path to your Nginx Config
NGINX_CONFIG_PATH = '/path/to/your/nginx.conf'

# Base rate limit in requests per second
BASE_RATE = 8

# Rate increase per healthy node
NODE_MULTIPLIER = 1

# Port numbers for RPC, gRPC, and LCD
RPC_PORT = 26657
GRPC_PORT = 9091
LCD_PORT = 1317

# Path to the file containing node URLs
FILE_PATH = '/path/to/your/file.txt'

# Time in seconds to determine if a node has fallen behind
TIME_BEFORE_FALLEN_BEHIND = 18

# Time interval in seconds for updating the status
UPDATE_TIME = 30

Replace the placeholders with actual values:

  • NGINX_CONFIG_PATH: The path to your NGINX configuration file.

  • BASE_RATE and NODE_MULTIPLIER: Adjust as needed.

  • RPC_PORT, GRPC_PORT, LCD_PORT: Set to your specific ports.

  • FILE_PATH: Path to the text file with node URLs.

  • TIME_BEFORE_FALLEN_BEHIND: Maximum allowed time before a node is considered unhealthy.

  • UPDATE_TIME: Time between health checks.

Create a File with Node URLs

Create a text file with the node URLs. For example, create nodes.txt with one URL per line, make sure to include the RPC port to each node here as well:

192.168.0.1:26657
192.168.0.2:26657
192.168.0.2:26657

In the AutoHealBot script, "upstream blocks" refer to sections in the NGINX configuration that specify which backend servers handle different types of traffic. This setup divides the backend nodes into separate streams: RPC, gRPC, and LCD. The script checks the health of these nodes and updates the corresponding upstream blocks to reflect the healthy nodes for each stream. It ensures that traffic is routed to servers that are online and functional.

As a reference, the upstream blocks are defined as:

upstream_blocks = { 
"rpc_stream": healthy_nodes_rpc, 
"grpc_stream": healthy_nodes_grpc, 
"lcd_stream": healthy_nodes_lcd, 
}

Execute the Script

Run the script to start the asynchronous health checks and NGINX updates:

sudo python3 main.py

Troubleshooting

  • Environment Variables Not Loaded: Ensure your .env file is in the same directory as the script or specify its path explicitly with dotenv_path.

  • NGINX Not Reloading: Check if you have the necessary permissions to reload NGINX and ensure systemctl or other command-line utilities are in your PATH.

With this setup, the script will run asynchronously, periodically checking node health, updating the NGINX configuration, and reloading the NGINX service as needed.

🔧
🖥️
python.org
git-scm.com