1 of 6

API Noderunning

This section should give you a rough overview of how to run multiple nodes in a cluster than you can provide for your own dApp to use.

You can find help in Telegram
Visit the Secret Network Discord and ask in #node-discussion or #node-support for help

Running Multiple Nodes on the Same Server

It's possible to run multiple Secret Nodes on the same Secret-compatible server, and it is fairly easy to do so.

Important Notes

There are 2 important things that must be done for each node:

A unique system file is necessary for each node
A unique sgx_secrets path is necessary for each node
All Secret Nodes should have their own user to simplify
It's easiest to do this with auto-register, but it's possible manual as well
Each node must be registered

Setup

This process assumes you already have a full node running. If you do not, proceed by Setting Up a Full Node, then returning.

1. Create a User

This isn't necessary, but will help with keeping nodes organized. From here on, the assumption is the username is secret, but it can be anything of your choosing.

adduser secret
usermod secret -aG sudo

2. Verify secretd Access

This will make it so you don't need to install secretd multiple times, and therefore, can upgrade all nodes at the same time.

secretd status

3. Begin Setting up a Node

On the new user, execute steps 1 and 2 of Setting Up a Full Node. You should now have a .secretd directory on the new user, and the correct genesis file.

4. Register the Node

The variables SCRT_ENCLAVE_DIR and SCRT_SGX_STORAGE are going to need to be custom for each user/node. These variables are NOT the same as the ones in step 3 of setting up a full node.

export SCRT_ENCLAVE_DIR=~/lib
export SCRT_SGX_STORAGE=~/.secretd/.sgx_secrets
secretd auto-register

5. Change Ports

In order for these nodes to work in tandem, they cannot use the same ports. I recommend this tool to help automate changing them.

Which will then create a command that looks like this:

sed -i.bak -e "s%^proxy_app = \"tcp://127.0.0.1:26658\"%proxy_app = \"tcp://127.0.0.1:10658\"%; s%^laddr = \"tcp://127.0.0.1:26657\"%laddr = \"tcp://127.0.0.1:10657\"%; s%^pprof_laddr = \"localhost:6060\"%pprof_laddr = \"localhost:10060\"%; s%^laddr = \"tcp://0.0.0.0:26656\"%laddr = \"tcp://0.0.0.0:10656\"%; s%^prometheus_listen_addr = \":26660\"%prometheus_listen_addr = \":10660\"%" $HOME/.secretd/config/config.toml && \
sed -i.bak -e "s%^address = \"tcp://0.0.0.0:1317\"%address = \"tcp://0.0.0.0:10317\"%; s%^address = \":8080\"%address = \":10080\"%; s%^address = \"0.0.0.0:9090\"%address = \"0.0.0.0:10090\"%; s%^address = \"0.0.0.0:9091\"%address = \"0.0.0.0:10091\"%" $HOME/.secretd/config/app.toml

6. Create Service File

Note that this service file has two environment variables that are set, as well as a --home directory. These will be unique to your user.

sudo tee /etc/systemd/system/secretd.service > /dev/null <<EOF
[Unit]
Description=Secret Node service
After=network.target

[Service]
Type=simple
Environment=SCRT_ENCLAVE_DIR=/home/secret/lib
Environment=SCRT_SGX_STORAGE=/home/secret/.secretd/.sgx_secrets
WorkingDirectory=/home/secret
ExecStart=/usr/local/bin/secretd start --home /home/secret/.secretd
User=secret
Restart=on-failure
StartLimitInterval=0
RestartSec=3
LimitNOFILE=65535
LimitMEMLOCK=209715200

[Install]
WantedBy=multi-user.target
EOF

7. Continue Setting Up a Full Node

From here, you can return to step 9 of setting up a full node. Note that the service file name is different. The following is what the system file commands would look like.

sudo systemctl daemon-reload && sudo systemctl enable secretd && \
sudo systemctl restart secretd && sudo journalctl -u secretd -f -o cat

Node Loadbalancing using Nginx

To provide endpoints to users, you can use Nginx. It is a powerful tool for providing a loadbalanced endpoint that distributes the load across multiple nodes.

Setup Nginx

SSH into Your Machine: Use an SSH client to log into your server.

Update System Packages

Frist, ensure that all packages are up to date. This can prevent security vulnerabilities.

Install Monitoring and Web Server Tools

Install Glances: An advanced system monitor for Linux.

Install Nginx: A high-performance web server and a reverse proxy, often used for load balancing.

Configure UFW Firewall for Nginx: Make sure Nginx can receive HTTP and HTTPS traffic.

Set Up Nginx Configuration

Remove Existing Nginx Configuration:

Edit New Configuration: Use a text editor like nano to create a new configuration file.

Modify RPC/LCD/gRPC server entries and domain names as required.
Save and exit the editor (CTRL + X, then Y to confirm, and Enter to save).

Test Nginx Configuration: Ensures your syntax is correct.

Enable and Start Nginx: This will make sure Nginx starts on boot and starts running immediately.

Secure with SSL/HTTPS

Install Certbot: This tool automates obtaining free SSL/TLS certificates from Let's Encrypt.

Obtain the SSL Certificate for your domain:

Finally, you can add a cronjob to crontab to enable auto-newing of the certificates:

Example Nginx config

Here is an example nginx.conf for Loadbalancing on Nginx:

user  www-data;
worker_processes  auto;#Relevant to handle as many connections as the server config could, is correlated  to number of cores in the CPU
                    #https://nginx.org/en/docs/ngx_core_module.html#worker_processes
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;#Relevant to handle as many connections as the server config could
                              #clients = worker_processes * worker_connections
}                             #https://nginx.org/en/docs/ngx_core_module.html#worker_connections

http {

    large_client_header_buffers 4 128k; 
    server_names_hash_bucket_size 128;

map $request_uri $short_uri {
    "~^(.{0,50})" $1;
    default       $request_uri;
}


    map $remote_addr $anonymized_addr {
                ~(?P<ip>\d+\.\d+)\.\d+\.    $ip.0.0;
                ~(?P<ip>[^:]+:[^:]+):       $ip::;
                # IP addresses to not anonymize (such as your server)
                127.0.0.1                   $remote_addr;
                ::1                         $remote_addr;
                #w.x.y.z                    $remote_addr;
                #a::c:d::e:f                $remote_addr;
                default                     0.0.0.0;
    }
    
    log_format combined_log '$anonymized_addr - $remote_user [$time_local] '
                        '"$short_uri" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';
                       
    access_log  /var/log/nginx/access.log  combined_log;

    limit_req_zone $binary_remote_addr zone=mylimit:1M rate=70r/s;
    limit_req zone=mylimit burst=9000 nodelay;
    #WSS compatibility config
    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    #Upstream group
    upstream rpc_stream {
        least_conn; #Redirect requests to the server with least number of active connections
        server  XXX.XXX.XXX.XXX:26657 max_fails=1000 fail_timeout=30s; 
	server  XXX.XXX.XXX.XXX:26657 max_fails=1000 fail_timeout=30s;
	server  XXX.XXX.XXX.XXX:26657 max_fails=1000 fail_timeout=30s;

    }

    upstream grpc_stream {
        least_conn; #Redirect requests to the server with least number of active connections
        server  XXX.XXX.XXX.XXX:9091 max_fails=1000 fail_timeout=30s; 
	server  XXX.XXX.XXX.XXX:9091 max_fails=1000 fail_timeout=30s;
        server  XXX.XXX.XXX.XXX:9091 max_fails=1000 fail_timeout=30s;

    }

    upstream lcd_stream {
        least_conn; #Redirect requests to the server with least number of active connections
        server  XXX.XXX.XXX.XXX:1317 max_fails=1000 fail_timeout=30s; 
	server  XXX.XXX.XXX.XXX:1317 max_fails=1000 fail_timeout=30s;
        server  XXX.XXX.XXX.XXX:1317 max_fails=1000 fail_timeout=30s;
    }

    
   server {
        listen 80;
        server_name rpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://rpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }


   server {
        listen 26657;
        server_name rpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://rpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }



    server {
        listen 80;
        server_name grpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://grpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }

    server {
        listen 9091;
        server_name grpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://grpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }

server {
        listen 1317;
        server_name lcd.YOUR_URL;
        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }


	    proxy_hide_header 'Access-Control-Allow-Origin';
	    proxy_hide_header 'Access-Control-Allow-Credentials';
	    proxy_hide_header 'Access-Control-Allow-Headers';
	    proxy_hide_header 'Access-Control-Allow-Methods'; 

            add_header 'Access-Control-Allow-Origin' * always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' * always;

            proxy_pass       http://lcd_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }

   
}

    server {
        listen 80;
        server_name lcd.YOUR_URL;
        location / {

            if ($request_method = 'OPTIONS') {
                return 204;
            }

	    proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods'; 

            add_header 'Access-Control-Allow-Origin' * always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' * always;

            proxy_pass       http://lcd_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }


    server {
        listen 127.0.0.1:80;
        server_name 127.0.0.1;
        location /nginx_status {
            stub_status on;
            allow 127.0.0.1;
            deny all;
        }
    }

}

Using Auto heal to improve cluster uptime for Nginx

Most noderunners have the common problem that stale nodes, meaning nodes that are stuck at a certain block height and will not continue keeping up the chain tip, provide horrible UX to users. To mitigate this, we can add some extra monitoring tools to dynamically add or remove nodes from the cluster.

The provided software gives you an entry level solution for this. As everything is written in Python, you can adjust this to your API setup.

To install, please do the following:

Prerequisites

Ensure you have Python installed. If not, download and install Python from python.org. You'll also need pip, Python's package manager, to install required libraries. If you're using a Linux-based system, ensure you have NGINX installed and properly configured.

Since we edit the nginx config directly, we need to give pyhton3 sudo rights.

Be aware when you follow this tutorial.

Clone Repository

To clone the AutoHealBot repository, use the following steps:

Install Git: Ensure Git is installed on your system. If not, install it from git-scm.com.

Clone the Repository: Open a terminal and run:

git clone https://github.com/SecretSaturn/AutoHealBot.git

Navigate to the Directory: After cloning, change to the repository directory:
```
cd AutoHealBot
```

This clones the entire repository to your local machine, allowing you to access all files and resources. To proceed with the tutorial, follow additional setup instructions provided in the repository's README or other documentation.

Install Required Libraries for Python

When installing, make sure to install this under root with sudo, otherwise the script will later not find the libraries later on.

sudo pip install aiohttp python-dotenv dateutil

Setup the enviroment file

To configure your environment variables, copy over the .env.example in the repository.

# Path to your Nginx Config
NGINX_CONFIG_PATH = '/path/to/your/nginx.conf'

# Base rate limit in requests per second
BASE_RATE = 8

# Rate increase per healthy node
NODE_MULTIPLIER = 1

# Port numbers for RPC, gRPC, and LCD
RPC_PORT = 26657
GRPC_PORT = 9091
LCD_PORT = 1317

# Path to the file containing node URLs
FILE_PATH = '/path/to/your/file.txt'

# Time in seconds to determine if a node has fallen behind
TIME_BEFORE_FALLEN_BEHIND = 18

# Time interval in seconds for updating the status
UPDATE_TIME = 30

Replace the placeholders with actual values:

NGINX_CONFIG_PATH: The path to your NGINX configuration file.
BASE_RATE and NODE_MULTIPLIER: Adjust as needed.
RPC_PORT, GRPC_PORT, LCD_PORT: Set to your specific ports.
FILE_PATH: Path to the text file with node URLs.
TIME_BEFORE_FALLEN_BEHIND: Maximum allowed time before a node is considered unhealthy.
UPDATE_TIME: Time between health checks.

Create a File with Node URLs

Create a text file with the node URLs. For example, create nodes.txt with one URL per line, make sure to include the RPC port to each node here as well:

192.168.0.1:26657
192.168.0.2:26657
192.168.0.2:26657

In the AutoHealBot script, "upstream blocks" refer to sections in the NGINX configuration that specify which backend servers handle different types of traffic. This setup divides the backend nodes into separate streams: RPC, gRPC, and LCD. The script checks the health of these nodes and updates the corresponding upstream blocks to reflect the healthy nodes for each stream. It ensures that traffic is routed to servers that are online and functional.

As a reference, the upstream blocks are defined as:

upstream_blocks = { 
"rpc_stream": healthy_nodes_rpc, 
"grpc_stream": healthy_nodes_grpc, 
"lcd_stream": healthy_nodes_lcd, 
}

Execute the Script

Run the script to start the asynchronous health checks and NGINX updates:

sudo python3 main.py

Troubleshooting

Environment Variables Not Loaded: Ensure your .env file is in the same directory as the script or specify its path explicitly with dotenv_path.
NGINX Not Reloading: Check if you have the necessary permissions to reload NGINX and ensure systemctl or other command-line utilities are in your PATH.

With this setup, the script will run asynchronously, periodically checking node health, updating the NGINX configuration, and reloading the NGINX service as needed.

Using Auto heal to improve cluster uptime for Nginx

The provided software gives you an entry level solution for this. As everything is written in Python, you can adjust this to your API setup.

To install, please do the following:

Prerequisites

Since we edit the nginx config directly, we need to give pyhton3 sudo rights.

Be aware when you follow this tutorial.

Clone Repository

To clone the AutoHealBot repository, use the following steps:

Install Git: Ensure Git is installed on your system. If not, install it from git-scm.com.

Clone the Repository: Open a terminal and run:

git clone https://github.com/SecretSaturn/AutoHealBot.git

Navigate to the Directory: After cloning, change to the repository directory:
```
cd AutoHealBot
```

Install Required Libraries for Python

When installing, make sure to install this under root with sudo, otherwise the script will later not find the libraries later on.

sudo pip install aiohttp python-dotenv dateutil

Setup the enviroment file

To configure your environment variables, copy over the .env.example in the repository.

# Path to your Nginx Config
NGINX_CONFIG_PATH = '/path/to/your/nginx.conf'

# Base rate limit in requests per second
BASE_RATE = 8

# Rate increase per healthy node
NODE_MULTIPLIER = 1

# Port numbers for RPC, gRPC, and LCD
RPC_PORT = 26657
GRPC_PORT = 9091
LCD_PORT = 1317

# Path to the file containing node URLs
FILE_PATH = '/path/to/your/file.txt'

# Time in seconds to determine if a node has fallen behind
TIME_BEFORE_FALLEN_BEHIND = 18

# Time interval in seconds for updating the status
UPDATE_TIME = 30

Replace the placeholders with actual values:

NGINX_CONFIG_PATH: The path to your NGINX configuration file.
BASE_RATE and NODE_MULTIPLIER: Adjust as needed.
RPC_PORT, GRPC_PORT, LCD_PORT: Set to your specific ports.
FILE_PATH: Path to the text file with node URLs.
TIME_BEFORE_FALLEN_BEHIND: Maximum allowed time before a node is considered unhealthy.
UPDATE_TIME: Time between health checks.

Create a File with Node URLs

Create a text file with the node URLs. For example, create nodes.txt with one URL per line, make sure to include the RPC port to each node here as well:

192.168.0.1:26657
192.168.0.2:26657
192.168.0.2:26657

As a reference, the upstream blocks are defined as:

upstream_blocks = { 
"rpc_stream": healthy_nodes_rpc, 
"grpc_stream": healthy_nodes_grpc, 
"lcd_stream": healthy_nodes_lcd, 
}

Execute the Script

Run the script to start the asynchronous health checks and NGINX updates:

sudo python3 main.py

Troubleshooting

Environment Variables Not Loaded: Ensure your .env file is in the same directory as the script or specify its path explicitly with dotenv_path.
NGINX Not Reloading: Check if you have the necessary permissions to reload NGINX and ensure systemctl or other command-line utilities are in your PATH.

With this setup, the script will run asynchronously, periodically checking node health, updating the NGINX configuration, and reloading the NGINX service as needed.

Example Nginx config

Here is an example nginx.conf for Loadbalancing on Nginx:

user  www-data;
worker_processes  auto;#Relevant to handle as many connections as the server config could, is correlated  to number of cores in the CPU
                    #https://nginx.org/en/docs/ngx_core_module.html#worker_processes
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;#Relevant to handle as many connections as the server config could
                              #clients = worker_processes * worker_connections
}                             #https://nginx.org/en/docs/ngx_core_module.html#worker_connections

http {

    large_client_header_buffers 4 128k; 
    server_names_hash_bucket_size 128;

map $request_uri $short_uri {
    "~^(.{0,50})" $1;
    default       $request_uri;
}


    map $remote_addr $anonymized_addr {
                ~(?P<ip>\d+\.\d+)\.\d+\.    $ip.0.0;
                ~(?P<ip>[^:]+:[^:]+):       $ip::;
                # IP addresses to not anonymize (such as your server)
                127.0.0.1                   $remote_addr;
                ::1                         $remote_addr;
                #w.x.y.z                    $remote_addr;
                #a::c:d::e:f                $remote_addr;
                default                     0.0.0.0;
    }
    
    log_format combined_log '$anonymized_addr - $remote_user [$time_local] '
                        '"$short_uri" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';
                       
    access_log  /var/log/nginx/access.log  combined_log;

    limit_req_zone $binary_remote_addr zone=mylimit:1M rate=70r/s;
    limit_req zone=mylimit burst=9000 nodelay;
    #WSS compatibility config
    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    #Upstream group
    upstream rpc_stream {
        least_conn; #Redirect requests to the server with least number of active connections
        server  XXX.XXX.XXX.XXX:26657 max_fails=1000 fail_timeout=30s; 
	server  XXX.XXX.XXX.XXX:26657 max_fails=1000 fail_timeout=30s;
	server  XXX.XXX.XXX.XXX:26657 max_fails=1000 fail_timeout=30s;

    }

    upstream grpc_stream {
        least_conn; #Redirect requests to the server with least number of active connections
        server  XXX.XXX.XXX.XXX:9091 max_fails=1000 fail_timeout=30s; 
	server  XXX.XXX.XXX.XXX:9091 max_fails=1000 fail_timeout=30s;
        server  XXX.XXX.XXX.XXX:9091 max_fails=1000 fail_timeout=30s;

    }

    upstream lcd_stream {
        least_conn; #Redirect requests to the server with least number of active connections
        server  XXX.XXX.XXX.XXX:1317 max_fails=1000 fail_timeout=30s; 
	server  XXX.XXX.XXX.XXX:1317 max_fails=1000 fail_timeout=30s;
        server  XXX.XXX.XXX.XXX:1317 max_fails=1000 fail_timeout=30s;
    }

    
   server {
        listen 80;
        server_name rpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://rpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }


   server {
        listen 26657;
        server_name rpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://rpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }



    server {
        listen 80;
        server_name grpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://grpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }

    server {
        listen 9091;
        server_name grpc.YOUR_URL;

        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }

            proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods';
            
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
            
            proxy_pass       http://grpc_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }

server {
        listen 1317;
        server_name lcd.YOUR_URL;
        location / {
            if ($request_method = 'OPTIONS') {
                return 204;
            }


	    proxy_hide_header 'Access-Control-Allow-Origin';
	    proxy_hide_header 'Access-Control-Allow-Credentials';
	    proxy_hide_header 'Access-Control-Allow-Headers';
	    proxy_hide_header 'Access-Control-Allow-Methods'; 

            add_header 'Access-Control-Allow-Origin' * always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' * always;

            proxy_pass       http://lcd_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }

   
}

    server {
        listen 80;
        server_name lcd.YOUR_URL;
        location / {

            if ($request_method = 'OPTIONS') {
                return 204;
            }

	    proxy_hide_header 'Access-Control-Allow-Origin';
            proxy_hide_header 'Access-Control-Allow-Credentials';
            proxy_hide_header 'Access-Control-Allow-Headers';
            proxy_hide_header 'Access-Control-Allow-Methods'; 

            add_header 'Access-Control-Allow-Origin' * always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Allow-Headers' * always;
            add_header 'Access-Control-Allow-Methods' * always;

            proxy_pass       http://lcd_stream;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade; #WSS compatibility config
            proxy_set_header Connection $connection_upgrade; #WSS compatibility config
        }
    }


    server {
        listen 127.0.0.1:80;
        server_name 127.0.0.1;
        location /nginx_status {
            stub_status on;
            allow 127.0.0.1;
            deny all;
        }
    }

}

Running Multiple Nodes on the Same Server

It's possible to run multiple Secret Nodes on the same Secret-compatible server, and it is fairly easy to do so.

Important Notes

There are 2 important things that must be done for each node:

A unique system file is necessary for each node
A unique sgx_secrets path is necessary for each node
All Secret Nodes should have their own user to simplify
It's easiest to do this with auto-register, but it's possible manual as well
Each node must be registered

Setup

This process assumes you already have a full node running. If you do not, proceed by Setting Up a Full Node, then returning.

1. Create a User

This isn't necessary, but will help with keeping nodes organized. From here on, the assumption is the username is secret, but it can be anything of your choosing.

adduser secret
usermod secret -aG sudo

2. Verify secretd Access

This will make it so you don't need to install secretd multiple times, and therefore, can upgrade all nodes at the same time.

secretd status

3. Begin Setting up a Node

On the new user, execute steps 1 and 2 of Setting Up a Full Node. You should now have a .secretd directory on the new user, and the correct genesis file.

4. Register the Node

The variables SCRT_ENCLAVE_DIR and SCRT_SGX_STORAGE are going to need to be custom for each user/node. These variables are NOT the same as the ones in step 3 of setting up a full node.

export SCRT_ENCLAVE_DIR=~/lib
export SCRT_SGX_STORAGE=~/.secretd/.sgx_secrets
secretd auto-register

5. Change Ports

In order for these nodes to work in tandem, they cannot use the same ports. I recommend this tool to help automate changing them.

Which will then create a command that looks like this:

sed -i.bak -e "s%^proxy_app = \"tcp://127.0.0.1:26658\"%proxy_app = \"tcp://127.0.0.1:10658\"%; s%^laddr = \"tcp://127.0.0.1:26657\"%laddr = \"tcp://127.0.0.1:10657\"%; s%^pprof_laddr = \"localhost:6060\"%pprof_laddr = \"localhost:10060\"%; s%^laddr = \"tcp://0.0.0.0:26656\"%laddr = \"tcp://0.0.0.0:10656\"%; s%^prometheus_listen_addr = \":26660\"%prometheus_listen_addr = \":10660\"%" $HOME/.secretd/config/config.toml && \
sed -i.bak -e "s%^address = \"tcp://0.0.0.0:1317\"%address = \"tcp://0.0.0.0:10317\"%; s%^address = \":8080\"%address = \":10080\"%; s%^address = \"0.0.0.0:9090\"%address = \"0.0.0.0:10090\"%; s%^address = \"0.0.0.0:9091\"%address = \"0.0.0.0:10091\"%" $HOME/.secretd/config/app.toml

6. Create Service File

Note that this service file has two environment variables that are set, as well as a --home directory. These will be unique to your user.

sudo tee /etc/systemd/system/secretd.service > /dev/null <<EOF
[Unit]
Description=Secret Node service
After=network.target

[Service]
Type=simple
Environment=SCRT_ENCLAVE_DIR=/home/secret/lib
Environment=SCRT_SGX_STORAGE=/home/secret/.secretd/.sgx_secrets
WorkingDirectory=/home/secret
ExecStart=/usr/local/bin/secretd start --home /home/secret/.secretd
User=secret
Restart=on-failure
StartLimitInterval=0
RestartSec=3
LimitNOFILE=65535
LimitMEMLOCK=209715200

[Install]
WantedBy=multi-user.target
EOF

7. Continue Setting Up a Full Node

At this point, all unique behavior for additional nodes is complete!

From here, you can return to step 9 of setting up a full node. Note that the service file name is different. The following is what the system file commands would look like.

sudo systemctl daemon-reload && sudo systemctl enable secretd && \
sudo systemctl restart secretd && sudo journalctl -u secretd -f -o cat

API Noderunning

Running Multiple Nodes on the Same Server

Important Notes

Setup

1. Create a User

2. Verify secretd Access

3. Begin Setting up a Node

4. Register the Node

5. Change Ports

6. Create Service File

7. Continue Setting Up a Full Node

Node Loadbalancing using Nginx

Setup Nginx

Login and User Setup

Update System Packages

Install Monitoring and Web Server Tools

Set Up Nginx Configuration

Secure with SSL/HTTPS

Example Nginx config

Using Auto heal to improve cluster uptime for Nginx

Prerequisites

Clone Repository

Install Required Libraries for Python

Setup the enviroment file

Create a File with Node URLs

Execute the Script

Using Auto heal to improve cluster uptime for Nginx

Prerequisites

Clone Repository

Install Required Libraries for Python

Setup the enviroment file

Create a File with Node URLs

Execute the Script

Example Nginx config

Running Multiple Nodes on the Same Server

Important Notes

Setup

1. Create a User

2. Verify secretd Access

3. Begin Setting up a Node

4. Register the Node

5. Change Ports

6. Create Service File

7. Continue Setting Up a Full Node

Node Loadbalancing using Nginx

Setup Nginx

Login and User Setup

Update System Packages

Install Monitoring and Web Server Tools

Set Up Nginx Configuration

Secure with SSL/HTTPS

API Noderunning