Setting up a Grafana based monitor for your Hypernode node

Blockpi, for those who are not familiar with it, is a decentralized RPC network with endless upscaling possibilities.

As you can see on https://blockpi.io it already offers RPC for Near, Polkadot, Matic, Solana, Kusama, Flow, Hecco, KCC. In addition to the services themselves, leading the requester to the nearest (geographically meaning) L2 RPC provider (That’s us!) is included, thus reducing latency and increasing QoS.

In this tutorial we will explain how to make use of the metrics offered by the Hypernode itself, on our case pointing to our own Klaytn RPC node, to make a Grafana Monitor and, based on it, trigger some alerts as needed if our Hypernode decides to go bananas…

I concentrated myself on the Hypernode specific metrics, but of course, as it might be seen on the second image, a bunch of general server metrics can be added as well, showing the overall health of our Hypernode node.

First of all, it is good to know that Hypernode spins up a prometheus exporter on http://ip:8899/metrics and that is what we are going to take advantage from in our setup.

Part 1: Setting up Grafana

Step 1: Set up the new server

First up you need to set up a new server to run your Hypernode node dashboard. You can set this up to run on your Hypernode server but it is NOT recommended.

Create new AWS EC2 instance

Start by creating for example a new AWS EC2 instance (https://aws.amazon.com/ec2/):

• Amazon Machine Image (AMI): Ubuntu Server 18.04 LTS (HVM), SSD Volume Type
• Instance Type: t3.micro

In this example we will use a t3.micro instance type but it likely work well on a server with less specifications.

Configure Security Groups

Add the following Custom TCP rules to allow inbound traffic to Grafana and Prometheus:

• Grafana
• Type: Custom TCP Rule
• Port Range: 3000
• Description: Grafana
• Prometheus
• Type: Custom TCP Rule
• Port Range: 9090
• Description: Prometheus

Step 2: Installing Prometheus on the new server

Next up you need to install Prometheus on your new server. Prometheus will collect all the real time metrics from your Hypernode machine and store them in a time series database.

Connect to your new AWS instance and follow the following guide to install Prometheus:

https://computingforgeeks.com/install-prometheus-server-on-debian-ubuntu-linux/)

If everything worked correctly you should now be able to see Prometheus running by going to http://IP_ADDRESS:9090

(replace IP_ADDRESS with the public IP address of your prometheus server)

Step 3: Install node_exporter on the Hypernode node

As well as the Hypernode specific metrics we also want to capture the server metrics so we can see how the server is performing (eg. RAM usage, CPU, etc.). To enable this you need to install the node_exporter.

Important note: If you want to capture metrics from multiple Hypernode servers where you have access to, then you will need to repeat this step on each node.

Create Prometheus system user / group

We’ll create a dedicated Prometheus system user and group. The -r or –system option is used for this purpose.

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

Install node_exporter

Download node_exporter archive.

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest \
| grep browser_download_url \
| grep linux-amd64 \
| cut -d '"' -f 4 \
| wget -qi -

Extract downloaded file and move the binary file to /usr/local/bin.

tar -xvf node_exporter*.tar.gz
cd  node_exporter*/
sudo cp node_exporter /usr/local/bin

Confirm installation.

node_exporter --version

*node_exporter, version 0.18.1 (branch: HEAD, revision: 3db77732e925c08f675d7404a8c46466b2ece83e)

build user: root@b50852a1acba

build date: 20190604–16:41:18

go version: go1.12.5*

Create node_exporter service.

sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter[Install]
WantedBy=default.target
EOF

Reload systemd and start the service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Confirm status:

systemctl status node_exporter.service

● node_exporter.service — Node Exporter

Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: enabled)

Active: active (running) since Wed 2019–08–21 23:41:11 CEST; 8s ago

Main PID: 22879 (node_exporter)

Tasks: 6 (limit: 4585)

Memory: 6.6M

CGroup: /system.slice/node_exporter.service

└─22879 /usr/local/bin/node_exporter

Step 4: Configure Prometheus to get Hypernode metrics

Ok, now you have Prometheus and the node exporter installed you need to set it up to get the metrics from the Hypernode node.

Configure Prometheus to retrieve metrics from the Hypernode server

Now you need to configure Prometheus to retrieve the Hypernode metrics.

On the prometheus server open the config file:

sudo nano /etc/prometheus/prometheus.yml

Update ‘job_name’ to blockpi_hypernode and replace 'targets' with the IP address of your hypernode server:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'general'    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.    static_configs:
    - targets: ['IP_ADDRESS:9100']
      labels:
        instance: 'NODE_NAME'
  - job_name: 'blockpi_hypernode'
     metrics_path: /metrics     static_configs:
    - targets: ['IP_ADDRESS:8899']
      labels:
        instance: 'NODE_NAME'

f you want to capture metrics from multiple Hypernode nodes then you can add additional targets accordingly.

Now restart Prometheus:

sudo systemctl restart prometheus

If Prometheus is now working as expected we should be able to view the metrics collected from our Hypernode server.

Open the Prometheus web client again: http://IP_ADDRESS:9090/

replace IP_ADDRESS with the IP address of your Prometheus server

Enter a valid Hypernode metrics such as grpc_server_msg_received_total{grpc_method=”Ping”,grpc_service=”x.blockpi.RelayService”,grpc_type=”unary”} and press execute.

If it’s working as expected you should see the data for the metric selected similar to below:

Step 5: Installing Grafana

Ok now we have all the metrics we want to visualise all them on a fancy dashboard that looks gorgeus :) Grafana is just what we need for this.

Use the following guide to install Grafana on the same server as Prometheus:

https://computingforgeeks.com/how-to-install-grafana-on-ubuntu-debian/

If everything worked correctly you should now be able to see Grafana running by going to http://IP_ADDRESS:3000

(replace IP_ADDRESS with the public IP address of your prometheus server)

Step 6: Connect Grafana to Prometheus Data Source

Now to connect Grafana to our data.

Click on the ‘Configuration’ menu option on the left hand side and then ‘Data Sources’ and ‘Add Data Source’.

In the configuration settings for the data source set the following and then click the ‘Save and Test’ button.

• Name: Prometheus Hypernode
• URL: http://localhost:9090

Step 7: Importing my sample Hypernode Node Dashboard

To import the sample Hypernode Node Dashboard click on the ‘Create’ menu on the left and select ‘Import’.

In the ‘Import via panel json’ section paste the following JSON:

https://bloclick.com/hypernode.json

You should now see your new Hypernode Node Dashboard!

Part 2: Setting up Grafana Alerts in Discord

If you would like to receive alerts in Discord when your Hypernode Node is down or not replying to requests then continue on!

Select the ‘+’ button in the bottom left of Discord to create a new Discord server, then select ‘Create my own’ and then ‘For me and my friends’.

Give the server a name and click ‘Create’.

Step 2: Set up the webhook integration

Select the ‘edit channel’ button next to the channel name and then select the ‘Integrations’ menu option on the left and select ‘Create Webhook’.

Give the webhook a name and select ‘Copy Webhook URL’ and save.

Step 3: Set up the Discord Notification Channel in Grafana

Open up your Grafana dashboard (http://IP_ADDRESS:3000/) and select ‘Alerts’ and then ‘Notification Channels’.

Select ‘Add Channel’ to add the new notification Channel, give it a name and then select ‘Discord’ as the type, finally paste in the Webhook URL you copied earlier. Select ‘Test’ and you should receive a notification in Discord if everything is working as it should be.

Then a test message is displayed on Discord:

Step 4: Setting up the Alert

Now open the dashboard and go to the panel where you want to set up an alert, select the header and then the ‘Edit’ option. Then, inside of the Alert section, as shown in the image, set up the alert according to your needs:

In my case I went for the sum of GRPC + Data errors > 0

If that triggers to many alerts I might need some tweaking.

So, that’s it. Hope you get it to work too! :)