> ## Documentation Index
> Fetch the complete documentation index at: https://validators.endaoment.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Validator SOP Part 4: Monitoring and Dashboard Creation

## Table of Contents

* [Abstract](/validators/validator-sop-part4-monitoring#abstract)
* [Step 1 Adjust Firewall](/validators/validator-sop-part4-monitoring#step-1-adjust-firewall)
* [Step 2 Install Prerequisite Software](/validators/validator-sop-part4-monitoring#step-2-install-prerequisite-software)
* [Step 3 Install Reporting Software](/validators/validator-sop-part4-monitoring#step-3-install-reporting-software)
  * [Prometheus](/validators/validator-sop-part4-monitoring#prometheus)
  * [Node Exporter](/validators/validator-sop-part4-monitoring#node-exporter)
  * [JSON Exporter](/validators/validator-sop-part4-monitoring#json-exporter)
  * [Ethereum Metrics Exporter](/validators/validator-sop-part4-monitoring#ethereum-metrics-exporter)
  * [Grafana](/validators/validator-sop-part4-monitoring#grafana)
* [Step 4 Configure Ethereum Clients](/validators/validator-sop-part4-monitoring#step-4-configure-ethereum-clients)
* [Step 5 Configure Reporting Software](/validators/validator-sop-part4-monitoring#step-5-configure-reporting-software)
  * [Configure JSON Exporter](/validators/validator-sop-part4-monitoring#configure-json-exporter)
  * [Configure Prometheus Rules](/validators/validator-sop-part4-monitoring#configure-prometheus-rules)
  * [Configure Ethereum Metrics Exporter](/validators/validator-sop-part4-monitoring#configure-ethereum-metrics-exporter)
* [Step 6 Configure Prometheus Jobs and Targets](/validators/validator-sop-part4-monitoring#step-6-configure-prometheus-jobs-and-targets)
  * [Prometheus Jobs](/validators/validator-sop-part4-monitoring#prometheus-jobs)
  * [Base Targets](/validators/validator-sop-part4-monitoring#base-targets)
  * [Validator Public Key Targets](/validators/validator-sop-part4-monitoring#validator-public-key-targets)
  * [Restart All Services](/validators/validator-sop-part4-monitoring#restart-all-services)
* [Step 7 Set up Dashboard](/validators/validator-sop-part4-monitoring#step-7-set-up-dashboard)

## Abstract

This document will detail the process for setting up a Grafana monitoring dashboard for an Ethereum validator. This guide assumes you followed [Validator SOP Part 1: Set Up](/validators/validator-sop-part1-setup). Please note that dashboards are highly customizable, and as such small changes can be made to either the reporting software or the dashboard itself to accommodate specific use cases. This guide is written with the initial Endaoment node set up in mind, a node with five validators utilizing Reth/Prysm. While the majority of things covered by this guide do not touch the underlying services that run the validator, caution should be exercised as there are some changes that must be made to validator/beacon-chain service files.

### Step 1 Adjust Firewall

Before we can get started, we need to open ports in the UFW firewall to allow Prometheus and Grafana listening ports:

```
sudo ufw allow 3000/tcp
sudo ufw allow 9090/tcp
```

Next, check to ensure the rules were properly applied:

```
sudo ufw status numbered
```

### Step 2 Install Prerequisite Software

This guide will utilize the `make` utility as well as `go`, and additionally relies on the Stake Local GitHub Repository. We'll get that all set up before we start installing the reporting software.

First, let's run a quick update, ensure we can use APT via HTTPS, and ensure we have common utilities like wget installed, and ensure they are automatically updated:

```
sudo apt-get update
sudo apt-get install apt-transport-https
sudo apt-get install software-properties-common wget
sudo apt-mark auto apt-transport-https
sudo apt-mark auto software-properties-common
sudo apt-mark auto wget
```

Install make:

```
sudo apt-get install make
```

Check to ensure it was properly installed:

```
make --version
```

Install go (version 1.23+ required for ethereum-metrics-exporter):

```
cd
wget https://go.dev/dl/go1.23.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.23.2.linux-amd64.tar.gz
sudo ln -s /usr/local/go/bin/go /usr/bin/go
```

Check to ensure it was properly installed:

```
go version
```

Clean up installation files:

```
rm go1.23.2.linux-amd64.tar.gz
```

### Step 3 Install Reporting Software

We'll now install Grafana, Prometheus, Node\_Exporter and JSON\_exporter software, that will all work together to generate our dashboard.

#### Prometheus

Download the latest release of Prometheus by going [here](https://prometheus.io/download/) and copying the download link for the latest linux-amd64.tar.gz file. Please be sure to replace the link below using the current version. Download the repository, extract the files, and move them to the /bin:

```
cd ~
curl -LO https://github.com/prometheus/prometheus/releases/download/v3.8.1/prometheus-3.8.1.linux-amd64.tar.gz
tar xvf prometheus-3.8.1.linux-amd64.tar.gz
sudo cp prometheus-3.8.1.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-3.8.1.linux-amd64/promtool /usr/local/bin/
```

Modify file names and clean up:

```
sudo cp -r prometheus-3.8.1.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-3.8.1.linux-amd64/console_libraries /etc/prometheus
rm prometheus-3.8.1.linux-amd64.tar.gz
rm -r prometheus-3.8.1.linux-amd64
```

Set up an account for the services to run under (note: this type of account cannot log into the server):

```
sudo useradd --no-create-home --shell /bin/false prometheus
```

Set up the directories for prometheus:

```
sudo mkdir -p /etc/prometheus /var/lib/prometheus
```

Create a configuration file for prometheus:

```
sudo nano /etc/prometheus/prometheus.yml
```

Copy and paste text into this configuration file so it matches [prometheus.yml](../.imgs/prometheus.yml).

Press CTRL + X then Y then ENTER to save and exit.

Set directory permissions for prometheus to modify the directories:

```
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus
```

Create a systemd service config file to configure the service:

```
sudo nano /etc/systemd/system/prometheus.service
```

Paste the following text into the configuration file:

```
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
```

Press CTRL + X then Y then ENTER to save and exit.

Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly:

```
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus
```

If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the Prometheus service).

Use the journal output to follow the progress or check for errors by running the following command:

```
sudo journalctl -fu prometheus | ccze
```

Finally, enable the service to automatically start on reboot:

```
sudo systemctl enable prometheus
```

**Troubleshooting Note:** If Prometheus fails to start with "address already in use" errors, you may have a snap-installed Prometheus conflicting with the manual installation. Check with `snap services` and if Prometheus is installed via snap, remove it with `sudo snap remove prometheus`.

#### Node Exporter

Download the latest release of Node Exporter by going [here](https://prometheus.io/download/#node%5Fexporter) and copying the download link for the latest linux-amd64.tar.gz file. Please be sure to replace the link below using the current version. Download the repository, extract the files, and move them to the /bin:

```
cd ~
wget https://github.com/prometheus/node_exporter/releases/download/v1.10.2/node_exporter-1.10.2.linux-amd64.tar.gz
tar xvf node_exporter-1.10.2.linux-amd64.tar.gz
sudo cp node_exporter-1.10.2.linux-amd64/node_exporter /usr/local/bin/
```

Clean up:

```
rm node_exporter-1.10.2.linux-amd64.tar.gz
rm -r node_exporter-1.10.2.linux-amd64
```

Set up an account for the services to run under (note: this type of account cannot log into the server):

```
sudo useradd --no-create-home --shell /bin/false node_exporter
```

Create a configuration file for node\_exporter:

```
sudo nano /etc/systemd/system/node_exporter.service
```

Copy and paste the following into this configuration file:

```
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
```

Press CTRL + X then Y then ENTER to save and exit.

Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly:

```
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl status node_exporter
```

If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the Node Exporter service).

Use the journal output to follow the progress or check for errors by running the following command:

```
sudo journalctl -fu node_exporter | ccze
```

Finally, enable the service to automatically start on reboot:

```
sudo systemctl enable node_exporter
```

**Note:** Some hardware configurations (particularly VMs or systems without thermal sensors) may trigger recurring `thermal_zone` collector errors in the logs. These warnings are harmless and can be safely ignored. If desired, you can disable this collector by adding `--no-collector.thermal_zone` to the `ExecStart` line in the service file.

#### JSON Exporter

Download the latest release of JSON Exporter by going [here](https://github.com/prometheus-community/json%5Fexporter/releases) and copying the download link for the latest linux-amd64.tar.gz file. Please be sure to replace the link below using the current version. Download the repository and move them to /git:

```
mkdir ~/git
cd git
git clone --depth 1 --branch v0.7.0 https://github.com/prometheus-community/json_exporter.git
cd ~
```

Build json\_exporter from the source code you just downloaded:

```
cd git/json_exporter
make build
```

Copy the executable file to the /bin, and check to ensure it installed correctly (should show the current version):

```
sudo cp json_exporter /usr/local/bin/
/usr/local/bin/json_exporter --version --help
```

Clean up time:

```
cd ~
rm -rf ~/git/json_exporter
```

Set up an account for the services to run under (note: this type of account cannot log into the server):

```
sudo useradd --no-create-home --shell /bin/false json_exporter
```

Create a directory and the configuration file for json\_exporter:

```
sudo mkdir /etc/json_exporter
sudo nano /etc/systemd/system/json_exporter.service
sudo chown -R json_exporter:json_exporter /etc/json_exporter
```

Copy and paste the following into this configuration file:

```
[Unit]
Description=JSON Exporter

[Service]
Type=simple
Restart=always
RestartSec=5
User=json_exporter
ExecStart=/usr/local/bin/json_exporter --config.file /etc/json_exporter/json_exporter.yml

[Install]
WantedBy=multi-user.target
```

Press CTRL + X then Y then ENTER to save and exit.

Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly:

```
sudo systemctl daemon-reload
sudo systemctl start json_exporter
sudo systemctl status json_exporter
```

It may not say "active (running)" in green text, as `'/etc/json_exporter/json_exporter.yml' does not exist`, this is expected. We will configure the YML file in step 5 below.

Use the journal output to follow the progress or check for errors by running the following command:

```
sudo journalctl -fu json_exporter | ccze
```

Running this command will confirm the above, as you'll see the error.

Finally, enable the service to automatically start on reboot:

```
sudo systemctl enable json_exporter
```

#### Ethereum Metrics Exporter

Clone and build the latest version of the Ethereum Metrics Exporter from source:

```bash theme={null}
cd ~/git
git clone https://github.com/ethpandaops/ethereum-metrics-exporter.git
cd ethereum-metrics-exporter
go build -o ethereum-metrics-exporter .
```

Copy the executable to /usr/local/bin and verify it works:

```bash theme={null}
sudo cp ethereum-metrics-exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/ethereum-metrics-exporter
/usr/local/bin/ethereum-metrics-exporter --help
```

Clean up the build files:

```bash theme={null}
cd ~
rm -rf ~/git/ethereum-metrics-exporter
```

Set up an account for the services to run under (note: this type of account cannot log into the server):

```
sudo useradd --no-create-home --shell /bin/false eth-metrics
```

Create a directory and the configuration file for eth\_exporter:

```
sudo mkdir -p /var/local/lib/eth-metrics /etc/eth-metrics
sudo nano /etc/systemd/system/eth-metrics.service
```

Copy and paste the following into this configuration file:

```
[Unit]
Description=Ethereum Metrics Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=eth-metrics
Group=eth-metrics
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/ethereum-metrics-exporter \
                --config /etc/eth-metrics/eth-metrics.yml \
                --metrics-port 9095

[Install]
WantedBy=multi-user.target
```

Press CTRL + X then Y then ENTER to save and exit.

Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly:

```
sudo systemctl daemon-reload
sudo systemctl start eth-metrics
sudo systemctl status eth-metrics
```

It may not say "active (running)" in green text, as `'/etc/eth-metrics/eth-metrics.yml' does not exist`, this is expected. We will configure the YML file in step 5 below.

Use the journal output to follow the progress or check for errors by running the following command:

```
sudo journalctl -fu eth-metrics | ccze
```

Running this command will confirm the above, as you'll see the error.

Finally, enable the service to automatically start on reboot:

```
sudo systemctl enable eth-metrics
```

#### Grafana

Download the Grafana GPG key and add Grafana to the APT sources:

```
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
```

You may get a warning about apt-key being deprecated, it is ok to ignore this warning.

Refresh the apt cache:

```
sudo apt install grafana && sudo apt update
```

You may need to press `Y` to continue.

Install Grafana:

```
sudo apt install -y grafana
```

Start Grafana, then check the status to make sure it’s running correctly:

```
sudo systemctl start grafana-server
sudo systemctl status grafana-server
```

If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the Grafana service)

Use the journal output command to check for errors:

```
sudo journalctl -fu grafana-server | ccze
```

Press CTRL+ C to exit (will not affect the Grafana service).

Next, enable the Grafana service to automatically start on reboot:

```
sudo systemctl enable grafana-server
```

Now that you have Grafana up and running, you can navigate to `https://{yourserverIP}:3000` to access the Grafana interface. You should see a login screen. Enter `admin` for both the username and password. You'll be immediately prompted to change your password, which you should, and note on your variables document.

Time to configure the Grafana data source! Navigate to `https://{yourserverIP}:3000/datasources`and click on `Add data source`. Choose Prometheus, enter `http://localhost:9090` for the URL, and then click on Save and Test.

If you see `Successfully queried the Prometheus API`, you're all set!

### Step 4 Configure Ethereum Clients

Each consensus and execution client must have the appropriate metrics services and APIs enabled and accessible to the reporting software. As a reminder, the below instructions assume Prysm/Reth.

Adjust your Prysm Beacon service file:

```
sudo nano /etc/systemd/system/prysmbeacon.service
```

Add the following flag at the bottom of the `ExecStart` list, note that you may need to add a`\` at the end of the previous line, depending on how your file is formatted.

`--monitoring-host 0.0.0.0`

Press CTRL + X then Y then ENTER to save and exit.

Adjust your Prysm Validator service file:

```
sudo nano /etc/systemd/system/prysmvalidator.service
```

Add the following flag at the bottom of the `ExecStart` list, note that you may need to add a`\` at the end of the previous line, depending on how your file is formatted.

`--monitoring-host 0.0.0.0`

Press CTRL + X then Y then ENTER to save and exit.

Adjust your Reth service file:

```
sudo nano /etc/systemd/system/reth.service
```

Update the `ExecStart` section to add monitoring support. Change the `--http.api` line and add the `--metrics` flag:

```
ExecStart=/usr/local/bin/reth node \
  --chain mainnet \
  --minimal \
  --datadir /var/lib/reth \
  --authrpc.jwtsecret /var/lib/jwtsecret/jwt.hex \
  --authrpc.addr 127.0.0.1 \
  --authrpc.port 8551 \
  --port 30303 \
  --discovery.port 30303 \
  --enable-discv5-discovery \
  --discovery.v5.port 30305 \
  --max-outbound-peers 35 \
  --max-inbound-peers 35 \
  --http \
  --http.addr 127.0.0.1 \
  --http.api eth,net,web3,txpool \
  --metrics 127.0.0.1:9001
```

Press CTRL + X then Y then ENTER to save and exit.

Finally, restart everything you just adjusted. Please note that if the validator is attesting, you should first ensure there are no upcoming attestations or proposals, as there will be \~30 seconds of downtime.

```
sudo systemctl daemon-reload
sudo systemctl restart reth
sudo systemctl restart prysmbeacon
sudo systemctl restart prysmvalidator
```

You should then check the status of each of the services you just restarted to ensure that changing the configuration files haven't messed with anything:

```
sudo systemctl status reth
sudo systemctl status prysmbeacon
sudo systemctl status prysmvalidator
```

After reviewing each one, press CTRL + C to exit, which will trigger the next status to show.

### Step 5 Configure Reporting Software

With all software installed and your Ethereum clients configured, we can now configure all the software to talk to each other properly. Remember those errors from before? Now we fix.

#### Configure JSON Exporter

Edit the configuration file for json exporter:

```
sudo nano /etc/json_exporter/json_exporter.yml
```

Copy and paste the contents of [this file](../.imgs/json_exporter.yml) into that file. Then press CTRL + X followed by Y to save.

#### Configure Prometheus Rules

Edit the rules.d file for prometheus:

```
sudo mkdir -p /etc/prometheus/rules.d/
sudo nano /etc/prometheus/rules.d/stakelocal_rules.yml
```

Copy and paste the contents of [this file](../.imgs/stakelocal_rules.yml) into that file. Then press CTRL + X followed by Y to save.

#### Configure Ethereum Metrics Exporter

Edit the configuration file for eth-metrics:

```
sudo nano /etc/eth-metrics/eth-metrics.yml
```

Copy and paste the following into that file:

```
consensus:
  enabled: true
  url: "http://127.0.0.1:3500"
  name: "Prysm Beacon"
execution:
  enabled: true
  url: "http://127.0.0.1:8545"
  name: "Reth"
  modules:
    - "eth"
    - "net"
    - "web3"
    - "txpool"
pair:
  enabled: true
```

**Important:** Make sure the consensus URL uses `http://` (not `https://`) for local connections.

Then press CTRL + X followed by Y to save.

### Step 6 Configure Prometheus Jobs and Targets

A Prometheus `job` represents a set of rules for processing metrics from similar data sources/targets. A Prometheus target is a single data source from which Prometheus should collect and process data using the rules of a specific job. Prometheus jobs are collections of targets for which the same rules are applied. Here, we'll be setting up the Prometheus jobs that feed data to the dashboard.

#### Prometheus Jobs

Since we've already edited the Prometheus configuration file, we simply need to check that the rules are being read:

```
sudo promtool check config /etc/prometheus/prometheus.yml
```

While there will be a number of warnings (expected), you should see `SUCCESS: 46 rules found` at the bottom of the returned text.

#### Base Targets

Before we can do specific configuring, we will need to copy in an archive of all target configuration files. To save time, all 24 files come pre-edited as needed, you're welcome.

Create the folder to store things in:

```
sudo mkdir -p /etc/prometheus/files_sd
```

Download the stakelocal configuration files directly from the GitHub repository:

```bash theme={null}
cd ~
sudo apt install unzip
wget https://github.com/endaoment/endaoment-validator/archive/refs/heads/main.zip
unzip main.zip
```

Copy the stakelocal folder to the correct location:

```bash theme={null}
sudo mkdir -p /etc/prometheus/files_sd/stakelocal/
sudo cp -r endaoment-validator-main/.imgs/stakelocal/* /etc/prometheus/files_sd/stakelocal/
```

Clean up the downloaded files:

```bash theme={null}
rm -rf endaoment-validator-main main.zip
```

Finally, check to ensure things were copied correctly (you should see a folder called stakelocal):

```
ls /etc/prometheus/files_sd/
```

#### Validator Public Key Targets

Edit the validator configuration file to include a list of comma separated public keys of your validator(s):

```
sudo nano /etc/prometheus/files_sd/stakelocal/validators.yml
```

Copy and paste the following into that file:

```
    - targets: [ "{validators}" ]
      labels:
        network: 'Mainnet'
        host: 'Default Host'
        service: 'Prysm Beacon'
        group: 'Prysm/Reth'
        instance: '127.0.0.1:3500'
        explorer: 'beaconcha.in'
```

Make sure to replace `{validators}` with a comma separated list of your validator public keys or indices (discoverable via Beaconcha.in if unknown). **Important:** Each validator key must be enclosed in quotes, in the style of `-targets: [ "0x123...", "0x456...", ... "0xXYZ..." ]`.

If the consensus client being queried is not on the local host or uses a non-standard port, update those on the `instance` line.

Then press CTRL + X followed by Y to save.

#### Restart All Services

We just made a lot of changes to many reporting services and their configuration files, let's now restart them and check that they've started up correctly.

Restart all reporting services:

```
sudo systemctl daemon-reload
sudo systemctl restart prometheus grafana-server node_exporter json_exporter eth-metrics
```

Now, check the status of each of these services:

```
sudo systemctl status prometheus grafana-server node_exporter json_exporter eth-metrics
```

If you need additional detail, you can check the logs:

```
sudo journalctl -fu prometheus | ccze
sudo journalctl -fu grafana-server | ccze
sudo journalctl -fu node_exporter | ccze
sudo journalctl -fu json_exporter | ccze
sudo journalctl -fu eth-metrics | ccze
```

### Step 7 Set up Dashboard

Log into your Grafana instance `({YourIp}:3000)` using the password you set up earlier. In the menu, click on Dashboard > New > Import. Copy and paste [this file](../.imgs/grafana-dashboard.json) into the `Import via dashboard JSON model` area and click `Load`. On the next screen, give your dashboard a unique name, feel free to change the UUID, and click `Import`.

⚠️ **You're going to see a lot of errors, that is expected!** ⚠️

These errors are because we need to update the data source for all panels you just imported. Head back to the data sources menu (Home > Connection > Data sources) and click on Prometheus - ensure the connection is set to [http://localhost:9090](http://localhost:9090) with `No Authentication`, then scroll down and click `Save and Test`.

Next, click on Dashboards, and click `New` then `Import` (we will delete this later), use the import code `3662`, then click load, select the prometheus data source, then import.

Navigate to that dashboard (it should automatically open), click edit and head to settings, and then click on JSON Model. Copy the UUID contained in line 19. This is your data source's UUID!

After ensuring you have the UUID copied, You can head to Dashboards and delete all additional ones we just created. Just make sure you hang onto that UUID.

Armed with the correct UUID, let's fix the validator dashboard. Click on Dashboards at the top of the screen, and navigate back to your dashboard. From there, click edit and head to settings, and then click on JSON Model. Press CTRL + F and click the down arrow to prepare a find and replace. Paste your data sources UUID into the Replace section, and `YOURUUID` into the Find section (note this is the UUID in line 37). Finally, click the replace all button to update your dashboard. To save, FIRST click `save changes` at the bottom and THEN `save dashboard` at the top of the screen, and finally click `save`.

IT'S TIME! Head back to your dashboard and witness a thing of beauty. Well done kid.