# Validator Guide: Readme Source: https://validators.endaoment.org/validators/introduction This series of documents provides guidance on setting up and managing an Ethereum Validator (including updating, pruning, etc.). Additionally, emergency protocols are provided in the case of an unrecoverable error on the node or the manager of a validator disappears (but the validator can be accessed). ## Abstract This guide assumes use of macOS on the connected device (LAN). While anyone can learn, this guide is meant for individuals who are tech savvy, and comfortable in command line without a GUI. ## Table of Contents 1. [Validator Readme](/validators/introduction) * Background Information * Disclaimers 2. [Validator SOP Part 1: Hardware, Software, & Deposit Setup](/validators/validator-sop-part1-setup) * Purchase Equipment and Assemble NUC * Generate Staking Data * Install Linux Server * Secure the Server * Configure Port Forwarding * Configure Timekeeping * Generate Client Auth Secret * Configure Reth * Install Prysm * Configure Beacon Node Service * Configure Validator Service * Import Validator Keys * Fund the Validator(s) 3. [Validator SOP Part 2: Upkeep, Updates, & Ongoing Management](/validators/validator-sop-part2-upkeep) * Package Management Updates * Updating Reth * Updating Prysm * Exiting a Validator * Checking Sync Committee Duties * MEV Boost * Launchpad Validator Actions * Resetting Reth Database 4. [Validator SOP Part 3: Emergency Action Plans](/validators/validator-sop-part3-emergency) * Validator Unrecoverable Error * Validator Operator Missing * Validator Operator Missing + No Validator Access 5. [Validator SOP Part 4: Monitoring and Dashboard Creation](/validators/validator-sop-part4-monitoring) * Adjust Firewall * Install Prerequisite Software * Install Reporting Software * Configure Ethereum Clients * Configure Reporting Software * Configure Prometheus Jobs + Targets * Set Up Dashboard ## Acknowledgements & Background Information This guide is heavily based off of Somer Esat's [guides](https://someresat.medium.com), CoinCashew's [guides](https://www.google.com/url?sa=t\&rct=j\&q=\&esrc=s\&source=web\&cd=\&ved=2ahUKEwjxvYzFvZmCAxWhD1kFHcQiAH0QFnoECBQQAQ\&url=https%3A%2F%2Fwww.coincashew.com%2Fcoins%2Foverview-eth%2Fguide-or-how-to-setup-a-validator-on-eth2-mainnet\&usg=AOvVaw1dMCm5BKdKm5z7JoK1TDtD\&opi=89978449), and the Stake Local Dashboard [guide](https://docs.stakelocal.io). We stand on the shoulders of those that came before us - thank you! Overall flow of the guide is as follows: * Generate the staking Deposit Data and Validator Keystore(s) **only if setting up a new validator** * Prepare the Ubuntu Server (updates, firewall, security, etc.) * Set up the Reth Execution Client node and sync it with the Ethereum mainnet * Set up the Prysm Consensus Client and sync with other Beacon Nodes * Deposit 32 ETH to activate the staking Validator(s) **only if setting up a new validator** * Importing Validator Keystore(s) from deprecated Validator **only if re-creating a defunct or otherwise offline validator** * Appendices covering updates, pruning, MEV-boost, exiting * Emergency Action Plans * Monitoring with use of Grafana locally In addition to hardware as described in the next step, you should have a printed copy of the [Variables](../images/variables.xlsx) document, which will be used frequently to keep track of information that should **never be stored electronically or otherwise shared**. ## Disclaimer 1. **Choice of Tools**: Our preference for utilizing Reth and Prysm in this guide is driven by the necessity for robust security and reliability—imperatives for our nonprofit operations. These platforms come with a wealth of community support and have undergone extensive audits, testing, and updates. While our choice may seem to marginalize other minority clients, it is purely a decision made in the interest of asset security. This is not a reflection of the quality or potential of alternative clients as generally we support client diversity. 2. **Not Financial Advice**: This document is intended solely as a procedural guide for setting up and managing an Ethereum Validator. It should not be construed as financial advice or an endorsement of cryptocurrency investment. Always conduct your own research and consult with financial advisors before making any investment or taking actions you're unfamiliar with. 3. **Dynamic Nature of Content**: This guide, like the software it aims to assist you with, is a living document. As such, it is subject to modifications, revisions, and updates. Changes will be managed through approved pull requests in the associated GitHub repository. Users are encouraged to stay updated with the most recent version of this guide and the software it covers. 4. **User Responsibility**: Following this guide involves interacting with blockchain technology, which carries inherent risks. Endaoment assumes no liability for any losses or damages that may occur due to the use or misuse of this guide. Exercise due caution and ensure you understand each step before proceeding. 5. **Additional Resources**: This guide incorporates lessons and methodologies from reputable sources and practitioners in the field. However, it is advisable to consult multiple sources for a well-rounded understanding and general verification. ChatGPT can also be a great resource to check the validity of Linux CLI commands mentioned in this guide. By proceeding with this guide, you acknowledge that you have read and understood the above disclaimers and risks. Endaoment is not responsible for how you choose to use this guide and/or any related documents. # Open Sourcing Endaoment's Ethereum Staking Guide Source: https://validators.endaoment.org/validators/validator-open-source-publish The Endaoment Ethereum Staking Guide is officially open source! This guide has been growing over the last few years. What began as one Endaoment employee's personal quest, transformed into [Endaoment](https://endaoment.org) spinning up five of its own Ethereum validators. Our guide aims to provide comprehensive instructions on setting up an Ethereum Proof of Stake (PoS) node containing a validator (or validators), maintaining its operations, and managing emergency scenarios effectively. We share these operational protocols in the spirit of open-source collaboration. We aim to contribute to the stability and robustness of the Ethereum network generally. Also, feedback never hurts! Before getting started let us be clear: we stand on the shoulders of giants. The work of [Somer Esat](https://twitter.com/SomerEsat), [Coin Cashew](https://twitter.com/coincashew%5F), and [mentanull.eth](https://twitter.com/MetanullOps) (Stake Local) have been instrumental in our ability to produce this guide and run our own set up. Explore more about their work in the resources section below. All right, let's get to it. ## A Quick Note: Client Diversity This guide uses a **Reth/Prysm** setup. We’re intentionally running **minority clients** (especially on the execution layer) and we hope this guide is helpful for anyone looking to run Reth and/or Prysm and contribute to client diversity. For current client distribution, see: [Client diversity](https://clientdiversity.org/) Client diversity remains important for network security and resilience (e.g., reducing correlated client risk). Our goal with this setup is to contribute to that diversity while maintaining the reliability and security standards required for our nonprofit operations. ## Humble Beginnings Endaoment's engagement in Ethereum staking is a manifestation of our faith in blockchain technology's potential to revolutionize philanthropic ventures (with specific emphasis on the Ethereum ecosystem post PoS transition). We hope here to detail the extensive trial, optimization, and real-world operation of our validator. Embracing a community-driven approach, we share our staking guide to enhance Ethereum's staking ecosystem and invite contributions for creating a reliable and accessible staking environment for all. This guide started as a humble many hundred line text file, constantly updated with the latest versions of clients, security findings, maintenance processes, and other insights gained from research as well as the guides mentioned above. It was regular practice to spend weekends and evenings tooling around with Linux CLI, experimenting with validating on Goerli, testing out peripheral software packages like [fail2ban](https://github.com/fail2ban/fail2ban), [grafana](https://github.com/grafana/grafana), and [prometheus](https://github.com/prometheus/prometheus) among others, and gathering best practices from any and all publicly available materials. Getting it right took a considerable amount of time - many months of rigorous evaluation, and at one time stress testing the selected hardware to see just how many validators could run off of the same node (we got to 100 without issue). ## Validate for Good In early 2023, the leadership of Endaoment made the decision to finalize internal policies related to Ethereum validating and deploy the majority of ETH in our [treasury](https://etherscan.io/address/0xd7d78Cf2f7c5a1B2E44080814D2f54a41A151A37) for staking. To that end, we made [five deposits](https://beaconcha.in/validators/deposits?q=a41406f86cc65063c25021422e2ee8e93e680b4c) for five newly spun up validators, entered the activation queue, and after a few weeks, finally began our journey in earnest on Mainnet. This decision was motivated by our desire to align our financial assets with our mission, using Ethereum staking as a means to further extend our impact and commitment to sustainable, community-focused investment strategies, while at the same time extending the runway of Endaoment.org by adding to our pool of operational capital. Our proactive involvement in the Ethereum network not only reflects our belief in blockchain as a tool for social good, but also underscores our commitment to being a forward-thinking, tech-embracing nonprofit. ## Operational Impact Ethereum staking is an exciting new way not just to help secure the Ethereum network on which Endaoment is built, but also to be able to deploy ETH for yield. With that said, this is not an endeavor that every nonprofit is suited to go on. The technical know-how required is considerable, and risks are high given the [32 ETH requirement](https://ethereum.org/en/staking/). While much of the work for getting set up for Ethereum staking happens at the beginning, ongoing maintenance is essential for prudent management. Execution and Consensus Clients continuously are getting upgraded, as is other peripheral software on the node. Databases may require pruning, hardware may become obsolete or otherwise need to be replaced, a bug could be introduced in a software upgrade, the list goes on - and this is to say nothing of the *outside the node* requirements like ongoing internet connectivity and an uninterrupted power supply in a secure location (and choosing that location!). Endaoment did not approach this process lightly, and we encourage all reading this guide to consider the risks as well as the rewards to Ethereum staking. Since activation, our validators have been an overall net positive. Aside from generating additional ETH and extending Endaoment's operational capital, participation in the Ethereum network coupled with alerts of block proposals and sync committees have energized the team and made us all additionally proud of our organization. ## Contribution and Feedback We welcome feedback and contributions from the community to enhance this guide. If you have suggestions or find any issues, please report them through GitHub issues or submit a pull request to our repository if you have changes to recommend. Your input is invaluable in making this a robust and user-friendly resource for the entire community. ## Future Outlook As we look to the future, Endaoment remains committed to its role within Ethereum network - not just as crypto's community foundation but as a contributor to network security and stability. Our journey does not end with the successful deployment and operation of our validators - we'll be here, updating the repo, handling the ongoing upkeep of our node and continuous improvement of our guide. To keep pace with the changing state of Ethereum (and the software required to run a validator or otherwise manage a node) we will be regularly updating this repo via public commits and pull requests. Our goal is to keep it as current and comprehensive as possible, integrating new learnings and strategies wherever we can. Our hope here is to demystify the process of Ethereum staking, while at the same time be transparent about our own internal processes. Endaoment recognizes the unique position it holds at the intersection of blockchain technology and philanthropy. Our ongoing involvement in Ethereum staking is more than a strategic decision; it's a reflection of our broader commitment to leveraging cutting-edge technology for social good. ## Acknowledgements Thank you, Ethereum community! It's been amazing to see the collective effort in transitioning to a more scalable and environmentally friendly PoS consensus mechanism. Special thanks to all contributors to this guide, who have played a part in making Ethereum staking more accessible and reliable. Read our previous article on [Endaoment Validators Are Live: Deepening Our Involvement in the Ethereum Ecosystem](https://endaoment.mirror.xyz/1ZKVUnusvN-nEOEhs0Lng36W0ykrdvOFLwEBhLyQwlc) to learn more about our initial foray into Ethereum staking. ## Resources * [Somer Esat's guides to staking on Ethereum](https://someresat.medium.com/) * [CoinCashew's guide on how to set up a validator for Ethereum staking on mainnet](https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet) * [StakeLocal Dashboard guide](https://docs.stakelocal.io) * [Beaconchain Knowledge Base](https://kb.beaconcha.in) ## Funding Endaoment If you'd like to contribute to Endaoment's mission and operations (including this guide!), please head [here](https://app.endaoment.org/orgs/84-4661797) and you can make a donation and get a tax receipt! If you'd like something a bit simpler, you can send any tips to `treasury.ndao.eth`. ## About the Author [Zach Bronstein](https://github.com/zbronstein) is [Endaoment's](https://github.com/endaoment/endaoment-validator/blob/main/endaoment.org) Chief Operating Officer; in addition to managing general operations of the organization, he is specifically in charge of all things Ethereum staking related and wrote the entirety of this guide (his baby) as well as the above statement. While Zach has built and configured Linux/Windows computers before, this is the first time he's published his work as a guide publicly, and is excited to share with the Ethereum community. Zach began his journey into web3 midway through 2020, eventually leaving his role at Morgan Stanley to join Endaoment as COO in early 2021. Zach lives with his wife Tracey, and their dog Ellie. You can find him at the social links below, as well as in [Endaoment's Discord](https://discord.gg/endaoment): [LinkedIn](https://www.linkedin.com/in/zbronstein/) [X](https://twitter.com/%5FZachBronstein) # Validator SOP Part 1: Hardware, Software, & Deposit Setup Source: https://validators.endaoment.org/validators/validator-sop-part1-setup ## Table of Contents * [Step 0 Purchase Equipment and Assemble NUC](/validators/validator-sop-part1-setup#step-0-purchase-equipment-and-assemble-nuc) * [Step 1 Generate Staking Data](/validators/validator-sop-part1-setup#step-1-generate-staking-data) * [Step 2 Install Linux Server](/validators/validator-sop-part1-setup#step-2-install-linux-server) * [Step 3 Secure the Server](/validators/validator-sop-part1-setup#step-3-secure-the-server) * [Modify the Default SSH Port](/validators/validator-sop-part1-setup#modify-the-default-ssh-port) * [Configure the Firewall](/validators/validator-sop-part1-setup#configure-the-firewall) * [Configure Fail2Ban](/validators/validator-sop-part1-setup#configure-fail2ban) * [Disable Root Access](/validators/validator-sop-part1-setup#disable-root-access) * [Step 4 Configure Port Forwarding](/validators/validator-sop-part1-setup#step-4-configure-port-forwarding) * [Step 5 Configure Timekeeping](/validators/validator-sop-part1-setup#step-5-configure-timekeeping) * [Step 6 Generate Client Auth Secret](/validators/validator-sop-part1-setup#step-6-generate-client-auth-secret) * [Step 7 Install Reth](/validators/validator-sop-part1-setup#step-7-install-reth) * [Step 8 Install Prysm](/validators/validator-sop-part1-setup#step-8-install-prysm) * [Step 9 Configure the Beacon Node Service](/validators/validator-sop-part1-setup#step-9-configure-the-beacon-node-service) * [Step 10 Configure the Validator Service](/validators/validator-sop-part1-setup#step-10-configure-the-validator-service) * [Step 11 Import Validator Keys](/validators/validator-sop-part1-setup#step-11-import-validator-keys) * [Import the Validator Keystore Files into Prysm](/validators/validator-sop-part1-setup#import-the-validator-keystore-files-into-prysm) * [Create a Wallet Password File](/validators/validator-sop-part1-setup#create-a-wallet-password-file) * [Step 12 Fund the Validators](/validators/validator-sop-part1-setup#step-12-fund-the-validators) * [Check the Status of Your Validator Deposits](/validators/validator-sop-part1-setup#check-the-status-of-your-validator-deposits) ### Step 0 Purchase Equipment and Assemble NUC First, you'll need hardware to run the validator software on. [Requirements can be found here](https://hackmd.io/G3MvgV2_RpKxbufsZO8VVg?view) as of 1/27/25, suggested hardware are as follows: * Asus NUC 14+ Pro * 32GB RAM (**64GB** preferred) * NVMe M.2 4TB * Ethernet connection with 100 Mbps symmetrical (**300 Mbps** preferred) * USB drive (used to install Linux) 4GB+ * HDMI (or USB) cord and connectable monitor for initial set up * USB keyboard and mouse (mouse is only required if using Ubuntu desktop to generate new validator keys) While not a hard requirement, it's also recommended to purchase a 1000+VA sine wave UPS to act as a battery backup in the case of a power failure. Note that in order for this to be effective, you should plug the validator **and** internet infrastructure into the UPS. The goal is to guard against both unclean shutdowns as well as downtime. Once equipment has been obtained, simply install the RAM and SSD in the NUC. Because some of the below processes require using an air-gapped machine, **do not connect the device to the internet**. Note: You will also need a small screwdriver to handle installation if assembling yourself. ### Step 1 Generate Staking Data **Only if setting up a new validator:** If setting up a new validator (and haven't already generated deposit data and validator key(s)), you'll need to run the deposit tool CLI, and should do so from an **air-gapped machine**. **If you are not generating new keys, skip to step 2.** In order to have a machine ready to go, we will need to install an OS on your new validator hardware built in step 0. We'll be using Ubuntu Desktop. On your laptop, navigate to [ubuntu.com](https://ubuntu.com/download/desktop), download the latest LTS copy of Ubuntu Desktop (as of 2025, *24.04.1 LTS*). In addition, download and install the [Etcher tool](https://www.balena.io/etcher) which will allow you to flash the operating system onto the USB drive. Lastly, download the latest version of the [deposit CLI tool](https://github.com/ethereum/staking-deposit-cli/releases) - you want the **AMD64** version - unzip the executable file. With all the software and data assembled, run the Etcher tool to flash the USB drive with Ubuntu Desktop using your laptop. Insert the USB into your air gapped machine (built in step 0), connect the machine to a monitor via HDMI (or USB), keyboard, and mouse, and go through the installation process (which should occur automatically). Make sure to choose the minimum settings allowed so things go quickly. Once the installation is complete, remove the USB drive, and reformat the disk to exFAT so you can move the deposit CLI tool onto the USB drive. You may also want to drop a text file into the USB drive with your withdrawal address to make the next step easier. Finally, re-insert the USB drive into the NUC and move the deposit CLI tool to your desktop. Now, run the binary file in a terminal window on your Linux desktop by using the below command (easier if you have your withdrawal address handy for a copy/paste!) ``` sudo ./deposit new-mnemonic --num_validators {#_Validators} --chain mainnet --eth1_withdrawal_address {YourWithdrawalAddress} ``` Replace `{#_Validators}` with the number of validators you want to run (32ETH per), and `{YourWithdrawalAddress}` with an Ethereum address you control. *NOTE: Once set, the withdrawal address* ***CANNOT*** *be changed, so be ABSOLUTELY SURE that it is an address you control the private keys for (or it's a multisig and you control the private keys to the underlying wallets) and correctly specified. For example: --eth1\_withdrawal\_address 0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045* **DO NOT** set the withdrawal address to a central exchange wallet or account. This MUST be an EOA or smart contract wallet, as ETH is not "transferred" but rather "generated" in the wallet via a state change, and that doesn't play nicely with Coinbase/Kraken/etc. Once you execute the above steps, confirm your withdrawal address within the generated files, and provide your language preferences. You will then be asked to create the validator keystore password. Note this password on the [variables document](../images/variables.xlsx). You will need this later to import the validator(s) into the Consensus Client. Please note that this document will be referenced a number of times below, and is meant to serve as an example for a way to keep a secure paper record of your passwords, file locations, etc. Next, a seed phrase (mnemonic) will be generated. Back it up somewhere safe. This is **CRITICAL**. You will need this to add additional validators or to regenerate keys if lost. IF YOU LOSE THIS MNEMONIC, YOU MAY LOSE ACCESS TO THE VALIDATOR! *Not your keys, not your coins.* Once you have confirmed your mnemonic your validator(s) will be created. The following files will be generated and placed on your desktop. * The deposit\_data-\[timestamp].json file contains the public key(s) for the validator(s) and information about the staking deposit. This file will be used to complete the ETH staking deposit process later in this guide. **This is where you can triple check the withdrawal address** * The keystore-\[..].json files contain the encrypted validator signing key. There is one keystore file per validator that you are funding. These will be imported into the Consensus Client for use during validation operations. * If you lose or accidentally delete the files it is possible to regenerate them using the Staking Deposit CLI tool and your mnemonic via the existing-mnemonic command (see [here](https://github.com/ethereum/staking-deposit-cli#commands)). Now that you have generated the deposit data, keystore file(s), validator password, and the mnemonic, we can begin setting up the node. You should now copy the generated deposit and keystore files back onto your USB drive. We'll need them later on. Note: there are no other steps in this process that require you to enter or view your mnemonic phrase, so the air-gapped portion of this process is completed. You can now plug in the Ethernet cord. ### Step 2 Install Linux Server Navigate to [ubuntu.com](https://ubuntu.com/download/server), download the latest LTS copy of Ubuntu Server (as of 2025, *24.04.1 LTS*). In addition, download and install the [Etcher tool](https://www.balena.io/etcher) which will allow you to flash the operating system onto the USB drive. Then, run the Etcher tool to flash the USB drive with Ubuntu Server. Once that's complete, pop the USB drive into your NUC, and restart it - also connect it to Ethernet if you haven't already. Upon restart, hold fn+F10 in order to boot from USB. Follow the installation instructions, making sure to select the following settings: * When setting up hard drive settings, uncheck the LVM box else the full HD cannot be used * You should make sure no software is downloaded or installed aside from OpenSSH * **Do not** select "minimum" when installing the software, you will need the items downloaded and installed in the normal process * Name your device whatever you like - and note this on your variable document * Create a user when prompted to do so, and set a password for that user that's sufficiently complex - note this on your variable document Once this process is complete, you should be presented with a terminal window on screen. Likely, you will need to enter your password to gain access to the server. Next, update the server and install useful utilities by running the following commands: ``` sudo apt -y update && sudo apt -y upgrade && sudo apt dist-upgrade && sudo apt autoremove sudo apt install -y ccze jq sudo reboot ``` **Note:** `ccze` provides colorized log output, and `jq` is a JSON parser useful for formatting RPC responses. After rebooting: ``` df -h ``` That last command is used to ensure that the expected amount of free space on your SSD is available. If this value is less than 75% of the full size of the drive, something went wrong (likely the LVM checkbox during Linux installation), and you'll need to start this step over. ### Step 3 Secure the Server After the updates and restart complete, it's time to change some security settings. #### Modify the Default SSH Port Port 22 is the default SSH port and a common attack vector. Change the SSH port to avoid this. Choose a port number between 1024–49151 and run the following command to check is not already in use: ``` sudo ss -tulpn | grep ':{YourSSHPortNumber}' ``` A blank response indicates not in use, a red text response indicates it is in use: try a different port. If confirmed available, modify the default SSH port number by updating your server’s SSH config. Open the config file with the following command: ``` sudo nano /etc/ssh/sshd_config ``` Find the line Port 22 in the file. Remove the # and change the value to your chosen port. Note this port number on the variables document as you'll need it for LAN access (we're almost ready to do the rest from your laptop!) Press CTRL + X then Y then ENTER to save and exit. Restart the SSH service to reflect the changes. ``` sudo systemctl restart ssh ``` #### Configure the Firewall Install UFW: ``` sudo apt install ufw ``` Next, apply UFW defaults: ``` sudo ufw default deny incoming && sudo ufw default allow outgoing ``` Next, we'll need to create some custom rules to allow the requisite network activity/connections Allow Custom SSH Port: ``` sudo ufw allow {YourSSHPortNumber}/tcp ``` Deny Default SSH Port: ``` sudo ufw deny 22/tcp ``` Allow Execution Client Ports: ``` sudo ufw allow 30303 sudo ufw allow 30305/udp ``` *Note: Port 30303 is used for P2P networking and Discovery v4. Port 30305 is used for Discovery v5 (improved peer discovery protocol). If you need to customize the networking ports, consult the [Reth documentation](https://github.com/paradigmxyz/reth) for the appropriate flags and ensure those ports are allowed in your firewall.* Allow Prysm: ``` sudo ufw allow 13000/tcp && sudo ufw allow 12000/udp ``` With settings imported, enable the firewall by running: ``` sudo ufw enable ``` Then run the following command to see a list of ports enabled/disabled: ``` sudo ufw status numbered ``` Finally, run the following command to get your local IP: ``` hostname -I ``` Note this on your variables document. **WITHOUT** disconnecting from the Node, head over to your laptop (or whichever LAN device you plan on accessing the Node from), and attempt to connect. Open a terminal window and input the following command to edit your hosts file so you can refer to the Node by the name you set it as in addition to its IP: ``` sudo nano /etc/hosts ``` When prompted, enter your password. Now, above the final `::1 localhost line`, create a new line, and add the local IP of your node *(see: variables)*, then a few spaces so you're aligned with the localhost text, and then enter the name of the Node you used above *(see: variables)*. Now, let's test to see if you can SSH into your Node. Simply open a new terminal window, and enter the following command, subbing in the name of the Node, user, and the SSH port *(see: variables)*. ``` ssh {yourusername}@{node_name OR node_IP} -p {YourSSHPortNumber} ``` Enter your password when prompted. If you are able to log in and see `{user}@{node_name}`, you're set! Once you see this, you can disconnect the Node from the monitor and keyboard, and can perform the remainder of tasks from any laptop or computer on the same LAN using the command above to SSH in. The Node **only** needs to remain connected to power and ethernet (powered via UPS if possible). #### Configure Fail2Ban Fail2Ban is an intrusion-prevention program that scans log files and bans IPs that show malicious activity. If a certain number of failed logins are detected from a specific IP address (within a specified amount of time), that IP address is blocked. This service provides basic protection against brute-force attacks, and can be configured to ignore local IPs. Install fail2ban, and enable and start the service: ``` sudo apt install -y fail2ban && sudo systemctl enable fail2ban ``` Create the configuration file for the service: ``` sudo nano /etc/fail2ban/jail.local ``` Then, paste the following configuration into the file: ``` [sshd] enabled = true port = {your SSH port} filter = sshd logpath = /var/log/auth.log maxretry = 4 # whitelisted IP addresses ignoreip = 192.168.1.0/24 127.0.0.1/8 ``` NOTE: The inclusion of ignoreip = 192.168.1.0/24 127.0.0.1/8 is to allow local IPs to connect (and potentially get passwords wrong) without being blocked. Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start fail2ban sudo systemctl status fail2ban ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the fail2ban service) #### Disable Root Access It is best practice to not log in as root in order to maintain security and follow the principle of least-privilege. First, check to ensure another user besides root can run the sudo command (if you've gotten this far, this is just a double check step). Let's first check sudo access: ``` sudo -l ``` If you see the text "User `{your-username}` may run the following commands on `{device-name}`: (ALL:ALL) ALL" you have sudo privileges with your current user. Now, lock the root account: ``` sudo passwd -l root ``` You should see the text "passwd: password expiry information changed." If for any reason you need to unlock the root account: ``` sudo passwd -u root ``` ### Step 4 Configure Port Forwarding From your laptop (or whichever device on the local network you plan on accessing the Node from), log into your router to edit the port forwarding settings. While access is different for each network set up and router, usually you can access your router at 192.168.1.1 or a similar local IP address. Occasionally, instructions can be found on the router itself. Once you've logged in to your router, edit the port forwarding settings for your Node to forward the following ports: **Reth (Execution Client):** * **Port 30303** - TCP and UDP - P2P networking and Discovery v4 * **Port 30305** - UDP only - Discovery v5 (improved peer discovery) **Prysm (Consensus Client):** * **Port 13000** - TCP and UDP - P2P networking * **Port 12000** - UDP only - QUIC discovery These are the same ports we allowed above in the *Secure the Server* step. Port forwarding these will make it easier for both the execution client (Reth) and consensus client (Prysm) to find suitable peers and accept inbound connections. Failure to do so may result in sub-optimal syncing and significantly reduced peer discovery (especially for Prysm, which targets 70 peers). ### Step 5 Configure Timekeeping Running validators against a blockchain requires accurate timekeeping in order to ensure proper synchronization with the network. Validators need time accuracy within **0.5 seconds** for proper attestations. While Ubuntu has time synchronization built in using `systemd-timesyncd`, we'll use **`chrony`** instead, which provides more accurate time synchronization - critical for validators. Install chrony (this will replace systemd-timesyncd): ``` sudo apt-get install chrony -y ``` Verify chrony is running and check synchronization sources: ``` sudo systemctl status chrony ``` Check the time servers chrony is using: ``` chronyc sources ``` You should see multiple time servers with various symbols (^\*, ^+, ^-, ^?) indicating their status. Check synchronization accuracy: ``` chronyc tracking ``` **Important values to check:** * **Leap status**: Should say "Normal" (after a few minutes) * **System time**: Should show offset in milliseconds (\< 100ms is good, \< 10ms is excellent) * **Stratum**: Should be 2-4 (lower is better) **Note:** Right after installation, chrony needs 5-10 minutes to synchronize with time servers. The "Leap status" will show "Not synchronized" initially - this is normal. Optionally, set your timezone (affects log timestamps only, doesn't impact validation): ``` sudo dpkg-reconfigure tzdata ``` Select your geographic region and city from the menu. ### Step 6 Generate Client Auth Secret On the server, communication between the Execution and Consensus clients is secured using a JSON Web Token (JWT) authentication scheme. The JWT is represented by a file that contains a randomly generated 32-byte hex string. The Execution and Consensus clients each make use of the file for message authentication. First, create a directory to store this file: ``` sudo mkdir -p /var/lib/jwtsecret ``` Then, generate the JWT file using openssl: ``` openssl rand -hex 32 | sudo tee /var/lib/jwtsecret/jwt.hex > /dev/null ``` Finally, check that the file was created as expected: ``` sudo nano /var/lib/jwtsecret/jwt.hex ``` Press CTRL+X to exit ### Step 7 Install Reth Time to install Reth! All commands provided below are based on the v2.6.0 version of Reth (current as of September 2026), but should be adjusted based on whatever the latest version of Reth is [here](https://github.com/paradigmxyz/reth/releases) - simply expand the Assets section and copy the URL for the 'reth-v...-x86\_64-unknown-linux-gnu.tar.gz' file. Note that MANY commands below will need to update if this version is updated. **Node Modes — Minimal vs Full vs Archive:** This guide configures Reth in **minimal storage mode** (using the `--minimal` flag), which is the optimal configuration for validators: * **Minimal Node**: State and headers with minimal history. \~224 GB with Storage V2. Sufficient for all validation operations including attestation, block proposals, and MEV-boost. Does not support historical transaction lookups or receipt queries. * **Full Node**: Recent state history plus transaction lookups and receipts for the last \~10,064 blocks. \~1 TB with Storage V2. Useful if you need local `eth_getTransactionReceipt` or `eth_getLogs` queries. * **Archive Node**: Complete historical state since genesis. \~2.3 TB with Storage V2. Only needed for specialized applications like block explorers or historical data analysis. **For validators, always use a minimal node.** Full and archive nodes provide no benefit for validation and waste disk space. All Engine API operations, MEV-boost relay validation, and standard monitoring (`eth_syncing`, `net_peerCount`, `txpool_*`) work with minimal mode. Curl the Reth build from the above link: ``` cd ~ curl -LO https://github.com/paradigmxyz/reth/releases/download/v2.6.0/reth-v2.6.0-x86_64-unknown-linux-gnu.tar.gz ``` Extract the files from the archive and copy to the /usr/local/bin directory. The Reth service will run it from there. Modify the file name to match the downloaded version: ``` tar xvf reth-v2.6.0-x86_64-unknown-linux-gnu.tar.gz sudo cp reth /usr/local/bin ``` Clean up the files. Modify the file name to match the downloaded version: ``` cd ~ rm reth-v2.6.0-x86_64-unknown-linux-gnu.tar.gz ``` Reth will be configured to run as a background service. Create an account for the service to run under. This type of account can't log into the server: ``` sudo useradd --no-create-home --shell /bin/false reth ``` Create the data directory. This is required for storing the Ethereum blockchain data: ``` sudo mkdir -p /var/lib/reth ``` Set directory permissions. The Reth user account needs permission to modify the data directory: ``` sudo chown -R reth:reth /var/lib/reth ``` Create a home directory for the Reth user. This is required for Reth's logging functionality: ``` sudo mkdir -p /home/reth sudo chown reth:reth /home/reth ``` Create a systemd service config file to configure the service: ``` sudo nano /etc/systemd/system/reth.service ``` Then, paste the following configuration into the file: ``` [Unit] Description=Reth Execution Client (Mainnet) After=network.target Wants=network.target [Service] User=reth Group=reth Type=simple Restart=always RestartSec=5 TimeoutStopSec=900 KillSignal=SIGINT ExecStart=/usr/local/bin/reth node \ --chain mainnet \ --minimal \ --datadir /var/lib/reth \ --authrpc.jwtsecret /var/lib/jwtsecret/jwt.hex \ --authrpc.addr 127.0.0.1 \ --authrpc.port 8551 \ --port 30303 \ --discovery.port 30303 \ --enable-discv5-discovery \ --discovery.v5.port 30305 \ --max-outbound-peers 35 \ --max-inbound-peers 35 \ --http \ --http.addr 127.0.0.1 \ --http.api eth,net,web3 [Install] WantedBy=default.target ``` **IMPORTANT NOTES:** * **`--minimal` flag**: Runs Reth in **minimal storage mode**, which aggressively prunes history not needed for validation. This is the recommended mode for validators — it fully prunes sender recovery, transaction lookup, and receipts while keeping the last \~10,064 blocks of account/storage history. Disk usage is \~224 GB with Storage V2. * **`TimeoutStopSec=900`**: Allows the Reth service 15 minutes to write cached data to disk on ***clean*** shutdown (important for large databases). * **`KillSignal=SIGINT`**: Explicitly tells systemd to use SIGINT for graceful shutdown (same as Ctrl+C). * **Port configuration**: Explicitly sets P2P networking ports to 30303 (default) for better peer discovery. This is the default, but better to make explicit. * **Discovery v5**: Enables the newer, more resilient peer discovery protocol on port 30305 UDP. Improves peer finding and network connectivity. * **Peer limits**: Limits connections to 35 outbound + 35 inbound (70 total) to reduce bandwidth and resource usage while maintaining excellent connectivity. For gigabit home validators, this balances network contribution with efficiency. Press CTRL + X then Y then ENTER to save and exit. **DO NOT add additional pruning flags** (`--prune.accounthistory.full`, `--prune.storagehistory.full`, etc.) as they will **conflict** with the `--minimal` flag's configuration and cause the pruner to fail. The `--minimal` flag handles all pruning automatically. #### Optional: Use Reth Snapshot for Faster Initial Sync **Why use a snapshot?** Syncing Reth from genesis can take days. Using an official snapshot from [snapshots.reth.rs](https://snapshots.reth.rs/) can reduce initial sync time to just a few hours. Using a snapshot is particularly recommended for new installations or after database corruption. The `reth download` command downloads a pre-built snapshot directly from the official Reth snapshot portal. It supports **minimal**, **full**, and **archive** snapshot tiers — for validators, use `--minimal`. Downloads use Storage V2 by default and support up to 8 concurrent download workers. **To download a snapshot:** While `reth download` supports resumable downloads (tolerating SSH disconnections), it's still recommended to use `tmux` as a safeguard: ``` sudo apt-get update sudo apt-get install -y tmux ``` Start a tmux session: ``` tmux new -s reth-download ``` Then, inside the tmux session, run the download: ``` sudo -u reth /usr/local/bin/reth download --chain mainnet --minimal --datadir /var/lib/reth --resumable ``` This command will: * Download the minimal snapshot from [snapshots.reth.rs](https://snapshots.reth.rs/) (state + headers + minimal history) * Extract it directly into your data directory with Storage V2 layout * Support resuming if interrupted (the `--resumable` flag downloads to a `.part` file with HTTP Range support before extracting) **If download gets interrupted:** With the `--resumable` flag, simply rerun the same command — it will pick up where it left off. No need to wipe and start over. **Tmux controls:** * **Detach from session** (leaves download running): Press `Ctrl+B` then `D` * **Reattach to session**: `tmux attach -t reth-download` * **Monitor progress** in another terminal: `watch -n 30 'df -h /var/lib/reth'` or `watch -n 60 'du -sh /var/lib/reth'` Once the snapshot extraction completes, proceed with starting the service as normal. **If not using a snapshot:** Reth will sync from genesis with the `--minimal` flag, which takes 1-3 days but results in a very small database (\~224 GB with Storage V2). Skip the snapshot download and proceed directly to starting the service. This is a valid approach if you have time, though the snapshot approach is significantly faster. #### Start Reth Service Reload systemd to reflect the changes and start the service. Check the status to make sure it's running correctly: ``` sudo systemctl daemon-reload sudo systemctl start reth sudo systemctl status reth ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the Reth service) The sync will begin immediately. Use the journal output to follow the progress or check for errors by running the following command. ``` sudo journalctl -fu reth | ccze ``` Press CTRL+ C to exit (will not affect the Reth service). **Note (initial sync / snapshot troubleshooting):** During the initial sync, Reth may occasionally hit an error during pruning or appear to get “stuck”. If you’re monitoring logs and see errors, Reth generally does a good job persisting progress. In many cases, a simple restart is enough to resume work: ``` sudo systemctl restart reth ``` Next, enable the Reth service to automatically start on reboot: ``` sudo systemctl enable reth ``` Finally, stop and start reth so it's using all the latest config information: ``` sudo systemctl stop reth sudo systemctl start reth ``` Now that Reth is all set up, you can check sync status and connected peers using JSON-RPC calls. Check the sync status using curl: ``` curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' http://localhost:8545 ``` You can also check the count of connected peers by running: ``` curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' http://localhost:8545 ``` After running Reth for a few minutes, check the peer count to ensure it's greater than 0. ### Step 8 Install Prysm The Prysm Consensus Client is made up of two binaries that provide the functionality of the beacon node and validator, respectively. This step will download and prepare the Prysm binaries. Time to install Prysm! All commands provided below are based on the v7.2.0 version of Prysm (current as of 9/28/26), but should be adjusted based on whatever the latest version of Prysm is [here](https://github.com/OffchainLabs/prysm/releases). Note that many commands below will need to update if this version is updated. In the Assets section (expand if necessary) copy the download links to the beacon-chain-v…-linux-amd64 file and the validator-v…-linux-amd64 file. Be sure to copy the correct links. Download the binaries using the commands below. Modify the URL to match the copied download links: ```bash theme={null} cd ~ curl -LO https://github.com/OffchainLabs/prysm/releases/download/v7.2.0/beacon-chain-v7.2.0-linux-amd64 curl -LO https://github.com/OffchainLabs/prysm/releases/download/v7.2.0/validator-v7.2.0-linux-amd64 ``` Now, let's rename the files and make them executable and copy them to the /usr/local/bin directory. The Prysm services will run them from there. Modify the file names to match the downloaded version: ```bash theme={null} mv beacon-chain-v7.2.0-linux-amd64 beacon-chain mv validator-v7.2.0-linux-amd64 validator ``` ```bash theme={null} chmod +x beacon-chain chmod +x validator ``` ```bash theme={null} sudo cp beacon-chain /usr/local/bin sudo cp validator /usr/local/bin ``` Finally, clean up the files ```bash theme={null} rm beacon-chain && rm validator ``` ### Step 9 Configure the Beacon Node Service Set up an account for the services to run under (note: this type of account cannot log into the server): ``` sudo useradd --no-create-home --shell /bin/false prysmbeacon ``` Next, set up the directories and permissions for prysmbeacon: ``` sudo mkdir -p /var/lib/prysm/beacon sudo chown -R prysmbeacon:prysmbeacon /var/lib/prysm/beacon ``` Next, create and configure the service file: ``` sudo nano /etc/systemd/system/prysmbeacon.service ``` And then paste the following data into the file: ```bash theme={null} [Unit] Description=Prysm Consensus Client BN (Mainnet) Wants=network-online.target After=network-online.target [Service] User=prysmbeacon Group=prysmbeacon Type=simple Restart=always RestartSec=5 ExecStart=/usr/local/bin/beacon-chain \ --mainnet \ --datadir=/var/lib/prysm/beacon \ --execution-endpoint=http://127.0.0.1:8551 \ --jwt-secret=/var/lib/jwtsecret/jwt.hex \ --suggested-fee-recipient={FeeRecipientAddress} \ --checkpoint-sync-url={CheckpointSyncURL} \ --genesis-beacon-api-url={CheckpointSyncURL} \ --accept-terms-of-use \ --blob-storage-layout=by-epoch \ --postpone-shutdown-for-proposals Environment=USE_PRYSM_MODERN=true [Install] WantedBy=multi-user.target ``` **Make sure** to replace `{FeeRecipientAddress}` with the address you'd like to receive the validator tip fees. **Make sure** to replace `{CheckpointSyncURL}` *(x2)* with a valid checkpoint sync URL [see here for more information](https://eth-clients.github.io/checkpoint-sync-endpoints/) - be sure to select a Mainnet endpoint. Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start prysmbeacon sudo systemctl status prysmbeacon ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the PrysmBeacon service) - the sync has already begun. Use the journal output to follow the progress or check for errors by running the following command: ``` sudo journalctl -fu prysmbeacon | ccze ``` Finally, enable the service to automatically start on reboot: ``` sudo systemctl enable prysmbeacon ``` ### Step 10 Configure the Validator Service Set up an account for the services to run under (note: this type of account cannot log into the server): ``` sudo useradd --no-create-home --shell /bin/false prysmvalidator ``` Create a directory to store the validator data: ``` sudo mkdir -p /var/lib/prysm/validator ``` Next, set up the permissions for prysmvalidator: ``` sudo chown -R prysmvalidator:prysmvalidator /var/lib/prysm/validator ``` Next, create and configure the service file by running: ``` sudo nano /etc/systemd/system/prysmvalidator.service ``` And then paste the following data into the file: ``` [Unit] Description=Prysm Consensus Client VC (Mainnet) Wants=network-online.target After=network-online.target [Service] User=prysmvalidator Group=prysmvalidator Type=simple Restart=always RestartSec=5 ExecStart=/usr/local/bin/validator \ --datadir=/var/lib/prysm/validator \ --wallet-dir=/var/lib/prysm/validator \ --wallet-password-file=/var/lib/prysm/validator/password.txt \ --suggested-fee-recipient={FeeRecipientAddress} \ --accept-terms-of-use [Install] WantedBy=multi-user.target ``` **Make sure** to replace `{FeeRecipientAddress}` with the address you'd like to receive the validator tip fees. Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start prysmvalidator sudo systemctl status prysmvalidator ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the PrysmValidator service) - the sync has already begun. Use the journal output to follow the progress or check for errors by running the following command: ``` sudo journalctl -fu prysmvalidator | ccze ``` Finally, enable the service to automatically start on reboot: ``` sudo systemctl enable prysmvalidator ``` Now that the beacon-chain, validator, and execution client software is all set up, you should run the following commands to check sync status: Check the sync status using JSON-RPC (it will work now!): ``` curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' http://localhost:8545 ``` You can also check the peer count: ``` curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' http://localhost:8545 ``` ### Step 11 Import Validator Keys To import the keys into Prysm, we'll need to get them onto the same machine. #### Copy Validator Keystore File(s) to Server Before we transfer over the Validator Keystore file(s), we'll need to create a place to store them: ``` sudo mkdir -p $HOME/staking-deposit-cli/validator_keys ``` Next, ensure your user has control of that folder: ``` sudo chown -R {yourusername}:{yourusername} $HOME/staking-deposit-cli/validator_keys ``` Now, plug the USB into your Node from Step 1 (remember Step 1?). Back in terminal, check to see if the drive is recognized: ``` sudo fdisk -l ``` This will list all available disks and partitions. You should see the USB drive listed as a block device, such as /dev/sdb or /dev/sdc, and specific drives underneath. Look for your USB drive and note its code (e.g. sdb1 or sdc1). Create a mount point for the USB drive by running the following command: ``` sudo mkdir /mnt/usb ``` Then, mount the USB drive to the mount point, replacing sdb1: ``` sudo mount /dev/{sdb1} /mnt/usb ``` Check that the USB drive is connected by looking at all mounted drives: ``` df -h ``` Now, it's time to copy the files from the USB to the proper folders within the Node. First, let's get a list of all files by running: ``` ls /mnt/usb ``` This should show something like the following > deposit\_data-1682896118.json keystore-m\_12381\_3600\_0\_0\_0-1682896115.json Note that while you will only ever have 1 deposit file (regardless of the amount of validators), you will have 1 keystore file for each validator you are setting up on the Node. You only need to move the keystore file(s) to the validator\_keys folder. Now that we have the file names, we issue a command to copy those files from /mnt/usb to \$HOME/staking-deposit-cli/validator\_keys: ``` cp /mnt/usb/{keystore-filename.json} $HOME/staking-deposit-cli/validator_keys/ ``` If you have more than 1 validator key, you will need to rerun the above command once for each. **BE SURE TO ONLY HAVE 1 COPY OF EACH VALIDATOR KEY**. Before unmounting the USB, let's verify that the files were placed into the folder correctly by running: ``` ls $HOME/staking-deposit-cli/validator_keys/ ``` If you see 1 copy of each validator key you intended to move, you're all set! Now, unmount the USB: ``` sudo umount /mnt/usb ``` #### Import the Validator Keystore Files into Prysm **REMEMBER: If you import the same keystore twice OR run the same keystore on multiple nodes at the same time YOU WILL GET** [**SLASHED**](https://www.blocknative.com/blog/an-ethereum-stakers-guide-to-slashing-other-penalties)**!** Now we will run the validator import process. You will need to provide the directory where the generated keystore-\[..].json files are located. E.g. \$HOME/staking-deposit-cli/validator\_keys (as determined above in this step) ``` sudo systemctl stop prysmvalidator sudo /usr/local/bin/validator accounts import --keys-dir=$HOME/staking-deposit-cli/validator_keys --wallet-dir=/var/lib/prysm/validator --mainnet sudo systemctl start prysmvalidator ``` You will then be presented with a terms of use which you will need to accept in order to continue. Following your acceptance, you'll be prompted to create a wallet password. Note that this is a **different** password than the validator password set in step 1 when generating the keystore files. This password will be used by Prysm to decrypt the validator files. Note this password on the variables document. After setting your wallet password, you'll be asked to enter the validator keystore password, this is the one referenced above and created during step 1. Once this password is entered, all validator keystore file(s) will be imported. #### Create a Wallet Password File Create a file to store the wallet password so the Prysm validator service (set up below) can access the wallet without you having to supply the password: ``` sudo nano /var/lib/prysm/validator/password.txt ``` This will create a new text file and open a nano text editor. Type your password you just created for your validator wallet into the file. Press CTRL + X then Y then ENTER to save and exit. Let's now check that Prysm can see the password: ``` sudo systemctl status prysmvalidator ``` Before continuing to the deposit step, let's do one more permissions check: ``` sudo chown -R prysmvalidator:prysmvalidator /var/lib/prysm/validator && sudo chown -R {yourusername}:{yourusername} /var/lib/prysm/validator sudo systemctl daemon-reload ``` ### Step 12 Fund the Validators Now that the Consensus Client is up and running, to actually begin staking on the Ethereum network you will need to deposit ETH to fund your validators. **Note: DO NOT continue until your execution and consensus clients are fully synced. If they have not and your validator(s) become active on the network, you would be subject to inactivity penalties. You can determine sync status by checking the status of the Beacon Chain or through JSON-RPC calls to Reth, both methods are described above.** With the deposit file in hand and an EOA with 32ETH per validator + some spare ETH for gas, head to the [Ethereum Launchpad](https://launchpad.ethereum.org/) Click on Become a Validator, click through the warning steps and continue through the screens until you get to the Generate Key Pairs section. Select the number of validators you are going to run. Choose a value that matches the number of validators you created in Step 1. Scroll down, check the box if you agree, and click Continue. You will be asked to upload the deposit\_data-\[timestamp].json file. You generated this file in Step 1. *(Note: There are no security concerns copying the file or having access to it on a public computer)*. Double check that inside the file the withdrawal address is set correctly. When satisfied, browse or drag the file to upload and click continue. Connect your wallet. Choose MetaMask (or one of the other supported wallets), log in, select the account where you have your ETH and click Continue. Your MetaMask balance will be displayed. The site will allow you to continue only if you have selected Mainnet and you have a sufficient ETH balance. A summary shows the number of validators and total amount of ETH required. Tick the boxes if you agree and click continue. If you are ready to deposit click on `Initiate All Transactions`. This will pop open MetaMask (or one of the other supported wallets) where you can confirm and broadcast each transaction. Once all the transactions have successfully completed you are done! WOOOOOOO! Congratulations you have deposited your stake! #### Check the Status of Your Validator Deposits Newly added validators can take a while to activate. You can check the status of your keys with these steps: 1. Copy your wallet address used to make the deposit 2. Go to [Beaconcha.in](https://beaconcha.in/) 3. Search for your key(s) using your wallet address Digging into a specific validator we can see a `Status` section that provides an activation estimate time for each validator. That’s it. You now have a functioning Execution and Consensus client and the staking deposit done. Once your deposit is active you will automatically begin staking and earning rewards. Probably a good time to *touch grass*. #### Clean Up `validator_keys` While not essential as you can always regenerate them using your mnemonic, it's good practice to back up the `validator_keys` directory (typically `$HOME/staking-deposit-cli/validator_keys`) on a USB for [emergencies](./validator-sop-part3-emergency.mdx). After ensuring you have a backup of these files as described in [Step 1 Generate Staking Data](/validators/validator-sop-part1-setup#step-1-generate-staking-data), you can safely remove the validator\_keys directory. Note that this will not impact your existing validators, as in [Step 11 Import Validator Keys](/validators/validator-sop-part1-setup#step-11-import-validator-keys) the validator\_keys are imported into the validator instance. The copies referenced above are "extra": ``` sudo rm -r $HOME/staking-deposit-cli/validator_keys ``` # Validator SOP Part 2: Upkeep, Updates, & Ongoing Management Source: https://validators.endaoment.org/validators/validator-sop-part2-upkeep ## Table of Contents * [Appendix A Package Management Updates](/validators/validator-sop-part2-upkeep#appendix-a-package-management-updates) * [Appendix B Updating Reth](/validators/validator-sop-part2-upkeep#appendix-b-updating-reth) * [Appendix C Updating Prysm](/validators/validator-sop-part2-upkeep#appendix-c-updating-prysm) * [Appendix D Exiting a Validator](/validators/validator-sop-part2-upkeep#appendix-d-exiting-a-validator) * [Appendix E Checking Sync Committee Duties](/validators/validator-sop-part2-upkeep#appendix-e-checking-sync-committee-duties) * [Appendix F MEV-Boost](/validators/validator-sop-part2-upkeep#appendix-f-mev-boost) * [MEV Boost Set Up](/validators/validator-sop-part2-upkeep#mev-boost-set-up) * [Update BeaconChain and Validator Software](/validators/validator-sop-part2-upkeep#update-beaconchain-and-validator-software) * [Update MEV Boost](/validators/validator-sop-part2-upkeep#update-mev-boost) * [Appendix G Launchpad Validator Actions](/validators/validator-sop-part2-upkeep#appendix-g-launchpad-validator-actions) * [Appendix H Resetting Reth Database](/validators/validator-sop-part2-upkeep#appendix-h-resetting-reth-database) ### Appendix A Package Management Updates To maintain optimal performance and security of your Node, it is crucial to regularly execute the following command: ``` sudo apt update && sudo apt -y full-upgrade && sudo apt -y autoremove && sudo apt -y autoclean ``` This will: * Update the package list for all configured repositories * Upgrade all packages to their latest versions and install all new dependencies required by updated packages * Remove any packages that were installed as dependencies that are no longer needed * Clean up the package cache * Not require your input (the -y flag prevents certain commands from being confirmed by you, you can feel free to remove if you'd like to see what will update and then permit that update in separate steps) Note that because it's a sudo command, you'll need to enter your password. It is not irregular for your Node to prompt you to restart it after some particular updates are made. Sometimes this is requested after packages are updated, other times it's requested and shown to you upon your next login. ### Appendix B Updating Reth First, go to the Reth Repository [here](https://github.com/paradigmxyz/reth/releases) and expand the Assets section for the latest release. Copy the download link for the `reth-v...-x86_64-unknown-linux-gnu.tar.gz` file. Be sure to copy the correct link. It should look something like `https://github.com/paradigmxyz/reth/releases/download/v2.6.0/reth-v2.6.0-x86_64-unknown-linux-gnu.tar.gz`. Modify the URL in the instructions below to match the download link for the latest version, which is v2.6.0 in this example (current as of September 2026). ``` cd ~ curl -LO https://github.com/paradigmxyz/reth/releases/download/v2.6.0/reth-v2.6.0-x86_64-unknown-linux-gnu.tar.gz ``` Then stop the Reth service ``` sudo systemctl stop reth ``` Note that stopping Reth always takes a moment and is not expected to complete immediately, don't stress, just be patient. Now extract the files from the archive and copy to the /usr/local/bin directory. Modify the file name to match the downloaded version: ``` tar xvf reth-v2.6.0-x86_64-unknown-linux-gnu.tar.gz sudo cp reth /usr/local/bin ``` Next, restart the services and check for errors: ``` sudo systemctl start reth ``` ``` sudo systemctl status reth # <-- Check for errors sudo journalctl -fu reth | ccze # <-- Monitor sudo journalctl -fu prysmbeacon | ccze # <-- Monitor ``` Finally, clean up the files, modifying the file name to match the downloaded version. Then check the Reth version to verify that all was installed correctly. This should match the Reth version shown on the repo site: ```bash theme={null} rm reth-v2.6.0-x86_64-unknown-linux-gnu.tar.gz reth --version ``` ### Appendix C Updating Prysm In order to update Prysm, we'll need to update both the validator and beacon-chain software. First, let's check the latest version of Prysm available on their GitHub repo, which can be accessed [here](https://github.com/OffchainLabs/prysm/releases). Be sure to copy the correct link, it should look something like `https://github.com/OffchainLabs/prysm/releases/download/v7.2.0/beacon-chain-v7.2.0-linux-amd64.sha256`. We will need to modify the commands below to match the latest version number, which is v7.2.0 in this example (current as of 9/28/26). Curl the latest software and rename: ```bash theme={null} curl -LO https://github.com/OffchainLabs/prysm/releases/download/v7.2.0/validator-v7.2.0-linux-amd64 curl -LO https://github.com/OffchainLabs/prysm/releases/download/v7.2.0/beacon-chain-v7.2.0-linux-amd64 mv beacon-chain-v7.2.0-linux-amd64 beacon-chain mv validator-v7.2.0-linux-amd64 validator ``` Then stop the Prysm services: ```bash theme={null} sudo systemctl stop prysmvalidator && sudo systemctl stop prysmbeacon ``` Then make the software executable, copy into the bin folder, and re-secure permissions: ```bash theme={null} chmod +x beacon-chain chmod +x validator sudo cp beacon-chain /usr/local/bin sudo cp validator /usr/local/bin sudo chown -R prysmbeacon:prysmbeacon /var/lib/prysm/validator/ && sudo chown -R prysmvalidator:prysmvalidator /var/lib/prysm/validator/ ``` Next, restart services and check for errors, then clean up the files: ```bash theme={null} sudo systemctl daemon-reload sudo systemctl start prysmbeacon sudo systemctl start prysmvalidator sudo rm beacon-chain && sudo rm validator ``` ```bash theme={null} sudo systemctl status prysmbeacon # <-- Check for errors sudo journalctl -fu prysmbeacon.service | ccze # <-- Monitor sudo systemctl status prysmvalidator # <-- Check for errors sudo journalctl -fu prysmvalidator.service | ccze # <-- Monitor ``` Finally, check that you installed the right version. Note that this command may show you both the previous version and the current version. As long as both commands show the same data for the validator and beacon-chain software, you're all set: ```bash theme={null} journalctl --since -1hour -u prysmvalidator.service | grep version journalctl --since -1hour -u prysmbeacon.service | grep version ``` ### Appendix D Exiting a Validator As of May 2025, you can now use the [Ethereum Launchpad Validator Actions](https://launchpad.ethereum.org/en/validator-actions) to perform a validator exit via GUI. Note that you will need to be able to connect your withdrawal address to the site in order to proceed. If this is not possible or you would prefer to use the CLI method, please consult the [Prysm documentation](https://prysm.offchainlabs.com/docs/manage-validator/exiting-a-validator/). Otherwise, refer to Appendix G Launchpad Validator Actions. #### Removing Exited Validators from Your Node After a validator has fully exited and you've withdrawn your funds, you may want to remove its keys from the validator client. This will clean up your Grafana dashboard and stop tracking the exited validator. First, list your current validator accounts to find the pubkey of the validator you want to remove: ``` sudo /usr/local/bin/validator accounts list --wallet-dir=/var/lib/prysm/validator ``` Then, stop the validator service, delete the specific key, and restart: ``` sudo systemctl stop prysmvalidator sudo /usr/local/bin/validator accounts delete --wallet-dir=/var/lib/prysm/validator --delete-public-keys=0xYOUR_PUBKEY_HERE sudo systemctl start prysmvalidator ``` Replace `0xYOUR_PUBKEY_HERE` with the full public key of the exited validator you want to remove. You can delete multiple validators at once by separating pubkeys with commas (no spaces). **Note:** Only remove validators that have **fully exited** and whose funds have been withdrawn. Removing an active validator's keys will cause missed attestations and penalties. ### Appendix E Checking Sync Committee Duties Before performing maintenance (like updating Reth or Prysm), it is prudent to check upcoming sync committee duties (described up to [\~27 hours in advance](https://github.com/ethereum/consensus-specs/pull/2453)) so as not to be offline when you would otherwise be part of a committee. If any of the validators (as identified by the validator index numbers in the command) have upcoming duties, there will be additional output that says *"validator: 123511 found in next sync committee"*. If no such text is present, the validators in question do not have upcoming sync duties. Note that this requires the following script be set up and made executable: ``` #!/bin/sh # Set the URL of the local beacon node (Prysm default port) BEACON_NODE="http://localhost:3500" # Convert space-separated validator indices from command line arguments to a regex pattern VALIDATOR_LIST=$(echo "$@" | tr ' ' '|') # Function to convert epoch to Unix time epoch_to_time() { expr 1606824000 + \( $1 \* 384 \) } # Function to convert Unix time to epoch time_to_epoch() { expr \( $(date +%s) - 1606824000 \) / 384 } # Function to get committee member indices from the Beacon Node API get_committee() { URL="${BEACON_NODE}/eth/v1/beacon/states/finalized/sync_committees?epoch=$1" RESPONSE=$(curl -s -X GET "$URL") if [ -z "$RESPONSE" ]; then echo "Error fetching data from Beacon Node API." exit 1 fi echo "$RESPONSE" | sed -e 's/["]/''/g' | cut -d'[' -f2 | cut -d']' -f1 | tr ',' '\n' } # Function to search for validator indices in the committee search_committee() { COMMITTEE=$(get_committee $2) echo "$COMMITTEE" | grep -Ex $VALIDATOR_LIST \ | awk -v c=$1 '{print "validator:", $1, "found in", c, "sync committee"}' } # Function to display the epoch and its corresponding date display_epoch() { EPOCH_TIME=$(epoch_to_time $1) DATE=$(date -d @"$EPOCH_TIME" '+%Y-%m-%d %H:%M:%S') echo "epoch: $1 : $DATE <-- $2" } # Main operation curr_epoch=$(time_to_epoch) curr_start_epoch=$(expr $curr_epoch / 256 \* 256) next_start_epoch=$(expr $curr_start_epoch + 256) nextb1_start_epoch=$(expr $next_start_epoch + 256) display_epoch $curr_start_epoch "current sync committee start" display_epoch $next_start_epoch "next sync committee start" display_epoch $nextb1_start_epoch "next-but-one sync committee start" if [ "$#" -gt 0 ]; then search_committee "current" $curr_start_epoch search_committee "next" $next_start_epoch search_committee "next-but-one" $nextb1_start_epoch fi ``` You can get this set up easily via: ``` sudo nano check_sync_committee.sh ``` Then paste in the text of the script above, save and exit, followed by: ``` sudo chmod +x check_sync_committee.sh ``` to make the script executable. Also, run the following command to ensure you have prerequisite software set up: ``` sudo apt update && sudo apt install -y curl gawk sed coreutils ``` Run the following command, it will show something like the below: ``` ./check_sync_committee.sh 655262 655258 655260 655226 1296208 ``` [](../images/synccommittee.png) ### Appendix F MEV-Boost MEV-boost (Maximal Extractable Value) is software that allows stakers to access MEV, earning them higher block rewards by using open-source middleware (MEV-boost by Flashbots) that allows access to the competitive block-building market. See more information about proposer-builder separation [here](https://ethresear.ch/t/proposer-block-builder-separation-friendly-fee-market-designs/9725). There are trade-offs and choices to make in terms of which block-builders are used here, namely, *is the relay censoring aka OFAC compliant* and *is the relay ethical aka does it employ frontrunning and sandwich attack methods to increase MEV*. Endaoment and this guide do not support or otherwise condone the use of unethical or illegal MEV-boost practices. While Endaoment will be using this software to increase staking rewards, we are of the opinion that MEV-burn is a preferable system where MEV-boost is rendered useless *(in this system, any and all MEV is automatically burnt, as opposed to being rewarded to randomly chosen proposer based on their willingness to run ethical or unethical MEV)*. #### MEV Boost Set Up Let's get to installing! All commands below are based on the current version of MEV-boost as of 02.23.26, v1.12, but should be adjusted based on whatever the latest version of MEV-boost is [here](https://github.com/flashbots/mev-boost/releases). Simply scroll to the bottom of the linked releases page, and right click the .tar.gz file that contains `linux_amd64`. Copy that link and replace all file names and links below. `v1.12` adds relay multiplexing, which allows operators to configure validator mux groups with distinct relay sets, timeouts, and timing game settings via the global config file. If you run multiple validator sets, review the release details and config guidance before upgrading: [v1.12 changelog](https://github.com/flashbots/mev-boost/compare/v1.11...v1.12), [feature PR](https://github.com/flashbots/mev-boost/pull/826). Docker users should use `flashbots/mev-boost:1.12`. Create mevboost service account: ``` sudo useradd --no-create-home --shell /bin/false mevboost ``` Curl the MEV-boost build from the aforementioned link: ``` cd ~ curl -LO https://github.com/flashbots/mev-boost/releases/download/v1.12/mev-boost_1.12_linux_amd64.tar.gz ``` For additional security, you can verify the checksum against the checksum found on the releases link above: ``` sha256sum mev-boost_1.12_linux_amd64.tar.gz ``` Extract the files from the archive, copy to the /usr/local/bin directory, modify file names, and remove downloaded leftovers: ``` tar xvf mev-boost_1.12_linux_amd64.tar.gz sudo cp mev-boost /usr/local/bin rm mev-boost LICENSE README.md mev-boost_1.12_linux_amd64.tar.gz sudo chown mevboost:mevboost /usr/local/bin/mev-boost ``` Next, create and configure the service file: ``` sudo nano /etc/systemd/system/mevboost.service ``` And then paste the following data into the file: ``` [Unit] Description=mev-boost ethereum mainnet Wants=network-online.target After=network-online.target [Service] User=mevboost Group=mevboost Type=simple Restart=always RestartSec=5 ExecStart=/usr/local/bin/mev-boost \ -mainnet \ -min-bid {MinBid} \ -relay-check \ -relay {RelayAddress} \ -relay {RelayAddress} [Install] WantedBy=multi-user.target ``` **Make sure** to replace `{MinBid}` with the minimum MEV bid required to use the block builder (else, you will build the block locally). Format this value like `0.05`. **Make sure** to replace `{RelayAddress}` with the relay you would like to use to build your blocks for MEV. You can remove or add additional relay lines as needed - any listed relays will compete for the best MEV. Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start mevboost sudo systemctl status mevboost ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the mevboost service). Use the journal output to follow the progress or check for errors by running the following command: ``` sudo journalctl -fu mevboost | ccze ``` Finally, enable the service to automatically start on reboot: ``` sudo systemctl enable mevboost ``` #### Update BeaconChain and Validator Software In order to get the mevboost service to properly communicate with Prysm, we'll need to adjust both service files. Open the prysmbeacon service file: ``` sudo nano /etc/systemd/system/prysmbeacon.service ``` Add the following ExecStart command, depending on how you've formatted this file, you may need to add a backslash to delineate the lines: ``` --http-mev-relay=http://127.0.0.1:18550 ``` Press CTRL + X then Y then ENTER to save and exit. Open the prysmvalidator service file: ``` sudo nano /etc/systemd/system/prysmvalidator.service ``` Add the following ExecStart command, depending on how you've formatted this file, you may need to add a backslash to delineate the lines: ``` --enable-builder ``` Press CTRL + X then Y then ENTER to save and exit. Finally, reload and restart your services: ``` sudo systemctl daemon-reload sudo systemctl restart prysmbeacon prysmvalidator ``` Verify your logs look error-free and show use of the new MEV configurations. ``` journalctl -fu prysmbeacon | ccze journalctl -fu prysmvalidator | ccze ``` MEV-boost is now configured and set up! #### Update MEV Boost Before updating, always review the latest [MEV-boost release notes](https://github.com/flashbots/mev-boost/releases) for new requirements and any breaking changes. Typically, you need to ensure that you have the latest execution and consensus client versions installed in order for MEV-boost to work properly. First, stop the service: ``` sudo systemctl stop mevboost ``` The latest version of MEV-boost can be found [here](https://github.com/flashbots/mev-boost/releases). Simply scroll to the bottom of the linked releases page, and right click the .tar.gz file that contains `linux_amd64`. Copy that link and replace all file names and links below, which uses `v1.12` (latest as of 02.23.26). If you use Docker instead of a systemd binary install, pull `flashbots/mev-boost:1.12`. Curl the MEV-boost build from the aforementioned link: ``` cd ~ curl -LO https://github.com/flashbots/mev-boost/releases/download/v1.12/mev-boost_1.12_linux_amd64.tar.gz ``` Extract the files from the archive, copy to the /usr/local/bin directory, modify file names, remove downloaded leftovers, and start the service back up: ``` tar xvf mev-boost_1.12_linux_amd64.tar.gz sudo mv mev-boost /usr/local/bin rm LICENSE README.md mev-boost_1.12_linux_amd64.tar.gz sudo chown mevboost:mevboost /usr/local/bin/mev-boost sudo systemctl restart mevboost ``` Now, verify the service is running correctly and verify the correct version was installed: ``` sudo systemctl status mevboost mev-boost --version ``` You should see `active (running)` in green text and the current mev-boost version below, nice! ### Appendix G Launchpad Validator Actions As of May of 2025, the [Ethereum Launchpad](https://launchpad.ethereum.org/) upgraded the list of actions you can take using the UI beyond simply making deposits into validators. If you navigate to the [Validator Actions](https://launchpad.ethereum.org/en/validator-actions) page and connect the wallet that represents your **withdrawal address**, you'll be able to select a current validator and take a number of different actions with it depending on your withdrawal credentials (`0x00`, `0x01`, or `0x02`). * **Update account to compounding** - this process converts your existing `0x00` or `0x01` withdrawal credentials to `0x02` (new in the Pectra hard fork). Having `0x02` withdrawal credentials allows you to increase your effective balance from 32 ETH to 2048 ETH so that you need not spin up a new validator for each tranche. When you make this change, regular withdrawals will cease, and partial withdrawals may be requested using this UI (see "Request partial withdrawal" below). In addition, rewards will be based on your current validator balance and increase as that balance increases, essentially compounding your rewards. * **Add funds to your validator** - only available to validators using `0x02` withdrawal credentials (see "Update account to compounding" above), this action allows you to add additional ETH to your validator that is currently held elsewhere. This allows compounding rewards, and has a minimum deposit amount of 1 ETH. * **Request partial withdrawal** - only available to validators using `0x02` withdrawal credentials (see "Update account to compounding" above), this action allows you to withdraw any portion of staked ETH over 32 to your withdrawal address. * **Absorb another validator** - only available to validators using `0x02` withdrawal credentials (see "Update account to compounding" above), this action allows you to transfer the entire balance of another validator you control to the one you have selected, consolidating those validators into one. * **Migrate funds** - this action allows you to transfer the entire balance of the validator you have selected to another one of your validators. The target consolidation validator must be using `0x02` withdrawal credentials (see "Update account to compounding" above). * **Exit validator** - this action allows you to perform a full exit of your validator and withdraw the entire balance thereof. ### Appendix H Resetting Reth Database In rare cases, you may need to reset the Reth database and re-sync from scratch. Common reasons include: * **Database corruption** from power outages or hardware failures * **Unclean shutdowns** that left the database in an inconsistent state * **Storage format upgrades** when a new Reth version requires a fresh sync * **Disk space recovery** if the database has grown unexpectedly large **This will delete all synced execution layer data** and require a full re-sync, which can take hours to days depending on your sync method. Only proceed if other troubleshooting steps have failed. #### Check If a Reset Is Necessary Before resetting, try restarting Reth first—many issues resolve with a clean restart: ``` sudo systemctl restart reth sudo journalctl -fu reth | ccze ``` If you see persistent errors like `Database corrupted`, `Checksum mismatch`, or `Failed to open database`, a reset may be necessary. #### Reset the Database First, stop the Reth service: ``` sudo systemctl stop reth ``` Remove all data from the Reth data directory: ``` sudo rm -rf /var/lib/reth/* ``` #### Re-sync Options **Option 1: Sync from Genesis (Slower)** Simply restart Reth and it will begin syncing from genesis using the `--minimal` flag configuration: ``` sudo systemctl start reth ``` This takes 1-3 days but results in a very small database (\~224 GB with Storage V2). **Option 2: Use Reth Snapshot (Faster, Recommended)** For faster recovery, download a minimal snapshot using `reth download`: ``` sudo -u reth /usr/local/bin/reth download --chain mainnet --minimal --datadir /var/lib/reth --resumable ``` This reduces re-sync time to a few hours. The `--resumable` flag allows the download to pick up where it left off if interrupted. See [Part 1 snapshot instructions](/validators/validator-sop-part1-setup#optional-use-reth-snapshot-for-faster-initial-sync) for full details. #### Monitor Re-sync Progress Watch the logs to monitor sync progress: ``` sudo journalctl -fu reth | ccze ``` Check sync status via JSON-RPC: ``` curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' http://localhost:8545 ``` A response of `{"jsonrpc":"2.0","id":1,"result":false}` indicates sync is complete. **Note:** Your beacon chain will show errors until Reth finishes syncing. This is expected—once Reth catches up, the beacon chain will resume normal operation. # Validator SOP Part 3: Emergency Action Plans Source: https://validators.endaoment.org/validators/validator-sop-part3-emergency ## Table of Contents * [Emergency Action Plan α Validator Unrecoverable Error](#emergency-action-plan-α-validator-unrecoverable-error) * [Emergency Action Plan β Validator Operator Missing](#emergency-action-plan-β-validator-operator-missing) * [Emergency Action Plan γ Validator Operator Missing and No Validator Access](#emergency-action-plan-γ-validator-operator-missing-and-no-validator-access) ### Emergency Action Plan α Validator Unrecoverable Error In the instance of an unrecoverable error, we will need to accomplish two main goals: 1. Ensure that the old machine is no longer connected to the internet, and if possible, separate (and destroy or reformat) the SSD. We must take every precaution to ensure that it is all but impossible to run the same validator keystores from multiple nodes at once. 2. Build and configure a new node, and import the validator keystore files that were previously used on the now defunct node. Importing validator keys to a new set is as simple as saving those keys on a USB drive as described in this guide, and keeping that USB drive in a safe place separate from the location of the Node. In order to import the keys, simply follow this guide (part1) and skip step 1 as described as you do not need to re-generate the keys - no deposit is required. Note: When moving validator keys from one node to another it is **essential** that you both verify that the first node is no longer functional and wait for that node to miss at least 10 attestations. If the validator keys have been lost, that's ok! They can be regenerated by going through step 1 of the SOP and using a slightly modified command: ``` sudo ./deposit existing-mnemonic --num_validators {#_Validators} --chain mainnet --eth1_withdrawal_address {YourWithdrawalAddress} --validator_start_index 0 ``` Following re-generation of the keystore files, simply continue to follow the guide through Step 10. After importing the validator keystore files to the newly configured Node, you're all set! ### Emergency Action Plan β Validator Operator Missing This is actually a bit trickier of a situation than in EAP α. As previously described, it is **critical** that the same validator keystore files are not used by two validators at the same time. In the instance that the validator operator is abducted by aliens, etc., the first thing to determine is if the validators are still functional. If not, revert to EAP α, but if so, we need to gain access to the validators physically. Once access is gained, an evaluation should be done to determine if we should: 1. Destroy the Node and revert to EAP α. 2. Move the validator to the custody of a new validator operator, and continue using existing hardware with current configuration. Note that this requires access to the validator operator's variable documentation. If the variables are not accessible but the Node is, see `1` above. While option 2 above may seem preferable to save on the cost of a new build, option 1 is recommended in order to establish full reliable control over the Node. ### Emergency Action Plan γ Validator Operator Missing and No Validator Access What do we do in the event that the validator operator is missing and access to the validator is impossible? If we have backups of the keystore files (or the mnemonic with which they were generated, used to regenerate them) we issue an exit command from a different Node without getting slashed. 1. Build another Node, and ensure the prysmvalidator service has been turned off, **this is critical as otherwise you will get slashed upon activation**. 2. Check the status of the prysmbeacon service to ensure the Node is fully synced. **Do not continue unless your Node is fully synced.** 3. Import the keystore files as per step 10 in part 1. 4. Issue the voluntary exit command as per Appendix D (Exiting a Validator) or Appendix G (Launchpad Validator Actions) in Part 2. 5. Power down the backup Node you just set up after issuing the command to minimize slashing risks. Note that the exit is queued - risk is not fully mitigated until the exit is completed to the withdrawal address. # Validator SOP Part 4: Monitoring and Dashboard Creation Source: https://validators.endaoment.org/validators/validator-sop-part4-monitoring ## Table of Contents * [Abstract](/validators/validator-sop-part4-monitoring#abstract) * [Step 1 Adjust Firewall](/validators/validator-sop-part4-monitoring#step-1-adjust-firewall) * [Step 2 Install Prerequisite Software](/validators/validator-sop-part4-monitoring#step-2-install-prerequisite-software) * [Step 3 Install Reporting Software](/validators/validator-sop-part4-monitoring#step-3-install-reporting-software) * [Prometheus](/validators/validator-sop-part4-monitoring#prometheus) * [Node Exporter](/validators/validator-sop-part4-monitoring#node-exporter) * [JSON Exporter](/validators/validator-sop-part4-monitoring#json-exporter) * [Ethereum Metrics Exporter](/validators/validator-sop-part4-monitoring#ethereum-metrics-exporter) * [Grafana](/validators/validator-sop-part4-monitoring#grafana) * [Step 4 Configure Ethereum Clients](/validators/validator-sop-part4-monitoring#step-4-configure-ethereum-clients) * [Step 5 Configure Reporting Software](/validators/validator-sop-part4-monitoring#step-5-configure-reporting-software) * [Configure JSON Exporter](/validators/validator-sop-part4-monitoring#configure-json-exporter) * [Configure Prometheus Rules](/validators/validator-sop-part4-monitoring#configure-prometheus-rules) * [Configure Ethereum Metrics Exporter](/validators/validator-sop-part4-monitoring#configure-ethereum-metrics-exporter) * [Step 6 Configure Prometheus Jobs and Targets](/validators/validator-sop-part4-monitoring#step-6-configure-prometheus-jobs-and-targets) * [Prometheus Jobs](/validators/validator-sop-part4-monitoring#prometheus-jobs) * [Base Targets](/validators/validator-sop-part4-monitoring#base-targets) * [Validator Public Key Targets](/validators/validator-sop-part4-monitoring#validator-public-key-targets) * [Restart All Services](/validators/validator-sop-part4-monitoring#restart-all-services) * [Step 7 Set up Dashboard](/validators/validator-sop-part4-monitoring#step-7-set-up-dashboard) ## Abstract This document will detail the process for setting up a Grafana monitoring dashboard for an Ethereum validator. This guide assumes you followed [Validator SOP Part 1: Set Up](/validators/validator-sop-part1-setup). Please note that dashboards are highly customizable, and as such small changes can be made to either the reporting software or the dashboard itself to accommodate specific use cases. This guide is written with the initial Endaoment node set up in mind, a node with five validators utilizing Reth/Prysm. While the majority of things covered by this guide do not touch the underlying services that run the validator, caution should be exercised as there are some changes that must be made to validator/beacon-chain service files. ### Step 1 Adjust Firewall Before we can get started, we need to open ports in the UFW firewall to allow Prometheus and Grafana listening ports: ``` sudo ufw allow 3000/tcp sudo ufw allow 9090/tcp ``` Next, check to ensure the rules were properly applied: ``` sudo ufw status numbered ``` ### Step 2 Install Prerequisite Software This guide will utilize the `make` utility as well as `go`, and additionally relies on the Stake Local GitHub Repository. We'll get that all set up before we start installing the reporting software. First, let's run a quick update, ensure we can use APT via HTTPS, and ensure we have common utilities like wget installed, and ensure they are automatically updated: ``` sudo apt-get update sudo apt-get install apt-transport-https sudo apt-get install software-properties-common wget sudo apt-mark auto apt-transport-https sudo apt-mark auto software-properties-common sudo apt-mark auto wget ``` Install make: ``` sudo apt-get install make ``` Check to ensure it was properly installed: ``` make --version ``` Install go (version 1.23+ required for ethereum-metrics-exporter): ``` cd wget https://go.dev/dl/go1.23.2.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.23.2.linux-amd64.tar.gz sudo ln -s /usr/local/go/bin/go /usr/bin/go ``` Check to ensure it was properly installed: ``` go version ``` Clean up installation files: ``` rm go1.23.2.linux-amd64.tar.gz ``` ### Step 3 Install Reporting Software We'll now install Grafana, Prometheus, Node\_Exporter and JSON\_exporter software, that will all work together to generate our dashboard. #### Prometheus Download the latest release of Prometheus by going [here](https://prometheus.io/download/) and copying the download link for the latest linux-amd64.tar.gz file. Please be sure to replace the link below using the current version. Download the repository, extract the files, and move them to the /bin: ``` cd ~ curl -LO https://github.com/prometheus/prometheus/releases/download/v3.8.1/prometheus-3.8.1.linux-amd64.tar.gz tar xvf prometheus-3.8.1.linux-amd64.tar.gz sudo cp prometheus-3.8.1.linux-amd64/prometheus /usr/local/bin/ sudo cp prometheus-3.8.1.linux-amd64/promtool /usr/local/bin/ ``` Modify file names and clean up: ``` sudo cp -r prometheus-3.8.1.linux-amd64/consoles /etc/prometheus sudo cp -r prometheus-3.8.1.linux-amd64/console_libraries /etc/prometheus rm prometheus-3.8.1.linux-amd64.tar.gz rm -r prometheus-3.8.1.linux-amd64 ``` Set up an account for the services to run under (note: this type of account cannot log into the server): ``` sudo useradd --no-create-home --shell /bin/false prometheus ``` Set up the directories for prometheus: ``` sudo mkdir -p /etc/prometheus /var/lib/prometheus ``` Create a configuration file for prometheus: ``` sudo nano /etc/prometheus/prometheus.yml ``` Copy and paste text into this configuration file so it matches [prometheus.yml](../.imgs/prometheus.yml). Press CTRL + X then Y then ENTER to save and exit. Set directory permissions for prometheus to modify the directories: ``` sudo chown -R prometheus:prometheus /etc/prometheus sudo chown -R prometheus:prometheus /var/lib/prometheus ``` Create a systemd service config file to configure the service: ``` sudo nano /etc/systemd/system/prometheus.service ``` Paste the following text into the configuration file: ``` [Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] Type=simple User=prometheus Group=prometheus Restart=always RestartSec=5 ExecStart=/usr/local/bin/prometheus \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/var/lib/prometheus \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries [Install] WantedBy=multi-user.target ``` Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start prometheus sudo systemctl status prometheus ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the Prometheus service). Use the journal output to follow the progress or check for errors by running the following command: ``` sudo journalctl -fu prometheus | ccze ``` Finally, enable the service to automatically start on reboot: ``` sudo systemctl enable prometheus ``` **Troubleshooting Note:** If Prometheus fails to start with "address already in use" errors, you may have a snap-installed Prometheus conflicting with the manual installation. Check with `snap services` and if Prometheus is installed via snap, remove it with `sudo snap remove prometheus`. #### Node Exporter Download the latest release of Node Exporter by going [here](https://prometheus.io/download/#node%5Fexporter) and copying the download link for the latest linux-amd64.tar.gz file. Please be sure to replace the link below using the current version. Download the repository, extract the files, and move them to the /bin: ``` cd ~ wget https://github.com/prometheus/node_exporter/releases/download/v1.10.2/node_exporter-1.10.2.linux-amd64.tar.gz tar xvf node_exporter-1.10.2.linux-amd64.tar.gz sudo cp node_exporter-1.10.2.linux-amd64/node_exporter /usr/local/bin/ ``` Clean up: ``` rm node_exporter-1.10.2.linux-amd64.tar.gz rm -r node_exporter-1.10.2.linux-amd64 ``` Set up an account for the services to run under (note: this type of account cannot log into the server): ``` sudo useradd --no-create-home --shell /bin/false node_exporter ``` Create a configuration file for node\_exporter: ``` sudo nano /etc/systemd/system/node_exporter.service ``` Copy and paste the following into this configuration file: ``` [Unit] Description=Node Exporter Wants=network-online.target After=network-online.target [Service] User=node_exporter Group=node_exporter Type=simple Restart=always RestartSec=5 ExecStart=/usr/local/bin/node_exporter [Install] WantedBy=multi-user.target ``` Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start node_exporter sudo systemctl status node_exporter ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the Node Exporter service). Use the journal output to follow the progress or check for errors by running the following command: ``` sudo journalctl -fu node_exporter | ccze ``` Finally, enable the service to automatically start on reboot: ``` sudo systemctl enable node_exporter ``` **Note:** Some hardware configurations (particularly VMs or systems without thermal sensors) may trigger recurring `thermal_zone` collector errors in the logs. These warnings are harmless and can be safely ignored. If desired, you can disable this collector by adding `--no-collector.thermal_zone` to the `ExecStart` line in the service file. #### JSON Exporter Download the latest release of JSON Exporter by going [here](https://github.com/prometheus-community/json%5Fexporter/releases) and copying the download link for the latest linux-amd64.tar.gz file. Please be sure to replace the link below using the current version. Download the repository and move them to /git: ``` mkdir ~/git cd git git clone --depth 1 --branch v0.7.0 https://github.com/prometheus-community/json_exporter.git cd ~ ``` Build json\_exporter from the source code you just downloaded: ``` cd git/json_exporter make build ``` Copy the executable file to the /bin, and check to ensure it installed correctly (should show the current version): ``` sudo cp json_exporter /usr/local/bin/ /usr/local/bin/json_exporter --version --help ``` Clean up time: ``` cd ~ rm -rf ~/git/json_exporter ``` Set up an account for the services to run under (note: this type of account cannot log into the server): ``` sudo useradd --no-create-home --shell /bin/false json_exporter ``` Create a directory and the configuration file for json\_exporter: ``` sudo mkdir /etc/json_exporter sudo nano /etc/systemd/system/json_exporter.service sudo chown -R json_exporter:json_exporter /etc/json_exporter ``` Copy and paste the following into this configuration file: ``` [Unit] Description=JSON Exporter [Service] Type=simple Restart=always RestartSec=5 User=json_exporter ExecStart=/usr/local/bin/json_exporter --config.file /etc/json_exporter/json_exporter.yml [Install] WantedBy=multi-user.target ``` Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start json_exporter sudo systemctl status json_exporter ``` It may not say "active (running)" in green text, as `'/etc/json_exporter/json_exporter.yml' does not exist`, this is expected. We will configure the YML file in step 5 below. Use the journal output to follow the progress or check for errors by running the following command: ``` sudo journalctl -fu json_exporter | ccze ``` Running this command will confirm the above, as you'll see the error. Finally, enable the service to automatically start on reboot: ``` sudo systemctl enable json_exporter ``` #### Ethereum Metrics Exporter Clone and build the latest version of the Ethereum Metrics Exporter from source: ```bash theme={null} cd ~/git git clone https://github.com/ethpandaops/ethereum-metrics-exporter.git cd ethereum-metrics-exporter go build -o ethereum-metrics-exporter . ``` Copy the executable to /usr/local/bin and verify it works: ```bash theme={null} sudo cp ethereum-metrics-exporter /usr/local/bin/ sudo chmod +x /usr/local/bin/ethereum-metrics-exporter /usr/local/bin/ethereum-metrics-exporter --help ``` Clean up the build files: ```bash theme={null} cd ~ rm -rf ~/git/ethereum-metrics-exporter ``` Set up an account for the services to run under (note: this type of account cannot log into the server): ``` sudo useradd --no-create-home --shell /bin/false eth-metrics ``` Create a directory and the configuration file for eth\_exporter: ``` sudo mkdir -p /var/local/lib/eth-metrics /etc/eth-metrics sudo nano /etc/systemd/system/eth-metrics.service ``` Copy and paste the following into this configuration file: ``` [Unit] Description=Ethereum Metrics Exporter Wants=network-online.target After=network-online.target [Service] User=eth-metrics Group=eth-metrics Type=simple Restart=always RestartSec=5 ExecStart=/usr/local/bin/ethereum-metrics-exporter \ --config /etc/eth-metrics/eth-metrics.yml \ --metrics-port 9095 [Install] WantedBy=multi-user.target ``` Press CTRL + X then Y then ENTER to save and exit. Reload systemd to reflect the changes and start the service. Check the status to make sure it’s running correctly: ``` sudo systemctl daemon-reload sudo systemctl start eth-metrics sudo systemctl status eth-metrics ``` It may not say "active (running)" in green text, as `'/etc/eth-metrics/eth-metrics.yml' does not exist`, this is expected. We will configure the YML file in step 5 below. Use the journal output to follow the progress or check for errors by running the following command: ``` sudo journalctl -fu eth-metrics | ccze ``` Running this command will confirm the above, as you'll see the error. Finally, enable the service to automatically start on reboot: ``` sudo systemctl enable eth-metrics ``` #### Grafana Download the Grafana GPG key and add Grafana to the APT sources: ``` wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main" ``` You may get a warning about apt-key being deprecated, it is ok to ignore this warning. Refresh the apt cache: ``` sudo apt install grafana && sudo apt update ``` You may need to press `Y` to continue. Install Grafana: ``` sudo apt install -y grafana ``` Start Grafana, then check the status to make sure it’s running correctly: ``` sudo systemctl start grafana-server sudo systemctl status grafana-server ``` If it says "active (running)" in green text, you've done it! Press Q to quit (this will not affect the Grafana service) Use the journal output command to check for errors: ``` sudo journalctl -fu grafana-server | ccze ``` Press CTRL+ C to exit (will not affect the Grafana service). Next, enable the Grafana service to automatically start on reboot: ``` sudo systemctl enable grafana-server ``` Now that you have Grafana up and running, you can navigate to `https://{yourserverIP}:3000` to access the Grafana interface. You should see a login screen. Enter `admin` for both the username and password. You'll be immediately prompted to change your password, which you should, and note on your variables document. Time to configure the Grafana data source! Navigate to `https://{yourserverIP}:3000/datasources`and click on `Add data source`. Choose Prometheus, enter `http://localhost:9090` for the URL, and then click on Save and Test. If you see `Successfully queried the Prometheus API`, you're all set! ### Step 4 Configure Ethereum Clients Each consensus and execution client must have the appropriate metrics services and APIs enabled and accessible to the reporting software. As a reminder, the below instructions assume Prysm/Reth. Adjust your Prysm Beacon service file: ``` sudo nano /etc/systemd/system/prysmbeacon.service ``` Add the following flag at the bottom of the `ExecStart` list, note that you may need to add a`\` at the end of the previous line, depending on how your file is formatted. `--monitoring-host 0.0.0.0` Press CTRL + X then Y then ENTER to save and exit. Adjust your Prysm Validator service file: ``` sudo nano /etc/systemd/system/prysmvalidator.service ``` Add the following flag at the bottom of the `ExecStart` list, note that you may need to add a`\` at the end of the previous line, depending on how your file is formatted. `--monitoring-host 0.0.0.0` Press CTRL + X then Y then ENTER to save and exit. Adjust your Reth service file: ``` sudo nano /etc/systemd/system/reth.service ``` Update the `ExecStart` section to add monitoring support. Change the `--http.api` line and add the `--metrics` flag: ``` ExecStart=/usr/local/bin/reth node \ --chain mainnet \ --minimal \ --datadir /var/lib/reth \ --authrpc.jwtsecret /var/lib/jwtsecret/jwt.hex \ --authrpc.addr 127.0.0.1 \ --authrpc.port 8551 \ --port 30303 \ --discovery.port 30303 \ --enable-discv5-discovery \ --discovery.v5.port 30305 \ --max-outbound-peers 35 \ --max-inbound-peers 35 \ --http \ --http.addr 127.0.0.1 \ --http.api eth,net,web3,txpool \ --metrics 127.0.0.1:9001 ``` Press CTRL + X then Y then ENTER to save and exit. Finally, restart everything you just adjusted. Please note that if the validator is attesting, you should first ensure there are no upcoming attestations or proposals, as there will be \~30 seconds of downtime. ``` sudo systemctl daemon-reload sudo systemctl restart reth sudo systemctl restart prysmbeacon sudo systemctl restart prysmvalidator ``` You should then check the status of each of the services you just restarted to ensure that changing the configuration files haven't messed with anything: ``` sudo systemctl status reth sudo systemctl status prysmbeacon sudo systemctl status prysmvalidator ``` After reviewing each one, press CTRL + C to exit, which will trigger the next status to show. ### Step 5 Configure Reporting Software With all software installed and your Ethereum clients configured, we can now configure all the software to talk to each other properly. Remember those errors from before? Now we fix. #### Configure JSON Exporter Edit the configuration file for json exporter: ``` sudo nano /etc/json_exporter/json_exporter.yml ``` Copy and paste the contents of [this file](../.imgs/json_exporter.yml) into that file. Then press CTRL + X followed by Y to save. #### Configure Prometheus Rules Edit the rules.d file for prometheus: ``` sudo mkdir -p /etc/prometheus/rules.d/ sudo nano /etc/prometheus/rules.d/stakelocal_rules.yml ``` Copy and paste the contents of [this file](../.imgs/stakelocal_rules.yml) into that file. Then press CTRL + X followed by Y to save. #### Configure Ethereum Metrics Exporter Edit the configuration file for eth-metrics: ``` sudo nano /etc/eth-metrics/eth-metrics.yml ``` Copy and paste the following into that file: ``` consensus: enabled: true url: "http://127.0.0.1:3500" name: "Prysm Beacon" execution: enabled: true url: "http://127.0.0.1:8545" name: "Reth" modules: - "eth" - "net" - "web3" - "txpool" pair: enabled: true ``` **Important:** Make sure the consensus URL uses `http://` (not `https://`) for local connections. Then press CTRL + X followed by Y to save. ### Step 6 Configure Prometheus Jobs and Targets A Prometheus `job` represents a set of rules for processing metrics from similar data sources/targets. A Prometheus target is a single data source from which Prometheus should collect and process data using the rules of a specific job. Prometheus jobs are collections of targets for which the same rules are applied. Here, we'll be setting up the Prometheus jobs that feed data to the dashboard. #### Prometheus Jobs Since we've already edited the Prometheus configuration file, we simply need to check that the rules are being read: ``` sudo promtool check config /etc/prometheus/prometheus.yml ``` While there will be a number of warnings (expected), you should see `SUCCESS: 46 rules found` at the bottom of the returned text. #### Base Targets Before we can do specific configuring, we will need to copy in an archive of all target configuration files. To save time, all 24 files come pre-edited as needed, you're welcome. Create the folder to store things in: ``` sudo mkdir -p /etc/prometheus/files_sd ``` Download the stakelocal configuration files directly from the GitHub repository: ```bash theme={null} cd ~ sudo apt install unzip wget https://github.com/endaoment/endaoment-validator/archive/refs/heads/main.zip unzip main.zip ``` Copy the stakelocal folder to the correct location: ```bash theme={null} sudo mkdir -p /etc/prometheus/files_sd/stakelocal/ sudo cp -r endaoment-validator-main/.imgs/stakelocal/* /etc/prometheus/files_sd/stakelocal/ ``` Clean up the downloaded files: ```bash theme={null} rm -rf endaoment-validator-main main.zip ``` Finally, check to ensure things were copied correctly (you should see a folder called stakelocal): ``` ls /etc/prometheus/files_sd/ ``` #### Validator Public Key Targets Edit the validator configuration file to include a list of comma separated public keys of your validator(s): ``` sudo nano /etc/prometheus/files_sd/stakelocal/validators.yml ``` Copy and paste the following into that file: ``` - targets: [ "{validators}" ] labels: network: 'Mainnet' host: 'Default Host' service: 'Prysm Beacon' group: 'Prysm/Reth' instance: '127.0.0.1:3500' explorer: 'beaconcha.in' ``` Make sure to replace `{validators}` with a comma separated list of your validator public keys or indices (discoverable via Beaconcha.in if unknown). **Important:** Each validator key must be enclosed in quotes, in the style of `-targets: [ "0x123...", "0x456...", ... "0xXYZ..." ]`. If the consensus client being queried is not on the local host or uses a non-standard port, update those on the `instance` line. Then press CTRL + X followed by Y to save. #### Restart All Services We just made a lot of changes to many reporting services and their configuration files, let's now restart them and check that they've started up correctly. Restart all reporting services: ``` sudo systemctl daemon-reload sudo systemctl restart prometheus grafana-server node_exporter json_exporter eth-metrics ``` Now, check the status of each of these services: ``` sudo systemctl status prometheus grafana-server node_exporter json_exporter eth-metrics ``` If you need additional detail, you can check the logs: ``` sudo journalctl -fu prometheus | ccze sudo journalctl -fu grafana-server | ccze sudo journalctl -fu node_exporter | ccze sudo journalctl -fu json_exporter | ccze sudo journalctl -fu eth-metrics | ccze ``` ### Step 7 Set up Dashboard Log into your Grafana instance `({YourIp}:3000)` using the password you set up earlier. In the menu, click on Dashboard > New > Import. Copy and paste [this file](../.imgs/grafana-dashboard.json) into the `Import via dashboard JSON model` area and click `Load`. On the next screen, give your dashboard a unique name, feel free to change the UUID, and click `Import`. ⚠️ **You're going to see a lot of errors, that is expected!** ⚠️ These errors are because we need to update the data source for all panels you just imported. Head back to the data sources menu (Home > Connection > Data sources) and click on Prometheus - ensure the connection is set to [http://localhost:9090](http://localhost:9090) with `No Authentication`, then scroll down and click `Save and Test`. Next, click on Dashboards, and click `New` then `Import` (we will delete this later), use the import code `3662`, then click load, select the prometheus data source, then import. Navigate to that dashboard (it should automatically open), click edit and head to settings, and then click on JSON Model. Copy the UUID contained in line 19. This is your data source's UUID! After ensuring you have the UUID copied, You can head to Dashboards and delete all additional ones we just created. Just make sure you hang onto that UUID. Armed with the correct UUID, let's fix the validator dashboard. Click on Dashboards at the top of the screen, and navigate back to your dashboard. From there, click edit and head to settings, and then click on JSON Model. Press CTRL + F and click the down arrow to prepare a find and replace. Paste your data sources UUID into the Replace section, and `YOURUUID` into the Find section (note this is the UUID in line 37). Finally, click the replace all button to update your dashboard. To save, FIRST click `save changes` at the bottom and THEN `save dashboard` at the top of the screen, and finally click `save`. IT'S TIME! Head back to your dashboard and witness a thing of beauty. Well done kid.