Validator SOP Part 3: Emergency Action Plans

​

Table of Contents

• Emergency Action Plan α Validator Unrecoverable Error
• Emergency Action Plan β Validator Operator Missing
• Emergency Action Plan γ Validator Operator Missing and No Validator Access

​

Emergency Action Plan α Validator Unrecoverable Error

In the instance of an unrecoverable error, we will need to accomplish two main goals:

1. Ensure that the old machine is no longer connected to the internet, and if possible, separate (and destroy or reformat) the SSD. We must take every precaution to ensure that it is all but impossible to run the same validator keystores from multiple nodes at once.

2. Build and configure a new node, and import the validator keystore files that were previously used on the now defunct node.

Importing validator keys to a new set is as simple as saving those keys on a USB drive as described in this guide, and keeping that USB drive in a safe place separate from the location of the Node. In order to import the keys, simply follow this guide (part1) and skip step 1 as described as you do not need to re-generate the keys - no deposit is required.

Note: When moving validator keys from one node to another it is essential that you both verify that the first node is no longer functional and wait for that node to miss at least 10 attestations.

If the validator keys have been lost, that’s ok! They can be regenerated by going through step 1 of the SOP and using a slightly modified command:

sudo ./deposit existing-mnemonic --num_validators {#_Validators} --chain mainnet --eth1_withdrawal_address {YourWithdrawalAddress} --validator_start_index 0

Following re-generation of the keystore files, simply continue to follow the guide through Step 10. After importing the validator keystore files to the newly configured Node, you’re all set!

​

Emergency Action Plan β Validator Operator Missing

This is actually a bit tricker of a situation than in EAP α. As previously described, it is critical that the same validator keystore files are not used by two validators at the same time. In the instance that the validator operator is abducted by aliens, etc., the first thing to determine is if the validators are still functional. If not, revert to EAP α, but if so, we need to gain access to the validators physically. Once access is gained, an evaluation should be done to determine if we should:

1. Destroy the Node and revert to EAP α.

2. Move the validator to the custody of a new validator operator, and continue using existing hardware with current configuration. Note that this requires access to the validator operator’s variable documentation. If the variables are not accessible but the Node is, see 1 above.

While option 2 above may seem preferable to save on the cost of a new build, option 1 is recommended in order to establish full reliable control over the Node.

​

Emergency Action Plan γ Validator Operator Missing and No Validator Access

What do we do in the event that the validator operator is missing and access to the validator is impossible? If we have backups of the keystore files (or the mnemonic with which they were generated, used to regenerate them) we issue an exit command from a different Node without getting slashed.

1. Build another Node, and ensure the prysmvalidator service has been turned off, this is critical as otherwise you will get slashed upon activation.

2. Check the status of the prysmbeacon service to ensure the Node is fully synced. Do not continue unless your Node is fully synced.

3. Import the keystore files as per step 10 in part 1.

4. Issue the voluntary exit command as per Appendix E.

5. Power down the back up Node you just set up after issuing the command to minimize slashing risks. Note that the exit is queued - risk is not fully mitigated until the exit is completed to the withdrawal address.