How to Manage a Watcher

By the end of this guide you should know how to manage your Watcher installed on VPS or a dedicated server. The guide is useful for enterprise and individual clients who want to know how to manage their Watcher and receive notifications about its status, byzantine events, and alarms.
Goals
You should use this guide if you need to accomplish one of the following goals:
Monitor the uptime of your Watcher's server.
Automate regular Watcher status check.
Receive important alarms from the Watcher.
Know how to work with Watcher containers.
Prerequisites
1.Active Watcher running on the VPS.
2.Basic knowledge of REST APIs.
3.Basic knowledge of Docker tooling.
1. Working with Docker Containers
One of the ways to run a Watcher is using Docker containers. If you're new to Docker, these are the commands that should help you to solve the most common problems:
Start/Stop/Restart/Update Docker Containers
docker-compose -f docker-compose-watcher.yml start
Stop Only Watcher Containers
docker-compose -f docker-compose-watcher.yml stop
Stop All Docker Containers
docker stop $(docker ps -aq)
Restart Only Watcher Containers
docker-compose -f docker-compose-watcher.yml restart
Restart All Docker Containers
docker restart $(docker ps -aq)
docker pull omisego/watcher:latest && docker pull omisego/watcher_info:latest
2. Server Uptime Monitoring
Watcher monitoring is one of the most important processes you need to set up if you're planning to work with the OMG Network over the long term. Receiving alarms and status changes can help to provide more accurate data for the services and applications you're building using the OMG Network, understand how reliable is your VPS provider, respond faster and more effectively to potential byzantine events on the network.
2.1 Status Cake
To monitor the status of your server, you can use Pingdom, Status Cake or similar software that provides website/VPS monitoring functionality. If you're using Status Cake, go to the MONITORING > Uptime Monitoring section, and select New Uptime Test. Choose the TCP test type and fill in the required values, including your server's IP address, port, test name, and contact group that contains details about users who will receive notifications.
Status Cake provides multiple ways to receive notifications, such as Datadog, Slack, Discord, Telegram, OpsGenie, etc. You can set your preferred method in the ALERTING > Integrations section.
3. Status Monitoring
Running the OMG Network's Watcher implies that you're running both Watcher and Watcher Info services that will serve separate API endpoints and provide different access levels to the network. Most of the time you'll be using the Watcher Info service but you should monitor both.
It is possible for Watcher and Watcher Info to become inactive, even if its server is up. This can happen due to sudden configuration changes, internet, or hardware issues with your VPS provider.
There are a few ways to monitor the status of these services. The simplest way is to send corresponding API requests on a certain schedule. This guide demonstrates Datadog as one of the tools you can use to monitor various parts of your infrastructure.
3.1 Watcher Status
To monitor Watcher status, use the following steps:
1.Create Datadog account.
2.Choose Ubuntu as OS for Datadog Agent.
3.Log in to the Watcher's server from your terminal:
ssh $USER@$REMOTE_SERVER -p $PORT
$USER - the name of the user with root privileges used to log into the remote server. Default: root.
$REMOTE_SERVER - an ip address of your remote server.
$PORT - a port used to connect to the server. Default: 22.
1.Paste the command from New installation to your server's terminal and press Enter:
The process may take a few minutes. After a successful installation on the server, your Datadog account will be redirected to the dashboard.
1.Go to UX Monitoring > Synthetic Tests. Click Get Started and select the New API Test.
2.Fill in the following values and press Test URL:
URL method: POST
URL path: http://$REMOTE_SERVER:7434/status.get
Name: some indication about this specific endpoint, e.g. Watcher Status
Environment: environment name/identifier
Locations: locations you want your API to be tested from
$REMOTE_SERVER - an IP address of your remote server.
1.Specify the test frequency.
Test frequency defines how often the API should be called. A general suggestion is 5-15 minutes.
1.Define assertions.
Assertions are conditions required for a test to pass. Our main condition is for a Watcher to return success status. To do that, select the success field in the BODY result section. This will add the corresponding assertion as follows:
1.Define alert conditions.
Alert conditions help to set the number of failed tests, locations, retries needed to notify the team.
1.Set up notifications.
You can notify your team via email, Slack, Jira, Webhook, or dozens of other tools. Check Datadog Notifications for more info.
1.Monitor the result.
You can monitor alerts, as well the status of your APIs in the Events or UX Monitoring > Synthetic Tests sections.
3.2 Watcher Info Status
Setting up the Watcher Info status check has the same approach as described above. When creating a new synthetic test, replace the Define request as follows:
http:
Then, define assertions with success:true and follow the rest of the steps.
How to Manage a VPS
Security
Last modified 2yr ago