In this blog post, we will take a look at how we can manually remove a client instance from Percona Monitoring and Management (PMM) version 2. Under normal operations, this shouldn’t be necessary, since a graceful PMM client uninstall shouldn’t leave any lingering instances around. However, it may happen that an instance is destroyed (for instance, an EC2 server that is abruptly terminated; or a vagrant machine or container removed) without executing proper uninstall steps.
Removing the instance won’t delete its associated data, but it will prevent the server from continuing to try to poll data from a non-existent client.
Nomenclature
Some PMM-related terms used below are:
- exporter/agent – process(es) in charge of making metrics (or data) available to the PMM server. Some examples are:
- mysqld_exporter – exports metrics for MySQL
- mysql_slowlog_agent – exports query-related data for MySQL
- service – a group of agents, working together to provide data from one database. Some examples are:
- MySQL
- MongoDB
- PostgreSQL
- node – a group of services, working together to provide data from one instance (which may have many different database processes running)
Graceful Termination
First, let’s go over the steps we should take to avoid running into issues that will require manual intervention. There are two ways in which we can let the PMM server know we no longer want an instance or process to be monitored. These cases are for instances that are still working correctly (i.e.: we can access the client nodes, and use the pmm-admin client tool).
1- The first way is to remove the pmm2-client package altogether, meaning that you don’t want any agent at all to continue working in that instance.
For RHEL/CentOS:
1 | shell> yum remove pmm2-client |
For Debian/Ubuntu:
1 | shell> apt-get remove pmm2-client |
2- The second way is to remove a service in particular. First, use pmm-admin list to get the list of services running. An example output is:
1 2 3 4 5 6 7 8 9 | shell> pmm-admin list Service type Service name Address and port Service ID MySQL node1-mysql 127.0.0.1:3306 /service_id/f5c1add5-e0f4-4858-a718-18c426490f67 Agent type Status Agent ID Service ID pmm_agent Connected /agent_id/f30fd2ba-63d1-4fbe-bf8f-309e9bea5a7b node_exporter Running /agent_id/ea8ce2ab-414d-4351-b882-7554b361543a mysqld_exporter Running /agent_id/03f85d08-8093-4e8e-8735-e51e902144f0 /service_id/f5c1add5-e0f4-4858-a718-18c426490f67 mysql_slowlog_agent Waiting /agent_id/17568192-33aa-4675-a0dc-6b111e29ca21 /service_id/f5c1add5-e0f4-4858-a718-18c426490f67 |
Let’s say we want to remove the MySQL service (note that it is not possible to remove one agent, only, since the remove subcommand operates at service level). We need to issue the following command:
1 2 | shell> pmm-admin remove mysql node1-mysql Service removed. |
In the following section, we’ll explore how to remove them when things go awry.
Manual Removal of Nodes
If the instance was not cleanly deregistered from PMM, it can indefinitely linger on in the Home Dashboard and node/service drop-down lists, while graphs will show no data being populated. After the node is deleted, it may continue to show in the Home Dashboard if there was recent data still collected. If this is the case, you will have to wait until the data retention period purges the data (but at least you won’t see it after that :)).
If we no longer have access to the instance, we should resort to the pmm-admin inventory “hidden” feature. In this case, we won’t have direct access to the old client node, but we can use the pmm-admin client tool from any other currently-working node.
It is not really a hidden feature, though, since we can consult the tool about it:
1 | shell> pmm-admin inventory --help |
However, it will not show listed as a subcommand if we check pmm-admin –help (hence, the quotes around hidden). This functionality is not intended for day-to-day operations, and should only be used while there is no other way to remove lingering phantom nodes. The Dev team is working on fixing this, but in the meantime, we thought we should share these steps.
Removing the Node
The only thing we need for this is a working pmm-admin command from any node. We can list all nodes subscribed to PMM with the following command:
1 2 3 4 5 6 7 | shell> pmm-admin inventory list nodes Nodes list. Node type Node name Address Node ID GENERIC_NODE node2 192.168.54.33 /node_id/854eff2d-ce11-4e82-82e7-dfa500584aae GENERIC_NODE node1 192.168.54.32 /node_id/c0e5414c-67e9-41c2-ba2b-02d251d3de2c GENERIC_NODE pmm-server 127.0.0.1 pmm-server |
And consult the help outputs on how to proceed:
1 2 3 4 5 6 7 | shell> pmm-admin inventory remove node --help usage: pmm-admin inventory remove node [<flags>] [<node-id>] Remove node from inventory [...] --force Remove node with all dependencies [...] |
Let’s suppose we want to remove node2, so we’ll need to execute the following command:
1 2 3 | shell> pmm-admin inventory remove node --force \ /node_id/854eff2d-ce11-4e82-82e7-dfa500584aae Node removed. |
Note how we are using its “Node ID” string as the last argument, which we previously got from the list nodes output above.
At this point, agents related to that node will no longer be scraped by PMM server, and should not appear as unhealthy services under the Prometheus targets page (located at http://<your_PMM_URL>/prometheus/targets).
Manual Removal of Nodes (Another Way)
Another thing we can do to overcome this is to use the –force argument while using the same node name to re-add the client. In this way, we will be effectively overriding the orphaned client with a new instance of it. Of course, this means that we will need to have a new client to add, which won’t always be the case. The command would look like:
1 2 | shell> pmm-admin config --force [other_args] \ IP_ADDRESS generic NODE_NAME |
In this case, choose NODE_NAME to be the same as the orphaned node, and it will be overwritten.
Conclusion
If you want to avoid these kinds of issues, it’s always best to ensure a clean uninstall procedure is run. But if you find out it’s too late for that, you can use the inventory commands, while we provide a cleaner way to work around this situation.
Download Percona Monitoring and Management today!
I had some trouble with this. Before finding this blog post, I tried to delete old MySQL instances (Aurora) by using the GUI, navigating to PMM -> Inventory and removing the nodes (with the force option to remove associated services) but the node count in home page still reflects that the primary/replica existed, along with PMM, since it says 3.
I connected to the docker instance in a bash but don’t see the nodes in the inventory that way:
# pmm-admin --server-url=http://admin:ABC123@localhost/ inventory list nodes
Nodes list.
Node type Node name Address Node ID
GENERIC_NODE pmm-server 127.0.0.1 pmm-server
Only PMM internals were shown in list nodes/agents/services and the pmm-admin command online help doesn’t seem to list subcommands like ‘inventory’:
# pmm-admin –help 2>&1 | grep inventory
#
Hm, maybe it really is a hidden feature.
I suppose these will fall out of view and be deleted from Prometheus once they are no longer in retention
Hi Mike,
You should try with:
# pmm-admin inventory –help
It is not listed in the main –help command, but if you use it in that way it will correctly show.
In more recent versions, there has been new functionality added in the web UI that will make these workarounds not needed anymore, so feel free to check them out, and see if they fix your issues.