In my previous blog post, ProcFS UDF: A Different Approach to Agentless Operating System Observability in Your Database, I wrote about the ProcFS UDF MySQL plugin, which allows you to get operating systems stats, through the MySQL database, without having shell access to the server and any local agent installation.
Some of you wondered whether there is a way to use this goodness in Percona Monitoring and Management (PMM), and this blog post will show you exactly how to do that.
Unfortunately, at this point, Percona Monitoring and Management does not support the ProcFS UDF MySQL plugin out of the box. It is in the backlog, along with many other cool things. However, if this particular feature would be valuable to you, please let us know.
That said, Percona Monitoring and Management is extensible, so you can actually make things work with a little bit of elbow grease using the external exporter functionality.
Here’s how:
1. Configure the MySQL host you wish to monitor with ProcFS UDF as described in this blog post.
2. Add this MySQL server as a remote instance using “Add Instance,” available in the PMM menu in the top right corner.
3. Pick the host to capture metrics.
While you do not need any agent installed on the MySQL server, if you’re looking to monitor, you’ll need a server to capture metrics from it and pass them to the PMM server. This server will need the PMM client installed and configured. You will also need to install Node Exporter with ProcFS UDF:
1 | docker/podman run -p 9100:9100 -d docker.io/perconalab/node_exporter:procfs --collector.mysqlprocfs="MYSQLUSER:MYSQLPASSWORD@tcp(MYSQLHOST:3306)" |
If you do not want to use docker, instructions on how to compile patched Node Exporter are included in the previously mentioned ProcFS UDF Introduction blog post.
You can use one host to monitor multiple MySQL servers — just run multiple Node Exporters on different ports.
4. Configure passing information to PMM.
Now that we have MySQL metrics flowing to PMM as a remote instance and Node Exporter running on a different host, which is ready to provide us metrics, how do we establish a connection so that those “node metrics” are attached to the correct host?
First, we need to find the node_id of the remote node we’ve added:
1 2 3 4 5 6 7 8 | root@client1:# pmm-admin inventory list nodes Nodes list. Node type Node name Address Node ID GENERIC_NODE mysql1 66.228.62.195 GENERIC_NODE client1 50.116.36.182 /node_id/9ba48cd4-a7c2-43f6-9fa6-9571d1f0aebf …. REMOTE_NODE procfstest 173.230.136.197 /node_id/29800e10-53fc-43f7-bba6-27c22ab3a483 |
Second. we need to get the external service added for this node:
1 2 3 4 5 6 7 8 9 10 | root@client1:~# pmm-admin inventory add service external --name=procfstest-node --node-id=/node_id/29800e10-53fc-43f7-bba6-27c22ab3a483 External Service added. Service ID : /service_id/c477453f-29fb-41e1-aa64-84ee51c38cd8 Service name : procfstest-node Node ID : /node_id/29800e10-53fc-43f7-bba6-27c22ab3a483 Environment : Cluster name : Replication set: Custom labels : map[] Group : external |
Note: The Node ID we use here is the Node ID of the remote node we are monitoring.
This creates the external service, but it is really orphan at this point — there is no agent to supply the data.
1 2 3 4 5 6 7 8 9 10 11 12 | root@client1:~# pmm-admin inventory add agent external --runs-on-node-id=/node_id/9ba48cd4-a7c2-43f6-9fa6-9571d1f0aebf --service-id=/service_id/c477453f-29fb-41e1-aa64-84eec38cd8 --listen-port=9100 External Exporter added. Agent ID : /agent_id/93e3856a-6d74-4f62-8e8d-821f7de73977 Runs on node ID : /node_id/9ba48cd4-a7c2-43f6-9fa6-9571d1f0aebf Service ID : /service_id/c477453f-29fb-41e1-aa64-84ee51c38cd8 Username : Scheme : http Metrics path : /metrics Listen port : 9100 Disabled : false Custom labels : map[] |
This command now specifies what external service we’ve created for our remote node will get data from the agent, which is running on the other node and will use the port specified by the listen-port option. This is what our ProcFS Enabled Node Exporter is using.
After you’ve done these steps, you should see OS data for the remote host appear on the home dashboard.
And even more important, you’ll have OS metrics populated in the Node Summary dashboard.
Summary
While this is much harder than it has to be, I think it serves as a great proof of concept (POC) of what is possible with the ProcFS UDF MySQL plugin — getting the full power of the operating system and MySQL observability without requiring any shell access to MySQL.
I think this can be extremely valuable for MySQL provided as a Database as a Service (DBaaS), as well as for enterprises practicing great separation between their teams responsible for operating system and database operations!
