Remote Access Checklist
Remote Access Checklist
This checklist will help you ensure that our access to your servers is correctly set up before we begin working with you.
Please make sure you’ve provided us with the items listed below, so that we can log in to your servers. If there’s more than one server, please tell us each server’s function (e.g., “primary” or “read-only replica”), and any other information that applies to each server. This helps ensure we are working on the intended servers. Be sure to point out areas of concern — things we should be careful not to disturb.
- Provide us the list of hosts to work with
- Install our public SSH keys on your hosts
- Provide us the username to log in
- Provide a database username and password for our exclusive use
- Share access details of any monitoring tools (username, password, url)
Public SSH Keys
Our public SSH keys are available from the percona.com domain and can be easily downloaded.
We provide two types of SSH key packages:
- Individual packages that create the username for the engineer should it not already exist on the system, as well as the Percona group if it does not already exist, and installs the public SSH keys into /home/username/.ssh/authorized_keys
- A shared package that does not perform any group or user setup, but instead installs all firstname.lastname.pub files into /usr/share/percona/engineer
For example:
Individual Engineer:
Packages for installing the user and key:
- RPM: https://www.percona.com/get/engineer/RPMS/noarch/percona-engineer-firstname.lastname-1-latest.noarch.rpm
- deb: https://www.percona.com/get/engineer/apt/percona-engineer-firstname.lastname_1-latest_all.deb
- SSH key only: https://www.percona.com/get/engineer/KEY/firstname.lastname.pub
- SSH key checksum: https://www.percona.com/get/engineer/KEY/firstname.lastname.pub.sha256
All Percona Engineers’ SSH keys, in a single package:
This package installs all engineer public SSH keys into /usr/share/percona/engineer/keys
Note: You will have to create the Linux user for Percona yourself; the packages do not create any users. For example:
adduser percona
In order to add all SSH keys to a single Percona Linux account, after installing the shared package:
mkdir -p /home/percona/.ssh/chmod 700 /home/percona/.ssh/cat /usr/share/percona/engineer/keys/*.pub >>/home/percona/.ssh/authorized_keyschmod 600 /home/percona/.ssh/authorized_keys
At the time of this writing, the percona.com SSL configuration gains an A+ grade on qualys SSL Labs, and we support Forward Secrecy.
Engineers connect from the Percona bastions from the following addresses:
54.214.47.252 bastion.percona.com
54.214.47.254 bastion2.percona.com
54.78.164.169 bastion4.percona.com
Please configure your firewall to allow access from these IP addresses to relevant services within your network.
Tailscale
Alternatively, Percona uses Tailscale as our zero-trust VPN solution. We can provide an ephemeral configuration to be installed in your servers so our engineers can connect to it via Tailscale only without the need to open internet ports.
MySQL
Percona User
In order to provide us full access to your environment, Percona asks that you create a fully privileged user in the database, and configure the MySQL client for easy access by setting parameters in /home/percona/.my.cnf or in the newly created user’s home directory. Please choose a strong password in place of PASSWORD referenced in the examples below.
- Execute the following statements in order to create a MySQL user for Percona:
CREATE USER percona@'%' IDENTIFIED BY 'PASSWORD' WITH MAX_USER_CONNECTIONS 10;
GRANT ALL PRIVILEGES ON *.* TO percona@'%'; - Please create the following file
/home/percona/.my.cnf
touch /home/percona/.my.cnf - Please add the following information to the file
/home/percona/.my.cnf:
[client]
user=percona
password=PASSWORD
Also specify the port and socket in case they have non-default values
Performance Audit Checklist
A performance audit is our most popular service. In order to expedite a performance audit, please ensure that the following tools are installed on each machine:
- Perl DBI and DBD::mysql
- MySQL client, vmstat, iostat, mpstat, sysstat and optionally numactl
- If there is no outside network access, please install the Percona Toolkit
These tools are very helpful for observing what the machine is doing, so we highly recommend you install them. If you can’t install any of the above for some reason, we can still perform a limited audit. It may result in less information being analyzed by the audit.
For our analysis, we need the slow query log. Please note the original values so that you can change them back:
SHOW VARIABLES LIKE '%slow%';
SHOW VARIABLES LIKE '%long%';
SHOW VARIABLES LIKE 'log_output';
Then, rotate your logs (you can move the current one and issue FLUSH LOGS;) and set these values:
SET GLOBAL log_output=FILE;
SET GLOBAL slow_query_log_use_global_control='log_slow_verbosity,long_query_time,log_slow_rate_limit';
SET GLOBAL log_slow_verbosity='full';
SET GLOBAL slow_query_log=1;
SET GLOBAL long_query_time=0;
SET GLOBAL log_slow_slave_statements=1;
SET GLOBAL log_slow_rate_limit=100;
Some of these variables are available only in Percona Server, so it is fine if they don’t get set.
Once you have 1 hour or 1 GB (whichever is first) of data, set the values back to their defaults and save the file off. Please upload the log to our sftp or another secure location, after compression, and provide it to our consultants.
It may be good to do this on both the primary and a read replica.
Note: When you set long query time to 0, it may have some impact as all queries will be collected. In that case, you may want to use 0.1 or 1.
MongoDB
Percona User
Create a MongoDB user named percona, grant the following privileges, and give us the password (or leave it in a file in the Percona engineer home directory).
db.getSiblingDB(“admin”).createUser({
user: “percona”,
pwd: “********”,
roles: [
{role: “root”, db: “admin”}
]
})
Extended information about the required MongoDB account on your database server that can also be provided:
- What is the username and password of the MongoDB account we should use?
- Have you granted the root role to the user?
- What is the IP address or hostname of the server(s) running mongod/mongos?
- On what IP address does MongoDB listen, if bindIP is set in its configuration?
- What is the connection string for a TLS-enabled environment?
- What are the ports for each component?
Performance Audit Checklist
A performance audit is our most popular service. In order to expedite a performance audit, please ensure that the following tools are installed on each machine:
- mtools
- Mongo client, vmstat, iostat, mpstat and sysstat
- If there is no outside network access, please install the Percona Toolkit
These tools are very helpful for observing what the machine is doing, so we highly recommend you install them. If you can’t install any of the above for some reason, we can still perform a limited audit. It may result in less information being analyzed by the audit.
For our analysis, we need the MongoDB logs. Please upload the log to our sftp or another secure location, after compression, and provide it to our consultants.
It may be good to do this on both the primary and one secondary.
Postgres
Percona User
Unrestricted access as the superuser to the database is required in order to execute the read-only audit scripts. If this is a dedicated host then ssh login privileges to the UNIX/Linux Postgres user account is normally sufficient, since it grants such privileges. If this isn’t possible, then a dedicated ROLE that is a member of the ROLE “postgres” should be created which can execute the scripts.
Performance Audit Checklist
A performance audit is our most popular service. In order to expedite a performance audit, please ensure that the following requirements are satisfied on each machine:
- The psql command line utility.
- vmstat, iostat, mpstat, sysstat and optionally numactl
- Superuser access to the Postgres monitoring metrics.
For our analysis, we need the Postgres log and the Postgres original copies of the configuration files used to manage each instance of PostgreSQL. In addition to obtaining the most up-to-date monitoring metrics, the following files are required:
- postgres log files
- pg_hba.conf
- postgresql.conf
- postgresql.auto.conf
- pg_ident.conf
- supplementary include files as declared in postgresql.conf
In order to obtain the most accurate data, all log parameters to record Postgres overhead activities should be on and verbose logging, log_min_duration_statement=0, invoked for not less than 1 hour or 1GB (whichever is first) of recorded logs. Reset the value for log_min_duration_statement back to its default once the verbose logging session is completed. Please upload the log to our sftp or another secure location, after compression, and provide it to our consultants.
The aforementioned process should be followed for all nodes that include the PRIMARY and REPLICA nodes.
Note: Verbose logging causes a slight performance degradation of approximately 1-3%. Of greater importance is ensuring sufficient space while executing verbose logging.
Amazon RDS / Google Cloud SQL / Azure Database
Ideally provide Percona remote SSH access to a secured Linux server in your own controlled VPC subnet. Securely (by way of GPG or other encryption medium) provide a database account for Percona so that all communication between the client and the managed database happens on the cloud provider’s private internal network.
Any auditing or compliance requirements are to be made known to Percona prior to engagement.
Monitoring Tools
Please provide access to any monitoring tools you are using. For example, Percona Monitoring and Management (PMM) access enables the engineer to clearly see trending information for all the important metrics.
If your servers are deployed in a cloud environment, read-only access to the respective dashboard (e.g. AWS Console) is nice to have.
Other Types of Access
We can work with nearly any type of access. Direct SSH access to the server is most efficient; however, any more elaborate types of access (such as proprietary VPN programs, platform-dependent technologies such as GoToMyPC, etc.,) can be less efficient – sometimes significantly less efficient.
Many of these applications cannot co-exist, so they require elaborate tricks with virtual machines, etc. Multiple hops are also less efficient, as keystroke latency becomes a problem. The use of a graphical desktop on a remote system is also difficult.
In general, Cisco VPN and OpenVPN are the most efficient VPNs for Percona to use. We can also support VPN termination via our bastion host. If you’re using VPN, please tell us all the necessary information:
- What type of VPN system are we connecting? Do we need to download a client?
- Please send us the configuration file; e.g., the .PCF file (if you’re using a Cisco VPN), OpenVPN, or other configuration information
- Otherwise, please tell us all of the following that apply: the domain server’s hostname and IP address, username, password, group name, group password
- VPN might not work well with DNS, so please tell us the IP addresses for the servers
If you’re using any other type of connectivity such as VNC, Remote Desktop, GoToMyPC, or similar, please provide the details.
Performance Audits
A short architectural overview can go a long way toward giving us a “head start” on the performance audit. Please provide brief documentation that includes:
- Programming language(s) used
- Deployment diagrams (if any)
- Components used (load balancers, caching systems, etc.)
- Database replication/sharding Information
- Planned upgrades/changes – alerting us in advance of any new features or changes planned during the time of the audit can save significant time and rework
- Anything else you feel is worth mentioning
You don’t have to go to extremes with any of this. Too much information can cause extra work, too (lists of tables and columns are not necessary). Keep in mind that we have probably seen dozens of systems like yours, so just giving us “hints” about what’s unique or noteworthy about your systems can be very effective. For example, “it’s a social networking application using Ruby On Rails and we are not sharded yet, but we do a lot of caching” tells us a great deal.
Security Policies
When you exchange information with us, or give us access to your servers, please make sure you do it securely. If you need to upload sensitive data to our servers securely for data recovery purposes, please request our GPG key. This lets you encrypt large files that only we can decrypt.
For security and accountability reasons, please do not do any of the following:
- Don’t permit Percona to log into a shared account. Create a dedicated Percona user.
- Don’t install our SSH keys in the root’s home directory.
- Don’t permit Percona to use a common MySQL account such as root. Create a dedicated Percona user in MySQL.
- Don’t permit our SSH key to be forwarded to another host. If we must access your systems through an intermediary server (ex: jump-host), please create an SSH key on that server and install the matching public key on the systems we should be able to access.
Troubleshooting
The most common reason we can’t log in is that the installed SSH key is for a different user than the one Percona was assigned. The next most common reason is that there are line breaks, incorrect permissions, or other problems with the key file. After you’ve set the key up, please let us know so we can test it.
If you are using a VPN, consultants will log in from their own machines using your VPN client, so do not restrict access to any specific range of IP addresses. Also, be sure to use the engineer’s own public SSH key, provide password authentication, or provide an SSH key of your choosing to communicate securely.