38. Maintenance steps

by Cover Tower - Updated October 15, 2021

Since this guide is complete, it includes the maintenance steps and their frequency, in order to keep the server in good condition and up-to-date.

1. Once a week, typically every Friday, download the backups from Nextcloud to two other places: one can be your laptop/desktop, and the other can be an external hard drive. A safe backup exists in three copies: one typically on the machine whose data is backed up, in a special directory with restrictive permissions, so as to have easy access to it when necessary, a second copy on your personal laptop/desktop to which you have exclusive access, and a third copy on an external hard drive only you can access. The golden rule is that the three copies of the backups should be stored in places situated physically far apart from each other, but in such a way that they would be easily accessible to only one person: you. Of course, your laptop/desktop and the external hard drive may be close to each other, but the distance between your computer and the remote server should be substantial, so that an accident affecting one would not affect the other. It’s not advisable to store your backups in more than three places because this increases the odds that they will fall into the hands of the wrong people. And you have to remember that the backups include all your important data, including sensitive information such as certain passwords. So, a backup in three different places means safety, while more than three backups actually decrease safety progressively.

Let’s suppose you rented three VPSs in the same data center of a hosting company, then you stored a copy of your backups on each of those VPSs and then conclude your data is safe. That’s wrong. Imagine what will happen if that entire data center is destroyed due to human error or natural disasters such as earthquakes, hurricanes, floods, landslides, volcanic eruptions, wildfire, etc.: all your data on the live websites and all your backups will be lost in a second, along with months or even years of work. This is why the backups need to be stored in different places physically located far apart from each other.

Estimated time if no errors occur: about 10 minutes per week (or 8 hours per year).

2. Once every four weeks, log in to your server via SSH, ensure that all the services are running smoothly, browse through /var/log/syslog , /var/log/mail.log, /var/log/mail.err, /var/log/dovecot.log, /var/log/ufw.log, /var/log/letsencrypt/letsencrypt.log, /var/log/asterisk/messages, /var/log/fail2ban.log, /var/log/nextcloud/nextcloud.log, /var/log/named/security.log, /var/log/nginx/webpublicist.net.error.log and other logs, to see if there are signs of suspect behavior, errors, malfunctions, then perform all the upgrades necessary with the apt-get update && apt-get dist-upgrade command. Next, check on their respective official websites if there are new versions for these applications: Postfix Admin, Roundcube, phpList, phpMyAdmin, WordPress, Dolibarr, Asterisk, LibreOffice Online, Nextcloud, Roundpin, MyBB, Friendica, Matomo, System Health and Security Probe, RED SCARF Suite Panel, and if necessary, perform the upgrade following the upgrade steps described in the chapters dedicated to these applications. We should point out that from the mentioned applications, Asterisk, LibreOffice Online and Nextcloud can be upgraded more rarely (once every few months or once per year).

Please note that you should perform the upgrade for a specific application only after you make sure that its newest version works well in the context of the entire software suite described in this guide. To check which is the newest version of an application that has been tested and confirmed to work well within the suite, visit this page.

Please keep in mind that the upgrading step is the point where much too often beautiful dreams turn into nightmares. Applications that worked flawlessly for years can become defective or crash entirely because of a simple upgrade. This happens because the applications’ complexity makes it very difficult for their developers to thoroughly test new versions. Even when new versions are tagged as ‘Stable’ they can still contain bugs. It’s also very difficult to test each new version in all possible scenarios. Therefore, instead of getting new features and improved performance by upgrading an application to its latest version, too often the system administrator finds himself in a difficult situation: the new version malfunctions, gives errors or even crashes. That is why you have to know in advance if a new version of an application is really ready to be installed. As mentioned, to find out if new versions can be safely installed and used within the suite, we recommend that you visit this page.

Estimated time if no errors occur: about 1 hour per four weeks (or 13 hours per year).

3. Once every two years, when a new Debian version is released, you will have to upgrade Debian to the newest version and take additional maintenance steps, as described below. Ideally the upgrade should be first tried on a test server, as identical as possible with the production server and if the upgrade goes well on the test server it should be done identically on the production server. However, if you don’t have a test server, you can perform the upgrade directly on the production server by carefully following these steps:

  • then inform all your users, at least one week in advance, by email or in a banner displayed on the websites/applications, about the precise date and hour when the upgrade will be scheduled and that the provided services will be unavailable at that specific moment.
  • ensure that you have a second way of accessing your server besides SSH, in case the worst thing happens during the upgrade: you remain locked out of your server, which becomes unaccessible via SSH due to some network configuration that gets changed, etc. Some hosting providers offer a web console that allows accessing a server without using SSH, while others, like OVH, offer a rescue mode boot up that you can use to log in to a different functional server from which you can mount the hard drive of your disfunctional server and edit any files as needed, then you can restart your server to apply the new changes.
  • check if everything is working well, if there are any services in failed state, etc.
  • download the latest version of the ‘Memcached Object Cache’ WordPress plugin from https://wordpress.org/plugins/memcached/ . Uncompress the archive, create a random string to use as salt by running openssl rand -base64 48 , add the salt at the beginning of the object-cache.php file from the uncompressed files, like this:
if ( ! defined( 'WP_CACHE_KEY_SALT' ) ) {
	define( 'WP_CACHE_KEY_SALT', 'vkCi6QNmBT4drFMXpCIj57iEGte54CGO9yathz8fkdAsfP0k2gOU6lGdbLGHLqSh' );

then replace the old /var/www/example.com/wp-content/object-cache.php file with the new object-cache.php file that you have just edited, for each website for which you use Memcached for object caching.

  • Let’s Encrypt stores the old SSL certificate files in the directories of the form /etc/letsencrypt/archive/www.example.com . Once every two years you can delete the old certificate files (having the form: cert123.pem, chain123.pem, fullchain123.pem, privkey123.pem), because they get more numerous every three months and once they have expired, they are not needed. Pay attention not to delete the most recent files which are needed. The most recent files have the highest number at the end of their name, before the pem extension.
  • build and install Asterisk again, as described in the ‘Build and Install Asterisk’ chapter. This is needed because Asterisk depends on software packages that in general change from one Debian version to another. When you build and install Asterisk, it’s recommended to use the newest Asterisk LTS version available.

Estimated time if no errors occur: about 4 hours per two years (or 2 hours per year).

Total estimated time for periodic server maintenance if no incidents happen: about 23 hours per year, which gives an average time of about 2 hours per month for each server on which the standard RED SCARF Suite is installed.

You can send your questions and comments to: