Posts

How to set-up unattended-upgrades

Making sure software is kept up to date is very important.  Especially when it comes to security updates.  Unattended-upgrades is a package for Ubuntu and Debian based systems that can be configured to update the system automatically.  We’ve already discussed manual patching vs auto patching, most of this post will assume you’d like to set-up automatic updates.  If you want complete control of updates you may need to disable unattended-upgrades, see the manual updates section below.

Automatic Updates

Make sure you have installed unattended-upgrades and update-notifier-common (in order to better determine when reboots are required).  On some recent operating systems unattended-upgrades will already be installed.

sudo apt-get install unattended-upgrades update-notifier-common

Once unattended-upgrades is installed you can find the configs in /etc/apt/apt.conf.d/.  The 50unattended-upgrades config file has the default settings and some useful comments.  20auto-upgrades defines that updates should be taken daily. The default configuration will install updates from the security repository

We would suggest creating a new file and overwriting the variables you want to set rather than changing files that are managed by the package.

You can create the following as /etc/apt/apt.conf.d/99auto-upgrades:

# Install updates from any repo (not just the security repos)
Unattended-Upgrade::Allowed-Origins {
"*:*";
};
# Send email to root but only if there are errors (this requires you to have root email set-up to go somewhere)
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";
# Remove packages that should no longer be required (this helps avoid filling up /boot with old kernels)
Unattended-Upgrade::Remove-Unused-Dependencies "true";
# How often to carry out various tasks, 7 is weekly, 1 is daily, 0 is never
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
# Use new configs where available but if there is a conflict always keep the current configs.
# This has potential to break things but without it the server won't be able to automatically update
# packages if you have changed a configuration file that the package now has an updated version of.
Dpkg::Options {
"--force-confdef";
"--force-confold";
}
# Some updates require a reboot to load in their changes, if you don't want to monitor this yourself then enable automatic reboots
Unattended-Upgrade::Automatic-Reboot "true";
# If you want the server to wait until a specific time to reboot you can set a time
#Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Have a look at the comments but the key things to point out here are:

  • The above config will install all updates.  You can define what repositories to update from as well as any packages to hold back but then you will obviously end up with some software out of data.
  • It is important to make sure you will be informed when something goes wrong.  One way to do this is to send errors to root and have all root email sent to you (you can define this in /etc/aliases).  To test you are receiving email for root you can run:
    echo "Test email body" | mail -s "Test email subject" root
  • If you aren’t going to follow security updates and decide when to reboot your server make sure you have automatic reboots enabled, it is probably worth setting an appropriate reboot time.

Manual updates

If you want to manually update your server then there is no need for you to install unattended-upgrades.  However as some operating systems have it pre-installed so you may have to disable it.  The easiest way to disable unattended-upgrades is to create the following as /etc/apt/apt.conf.d/99disable-auto-upgrades:

APT::Periodic::Unattended-Upgrade "0";

Feature image by Les Chatfield licensed CC BY 2.0.

Holey jeans

Manual patching vs auto patching

Everyone agrees keeping your software and devices updated is important.  These can be manually or automatically installed.  People assume that automatic is the better option however both have their advantages.

I’m Rob, I look after maintenance packages here at Dogsbody Technology. I want to explain the advantages between the two main patching approaches.

What to Patch

Before we get into the differences of how to patch it’s worth discussing what to patch.

Generally speaking we want to patch everything.  A patch has been produced for a reason to either fix a bug or security issue.

Sometimes patches add new features to a package and this can be when issues occur.  Adding new features can cause things to break (usually due to broken configuration files).

Identifying when a patch release is for a bug, a security fix or adding a feature can be hard. In some cases the patch can be all three things.  Some operating systems try and separate or tag security patches separately however our experience shows that these are rarely accurate.

One of the reasons we like manual patching so much is that it allows us to treat each patch/customer/server combination independently and only install what is required, when it is required.

Auto Patching Advantages

The server checks and updates itself regularly (hourly/daily/weekly).

  • Patches can easily be installed out of hours overnight.
  • Patches are installed during the weekend and bank holidays.
  • Perfect for dev environments where downtime is OK.
  • Perfect for use in Constant Integration (CI) workflows where new patches can be tested before being put into production.

Our automatic patching strategy is to typically install all patches available for the system as it is the only sure way to know you have all the security patches required.

Manual Patching Advantages

A notification (e-mail or internal ticket) is sent to the server admin who logs onto the server and installs the latest updates.

  • Patches can be held during busy/quiet periods.
  • The admin can ensure that services are always restarted to use the patch.
  • The admin can search for dependant applications that maybe using a library that has been patched (e.g. glibc patches)
  • The admin is already logged onto the server ready to act in case something does break.
  • Kernel reboots (e.g. Meltdown or Stack Clash) can be scheduled in and mitigated.
  • Configuration changes can be reviewed and new options implemented when they are released. Catching issues before something tries to load a broken configuration file.
  • Perfect for production environments where you need control. Manual patching works around your business.

Because we manually track the packages used by a customer we can quickly identify when a patch is a security update for that specific server.  We typically patch security updates on the day it is released also install non-security updates at the same time to ensure the system has the latest and greatest.

 

Are you unsure of your current patch strategy? Unsure what the best solution is for you? Contact us today!

 

Feature image background by Courtnei Moon licensed CC BY 2.0.

End of Life for New Relic ‘Servers’ – What are your options?

Today (14 Nov 2017) New Relic are making their ‘Alerts’ and ‘Server’ services end of life (EOL). This will impact anyone who used this service to monitor server resources such as CPU, Memory, Disk Space and Disk IO. All existing alert policies will cease from today.

If you rely on these alerts to monitor your servers then hopefully you have a contingency plan in place already but if not below are your options….

If you do nothing

New Relic Servers will go EOL TODAY (14 Nov 2017) and data will stop being collected.  You would no longer be able to monitor your system resources meaning outages that could have otherwise been prevented could sneak up on you. We do not recommend this option.  See below on how to remove the `newrelic-sysmond` daemon.

Upgrade to New Relic Infrastructure

“Infrastructure” is their new paid server monitoring offering. Infrastructure pricing is based on your servers CPU so prices vary and offers added functionality over the legacy New Relic Servers offering.  The maximum price per server per month is $7.20 however the minimum monthly charge is $9.90 so it’s not effective if you’re only looking to monitor your main production system. Most of the new functionality is integration into other products (including their own) so it’s up to you if this additional functionality is useful and worth the cost for your requirements.

Dogsbody Technology Minder

Over the last year we have been developing our own replacement for New Relic Servers using open source solutions. This product has old New Relic Server customers in mind giving all the information needed to run and maintain a Linux server. It also has the monitoring and hooks required to alert the relevant people of issues allowing us to prevent issues before they happen.  This is a paid service but it is included as standard with all our maintenance packages so any customers using New Relic Servers are being upgraded automatically. If you would like further information please do contact us.

Another alternative monitoring solution

There are plenty of other monitoring providers and solutions out there from in-house build your own open source solutions to paid services.  Monitoring your system resources is essential in helping to prevent major outages of your systems. Pick the best one for you and let the service take the hard work out of monitoring your servers.  We have experience with a number of implementations including the TICK stack (Telegraf, InfluxDB, Chronograf, Kapacitor) and Prometheus.

Removing the `newrelic-sysmond` daemon

If you were using New Relic Servers then you are running the `newrelic-sysmond` daemon on your systems.  While New Relic have turned the service off we have confirmed with them that the daemon will keep running using valuable system resources.

We highly recommend that you uninstall the daemon (tidy server tidy mind) following New Relic uninstallation guide.  That way it won’t take much of your system’s resources, and minimal impact is to be expected.

 

Happy Server Monitoring

If you need help, further advise or to discuss our monitoring solutions please do contact us.

DROWN vulnerability

Dogsbody Technology maintenance customers are already protected against the newly disclosed DROWN attack, but as of the 1st March, 33% of all HTTPS sites are affected by this vulnerability.

The DROWN (Decrypting RSA with Obsolete and Weakened eNcryption) vulnerability affects HTTPS and other services that rely on SSL and TLS, these cryptographic protocols that make security over the Internet possible.

The attack affects all SSLv2 servers and allows attackers to decrypt HTTPS traffic during transfer letting them spy on traffic. In some cases encryption can be broken within minutes!

The fix web servers is to disable SSLv2 support:

  • For Apache: SSLProtocol all -SSLv2 -SSLv3
  • For Nginx: ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

For more information on the attack and research paper take a look at the official DROWN Attack website.

Dogsbody Technology are Linux SysAdmin’s, building secure scalable reliable servers for the internet. We keep our servers up-to date and in doing so have already mitigated this attack.

If you want your site checked or have any questions please contact us.