Posts

Holey jeans

Manual patching vs auto patching

Everyone agrees keeping your software and devices updated is important.  These can be manually or automatically installed.  People assume that automatic is the better option however both have their advantages.

I’m Rob, I look after maintenance packages here at Dogsbody Technology. I want to explain the advantages between the two main patching approaches.

What to Patch

Before we get into the differences of how to patch it’s worth discussing what to patch.

Generally speaking we want to patch everything.  A patch has been produced for a reason to either fix a bug or security issue.

Sometimes patches add new features to a package and this can be when issues occur.  Adding new features can cause things to break (usually due to broken configuration files).

Identifying when a patch release is for a bug, a security fix or adding a feature can be hard. In some cases the patch can be all three things.  Some operating systems try and separate or tag security patches separately however our experience shows that these are rarely accurate.

One of the reasons we like manual patching so much is that it allows us to treat each patch/customer/server combination independently and only install what is required, when it is required.

Auto Patching Advantages

The server checks and updates itself regularly (hourly/daily/weekly).

  • Patches can easily be installed out of hours overnight.
  • Patches are installed during the weekend and bank holidays.
  • Perfect for dev environments where downtime is OK.
  • Perfect for use in Constant Integration (CI) workflows where new patches can be tested before being put into production.

Our automatic patching strategy is to typically install all patches available for the system as it is the only sure way to know you have all the security patches required.

Manual Patching Advantages

A notification (e-mail or internal ticket) is sent to the server admin who logs onto the server and installs the latest updates.

  • Patches can be held during busy/quiet periods.
  • The admin can ensure that services are always restarted to use the patch.
  • The admin can search for dependant applications that maybe using a library that has been patched (e.g. glibc patches)
  • The admin is already logged onto the server ready to act in case something does break.
  • Kernel reboots (e.g. Meltdown or Stack Clash) can be scheduled in and mitigated.
  • Configuration changes can be reviewed and new options implemented when they are released. Catching issues before something tries to load a broken configuration file.
  • Perfect for production environments where you need control. Manual patching works around your business.

Because we manually track the packages used by a customer we can quickly identify when a patch is a security update for that specific server.  We typically patch security updates on the day it is released also install non-security updates at the same time to ensure the system has the latest and greatest.

 

Are you unsure of your current patch strategy? Unsure what the best solution is for you? Contact us today!

 

Feature image background by Courtnei Moon licensed CC BY 2.0.

CVE-2015-7547 glibc vulnerability

In the past few days Google has identified a vulnerability in glibc (GNU C Library). It allows attackers to crash processes and potentially run code remotely on your server.

The vulnerability itself is best described by the Google Security Team’s blog-post. To summarise:

“The glibc DNS client side resolver is vulnerable to a stack-based buffer overflow when the getaddrinfo() library function is used. Software using this function may be exploited with attacker-controlled domain names, attacker-controlled DNS servers, or through a man-in-the-middle attack… …Remote code execution is possible, but not straightforward.”

glibc is a library which provides many basic functions and system calls to C programs. Since libraries are only loaded in when a program is started, this means that only daemonised (a process which is left running in the background) programs are effected. When those programs are restarted they will load in the new glibc library which mitigates the issue.

You can get a list of all programs using glibc by running a command such as:

sudo lsof | grep libc | cut -d' ' -f 1 | sort | uniq

This shows that glibc is tied into nearly every service on a typical Linux system.  It can quickly become a large job to restart each process, especially in the correct order.  The quickest way of doing this is by rebooting your server.

Our advice regarding this matter is:

  1. Ensure the latest glibc packages are installed.
  2. Reboot your server (or restart all processes that use glibc)

Feel free to get in touch if we can help with this.