Posts

Holey jeans

Manual patching vs auto patching

Everyone agrees keeping your software and devices updated is important.  These can be manually or automatically installed.  People assume that automatic is the better option however both have their advantages.

I’m Rob, I look after maintenance packages here at Dogsbody Technology. I want to explain the advantages between the two main patching approaches.

What to Patch

Before we get into the differences of how to patch it’s worth discussing what to patch.

Generally speaking we want to patch everything.  A patch has been produced for a reason to either fix a bug or security issue.

Sometimes patches add new features to a package and this can be when issues occur.  Adding new features can cause things to break (usually due to broken configuration files).

Identifying when a patch release is for a bug, a security fix or adding a feature can be hard. In some cases the patch can be all three things.  Some operating systems try and separate or tag security patches separately however our experience shows that these are rarely accurate.

One of the reasons we like manual patching so much is that it allows us to treat each patch/customer/server combination independently and only install what is required, when it is required.

Auto Patching Advantages

The server checks and updates itself regularly (hourly/daily/weekly).

  • Patches can easily be installed out of hours overnight.
  • Patches are installed during the weekend and bank holidays.
  • Perfect for dev environments where downtime is OK.
  • Perfect for use in Constant Integration (CI) workflows where new patches can be tested before being put into production.

Our automatic patching strategy is to typically install all patches available for the system as it is the only sure way to know you have all the security patches required.

Manual Patching Advantages

A notification (e-mail or internal ticket) is sent to the server admin who logs onto the server and installs the latest updates.

  • Patches can be held during busy/quiet periods.
  • The admin can ensure that services are always restarted to use the patch.
  • The admin can search for dependant applications that maybe using a library that has been patched (e.g. glibc patches)
  • The admin is already logged onto the server ready to act in case something does break.
  • Kernel reboots (e.g. Meltdown or Stack Clash) can be scheduled in and mitigated.
  • Configuration changes can be reviewed and new options implemented when they are released. Catching issues before something tries to load a broken configuration file.
  • Perfect for production environments where you need control. Manual patching works around your business.

Because we manually track the packages used by a customer we can quickly identify when a patch is a security update for that specific server.  We typically patch security updates on the day it is released also install non-security updates at the same time to ensure the system has the latest and greatest.

 

Are you unsure of your current patch strategy? Unsure what the best solution is for you? Contact us today!

 

Feature image background by Courtnei Moon licensed CC BY 2.0.

End of Life for New Relic ‘Servers’ – What are your options?

Today (14 Nov 2017) New Relic are making their ‘Alerts’ and ‘Server’ services end of life (EOL). This will impact anyone who used this service to monitor server resources such as CPU, Memory, Disk Space and Disk IO. All existing alert policies will cease from today.

If you rely on these alerts to monitor your servers then hopefully you have a contingency plan in place already but if not below are your options….

If you do nothing

New Relic Servers will go EOL TODAY (14 Nov 2017) and data will stop being collected.  You would no longer be able to monitor your system resources meaning outages that could have otherwise been prevented could sneak up on you. We do not recommend this option.  See below on how to remove the `newrelic-sysmond` daemon.

Upgrade to New Relic Infrastructure

“Infrastructure” is their new paid server monitoring offering. Infrastructure pricing is based on your servers CPU so prices vary and offers added functionality over the legacy New Relic Servers offering.  The maximum price per server per month is $7.20 however the minimum monthly charge is $9.90 so it’s not effective if you’re only looking to monitor your main production system. Most of the new functionality is integration into other products (including their own) so it’s up to you if this additional functionality is useful and worth the cost for your requirements.

Dogsbody Technology Minder

Over the last year we have been developing our own replacement for New Relic Servers using open source solutions. This product has old New Relic Server customers in mind giving all the information needed to run and maintain a Linux server. It also has the monitoring and hooks required to alert the relevant people of issues allowing us to prevent issues before they happen.  This is a paid service but it is included as standard with all our maintenance packages so any customers using New Relic Servers are being upgraded automatically. If you would like further information please do contact us.

Another alternative monitoring solution

There are plenty of other monitoring providers and solutions out there from in-house build your own open source solutions to paid services.  Monitoring your system resources is essential in helping to prevent major outages of your systems. Pick the best one for you and let the service take the hard work out of monitoring your servers.  We have experience with a number of implementations including the TICK stack (Telegraf, InfluxDB, Chronograf, Kapacitor) and Prometheus.

Removing the `newrelic-sysmond` daemon

If you were using New Relic Servers then you are running the `newrelic-sysmond` daemon on your systems.  While New Relic have turned the service off we have confirmed with them that the daemon will keep running using valuable system resources.

We highly recommend that you uninstall the daemon (tidy server tidy mind) following New Relic uninstallation guide.  That way it won’t take much of your system’s resources, and minimal impact is to be expected.

 

Happy Server Monitoring

If you need help, further advise or to discuss our monitoring solutions please do contact us.

DROWN vulnerability

Dogsbody Technology maintenance customers are already protected against the newly disclosed DROWN attack, but as of the 1st March, 33% of all HTTPS sites are affected by this vulnerability.

The DROWN (Decrypting RSA with Obsolete and Weakened eNcryption) vulnerability affects HTTPS and other services that rely on SSL and TLS, these cryptographic protocols that make security over the Internet possible.

The attack affects all SSLv2 servers and allows attackers to decrypt HTTPS traffic during transfer letting them spy on traffic. In some cases encryption can be broken within minutes!

The fix web servers is to disable SSLv2 support:

  • For Apache: SSLProtocol all -SSLv2 -SSLv3
  • For Nginx: ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

For more information on the attack and research paper take a look at the official DROWN Attack website.

Dogsbody Technology are Linux SysAdmin’s, building secure scalable reliable servers for the internet. We keep our servers up-to date and in doing so have already mitigated this attack.

If you want your site checked or have any questions please contact us.