Posts

Holey jeans

Manual patching vs auto patching

Everyone agrees keeping your software and devices updated is important.  These can be manually or automatically installed.  People assume that automatic is the better option however both have their advantages.

I’m Rob, I look after maintenance packages here at Dogsbody Technology. I want to explain the advantages between the two main patching approaches.

What to Patch

Before we get into the differences of how to patch it’s worth discussing what to patch.

Generally speaking we want to patch everything.  A patch has been produced for a reason to either fix a bug or security issue.

Sometimes patches add new features to a package and this can be when issues occur.  Adding new features can cause things to break (usually due to broken configuration files).

Identifying when a patch release is for a bug, a security fix or adding a feature can be hard. In some cases the patch can be all three things.  Some operating systems try and separate or tag security patches separately however our experience shows that these are rarely accurate.

One of the reasons we like manual patching so much is that it allows us to treat each patch/customer/server combination independently and only install what is required, when it is required.

Auto Patching Advantages

The server checks and updates itself regularly (hourly/daily/weekly).

  • Patches can easily be installed out of hours overnight.
  • Patches are installed during the weekend and bank holidays.
  • Perfect for dev environments where downtime is OK.
  • Perfect for use in Constant Integration (CI) workflows where new patches can be tested before being put into production.

Our automatic patching strategy is to typically install all patches available for the system as it is the only sure way to know you have all the security patches required.

Manual Patching Advantages

A notification (e-mail or internal ticket) is sent to the server admin who logs onto the server and installs the latest updates.

  • Patches can be held during busy/quiet periods.
  • The admin can ensure that services are always restarted to use the patch.
  • The admin can search for dependant applications that maybe using a library that has been patched (e.g. glibc patches)
  • The admin is already logged onto the server ready to act in case something does break.
  • Kernel reboots (e.g. Meltdown or Stack Clash) can be scheduled in and mitigated.
  • Configuration changes can be reviewed and new options implemented when they are released. Catching issues before something tries to load a broken configuration file.
  • Perfect for production environments where you need control. Manual patching works around your business.

Because we manually track the packages used by a customer we can quickly identify when a patch is a security update for that specific server.  We typically patch security updates on the day it is released also install non-security updates at the same time to ensure the system has the latest and greatest.

 

Are you unsure of your current patch strategy? Unsure what the best solution is for you? Contact us today!

 

Feature image background by Courtnei Moon licensed CC BY 2.0.

Intel vulnerabilities (Meltdown & Spectre)

On 3rd January 2018 engineers around the world scrambled to respond to the announcement that most CPUs on the planet had a vulnerability that would allow attackers to steal data from affected computers.  Almost two weeks later and we do know a lot more however the outlook is still bleak.

Am I vulnerable?

Almost definitely.  While only Intel CPUs are affected by the Meltdown vulnerability (CVE-2017-5754) CPUs made by AMD, ARM, Nvidia and other manufactures are all affected by the Spectre vulnerabilities (CVE-2017-5753 &  CVE-2017-5715).

Additionally, Spectre is a collection of vulnerabilities.  Only two of the easiest to implement attacks are currently being patched for.  There are literally hundreds of ways to exploit Spectre and many do not have an easy fix. The Spectre collection of vulnerabilities are responsible for the slowdown of CPUs in your computer as they target a major part of the CPU responsible for the speed (speculative execution).

There are a few exceptions for CPUs not affected by these vulnerabilities however so far these have all been low powered ARM devices such as the Raspberry Pi.

It is worth pointing out that while most computers, servers & mobile phones are vulnerable, an attacker would still have to be able to run code on the same CPU you are using in order for you the be affected. For cloud computing providers this is a big issue as the same CPU is being used by many guest systems. For desktop systems this is a problem as most websites nowadays require that browsers run untrusted Javascript.  For dedicated servers being used by one company however, the only code that should be running on the system is trusted code. While this doesn’t make dedicated servers any less vulnerable, it does severely reduce the attack surface.

How does it work?

Better people than us have already covered this.  We recommend these two blog posts…

How do I fix this?

You replace your CPU.  Seriously! This is currently the only 100% guaranteed method to be free of these vulnerabilities.  However, that there currently aren’t actually any replacement CPUs that aren’t vulnerable! This issue may speed up some providers depreciation of old technology.

Patches for the Meltdown vulnerability have been made available for all major operating systems now.  Make sure you have installed and rebooted to ensure that the patch is loaded in.

If you are using any sort of virtualisation or cloud infrastructure then make sure that your host is patched too. Most cloud providers are announcing reboots at very short notice.

Patches for the Spectre vulnerabilities are still dribbling out and new patches will likely be required for years to come as new fixes are developed.  The current two Spectre patches include a microcode patch for the actual CPU firmware.  This firmware update should still be shipped out via the standard operating system updates.  These patches will also require systems to be rebooted (again).

But I’m a customer!

Don’t worry, we got you.  We are actively working with all our customers to patch systems and mitigate issues.

Timeline

In tracking these vulnerabilities and writing this blog post we built up a comprehensive timeline of events linking to sources of more information that maybe useful…

  • Between Aug 2016 & Jun 2017 – Multiple vulnerabilities are discovered and published by multiple researchers, mostly building on each others work.
  • 01 Feb 2017 – CVE numbers 2017-5715, 2017-5753 and 2017-5754 are assigned to/reserved by Intel to cover these vulnerabilities.
  • 01 Jun 2017 – The two attack vectors are independently found by Google’s Project Zero researchers and researchers from the academic world which are shared with Intel, AMD and ARM.
  • Sep 2017 – Google deploys fixes in their Linux based infrastructure to protect their customers.  Google proposes to pass the patches upstream to the Linux kernel after the public disclosure of Spectre/Meltdown.
  • 09 Nov 2017 – Intel informs partners and other interested parties under Non Disclosure Agreement (NDA).
  • 20 Nov 2017 – The CRD (Coordinated Release Date) is agreed upon to be 09 Jan 2018 by the parties involved.
  • 13 Dec 2017 – Apple releases iOS 11.2, MacOS 10.13.2 and TVos 11.2. These update contain fixes for Meltdown but that is not mentioned in the release notes.
  • 15 Dec 2017 – Amazon starts sending emails to AWS customers, informing them of a scheduled reboot of EC2 instances on or around the 06 Jan 2018. People that reboot following that email notice degraded performance and start discussing this.
  • 20 Dec 2017 – Jonathan Corbet publishes an article and remarks that the KPTI patches have “all the markings of a security patch being readied under pressure from a deadline”.
  • 01 Jan 2018 – A pythonsweetness post appears, speculating about what’s behind the KPTI patches for the Linux kernel.
  • 02 Jan 2018 – The Register publishes an article that puts enough of the information together.
  • 02 Jan 2018 – Andres Freund posts to the PostgreSQL mailing list showing a 17-23% slowdown in PostgreSQL when using the KPTI patch.
  • 03 Jan 2018 – Google breaks the agreed CRD and makes everything public.
  • 03 Jan 2018Two websites are launched to explain the findings.  The vulnerabilities are “officially” named Meltdown and Spectre.
  • 03 Jan 2018 – Microsoft rushes out a series of fixes, including security updates and patches for its cloud services, which were originally planned for a January 9 release.
  • 03 Jan 2018 – Amazon says it has secured almost all of its affected servers.
  • 03 Jan 2018 – Google details its efforts to safeguard its systems and user data.
  • 03 Jan 2018 – Intel acknowledges the existence of the vulnerability, but refutes reports implying it is the only chipmaker affected.
  • 04 Jan 2018 – Media organisations such as the BBC pick up the story.
  • 04 Jan 2018 – Apple confirms its iPhones, iPads, and Macs are affected by the Meltdown and Spectre vulnerabilities.
  • 09 Jan 2018 – Microsoft confirms that patches rolled out to close Meltdown and Spectre security loops have caused PC and server performance slowdowns.

Cyber Security Awareness Month 2017

Dogsbody Technology is happy to be a champion of National Cyber Security Awareness Month (NCSAM) to get everyone thinking about their security online.

Online safety is our shared responsibility, and it starts with STOP. THINK. CONNECT.

STOP: make sure security measures are in place.
THINK: about the consequences of your actions and behaviours online.
CONNECT: and enjoy the internet.

We actively believe that security is not something you “do” (I’ve built this server now I’m going to secure it), it is something that has to be thought about as part of the culture of the business we are in. It is also something that has to be done at all levels of the business including customers and suppliers.

Follow these basic tips throughout October – and all year-round! – to help protect yourself, your information and promote a more trusted internet for everyone.

Own your online presence – Set the privacy and security settings on websites to your comfort level for information sharing. It’s OK to limit how and with whom you share information.

Personal information is like money. Value it. Protect it. – Information about you, such as purchase history or location, has value – just like money. Be thoughtful about who gets that information and how it’s collected by apps and websites.

Keep a clean machine – Keep all software on internet-connected devices – including PCs, smartphones and tablets – up to date to reduce risk of infection from malware.

Get 2 steps ahead – Your usernames and passwords are not enough to protect key accounts like email, banking and social media. Turn on two-factor authentication (2FA) – also known as two-step verification or multi-factor authentication (MFA) – on accounts where available. Two-factor authentication can use anything from a text message to your phone to a token to a biometric like your fingerprint to provide enhanced account security.

Share with care – Think before posting about yourself and others online. Consider what a post reveals, who might see it and how it could be perceived now and in the future.

Declutter your mobile life –  Most of us have apps we no longer use and some that need updating. Delete unused apps and keep others current, including the operating system on your mobile device.

Do a digital life purge –  Perform a good, thorough review of your online files. Tend to digital records, PCs, phones and any device with storage just as you do for paper files. Get started by doing the following:

  • Clean up your email: Save only those emails you really need and unsubscribe to email you no longer need/want to receive.
  • Back it up: Copy important data to a secure cloud site or another computer/drive where it can be safely stored. Password protect backup drives. Always back up your files before getting rid of a device, too. You can’t go wrong with the classic 3-2-1 Backup Strategy -3 total copies of your data, 2 of which are local but on different mediums (read: devices), and at least 1 copy offsite (for if your house/office burns down).

Know what devices to digitally “shred” –  Computers and mobile phones aren’t the only devices that capture and store sensitive, personal data. External hard drives and USBs, tape drives, embedded flash memory, wearables, networking equipment and office tools like copiers, printers and fax machines all contain valuable personal information.

Clear out stockpiles –  If you have a stash of old hard drives or other devices – even if they’re in a locked storage area – information still exists and could be stolen. Don’t wait: wipe and/or destroy unneeded hard drives as soon as possible.

Empty your trash or recycle bin on all devices and be certain to wipe and overwrite – Simply deleting and emptying the trash isn’t enough to completely get rid of a file. Permanently delete old files using a program that deletes the data, “wipes” it from your device and overwrites it by putting random data in place of your information ‒ that then cannot be retrieved.

For devices like tape drives, remove any identifying information that may be written on labels before disposal, and use embedded flash memory or networking or office equipment to perform a full factory reset and verify that no potentially sensitive information still exists on the device.

 

Most of these suggestions just require time.  There really is no excuse.

HashGate

HashGate: An intrusion detection tool

HashGate is a simple intrusion detection tool we wrote for use internally and in customer environments to monitor files and alert us on any unauthorised changes to them.

We try very hard not to re-invent the wheel and are already big users of tools such as Tripwire and Rookit Hunter but we wanted something lightweight for monitoring site files, not system files.

HashGate is written in Python using only core modules and aims to work on all platforms that can run Python 2.7, not just Linux!

Our main use for HashGate is for monitoring files on WordPress & Magento installations which more commonly are exposed to vulnerabilities allowing hackers to modify files. HashGate records the hashsum of all files in the specified directory and stores them for checking periodically, we run our checks hourly via cron.

Below is an basic example output where a file has been modified:


alex@dogsbody-alex:~$ ./hashgate.py -ca /tmp/files.cache -f /home/alex/Documents/Junk/ -t check
The following files were modified:
/home/alex/Documents/Junk/wordpress/index.php
----------------------------------

Other features of HashGate include whitelisting, which allows us to ignore files that frequently change and don’t need to be monitored such as WordPress’ cache files or Magento’s sessions directory.

There is also VirusTotal checking, this is where HashGate will check flagged files hashes against VirusTotal’s database of malicious files to determine if the change was malicious or not. Due to the nature of VirusTotal’s API we’re only able to do 4 requests per minute so if lot’s of files are flagged it will add some extra time to hash checks.

We have recently open sourced this tool and you can find some more information and a list of the full features and usage in the Github repo, if you feel something can be written better or there’s a feature you’d like to add we invite you to contribute and help us build a better tool. We make use of tools like HashGate in some of our server monitoring packages so be sure to check them out and get in contact if they could be of use.

Let’s Encrypt: Security Everywhere

Let’s Encrypt is a new Certificate Authority (CA) who are making waves in the web community. They have lowered the access barrier for SSL certificates significantly and are pushing their competition to improve; fast.

“A Certificate Authority is an entity that validates other digital certificates… …Creating a Chain of Trust between a website and the browser.”

Read more about Certificate Authorities or how to trust over the Internet.

Why Lets Encrypt is revolutionary:

  • Let’s Encrypt removes the pay wall for SSL certificate’s making them free for everyone.
  • Its quick. Seemingly instant certificate authentication and provisioning.
  • Open client options for many different programming languages and environments.
  • Certbot (the official client, developed by the Electronic Frontier Foundation (EFF)) is incredibly simple to set up and run HTTPS in seconds. See for yourself.
  • Automated SSL regeneration. A new certificate just when the old one expires.
  • Raising the standards for CA security checks. Let’s Encrypt have implemented new security checks which ensure that you are the domains owner and that it’s secure to issue you the certificate. Read more.
  • Short validation periods. Let’s Encrypt certificates are only valid for three months which in comparison to other CA signed certificates is shorter. You may be thinking this is bad, long validation periods means less work to maintain. But should the next Heartbleed vulnerability come along and your certificate is leaked to the public, the perpetrator only has less than three months to use it then it will no longer be valid.
  • Supported, as of last year Let’s Encrypt are trusted in most browsers. Test it for yourself. Read more.

It’s free, easy and simple to do so there is no reason not to get started straight away.

Quick (nearly instant) certificate provisioning is our favourite benefit. We often have new customers come to us that have been caught out by expiring SSL certificates not leaving enough time for the renewal to take place, which with Extended Validation certificates can be weeks! Let’s Encrypt is our first port of call to mitigate the missing certificate. Giving us a temporary solution while their other certificate is renewed.

At Dogsbody Technology we love SSL and have already started implementing Let’s Encrypt when we can. If you want to see the benefit of SSL drop us a line.

Feature image made by Got Credit licensed CC BY 2.0.

DROWN vulnerability

Dogsbody Technology maintenance customers are already protected against the newly disclosed DROWN attack, but as of the 1st March, 33% of all HTTPS sites are affected by this vulnerability.

The DROWN (Decrypting RSA with Obsolete and Weakened eNcryption) vulnerability affects HTTPS and other services that rely on SSL and TLS, these cryptographic protocols that make security over the Internet possible.

The attack affects all SSLv2 servers and allows attackers to decrypt HTTPS traffic during transfer letting them spy on traffic. In some cases encryption can be broken within minutes!

The fix web servers is to disable SSLv2 support:

  • For Apache: SSLProtocol all -SSLv2 -SSLv3
  • For Nginx: ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

For more information on the attack and research paper take a look at the official DROWN Attack website.

Dogsbody Technology are Linux SysAdmin’s, building secure scalable reliable servers for the internet. We keep our servers up-to date and in doing so have already mitigated this attack.

If you want your site checked or have any questions please contact us.

CVE-2015-7547 glibc vulnerability

In the past few days Google has identified a vulnerability in glibc (GNU C Library). It allows attackers to crash processes and potentially run code remotely on your server.

The vulnerability itself is best described by the Google Security Team’s blog-post. To summarise:

“The glibc DNS client side resolver is vulnerable to a stack-based buffer overflow when the getaddrinfo() library function is used. Software using this function may be exploited with attacker-controlled domain names, attacker-controlled DNS servers, or through a man-in-the-middle attack… …Remote code execution is possible, but not straightforward.”

glibc is a library which provides many basic functions and system calls to C programs. Since libraries are only loaded in when a program is started, this means that only daemonised (a process which is left running in the background) programs are effected. When those programs are restarted they will load in the new glibc library which mitigates the issue.

You can get a list of all programs using glibc by running a command such as:

sudo lsof | grep libc | cut -d' ' -f 1 | sort | uniq

This shows that glibc is tied into nearly every service on a typical Linux system.  It can quickly become a large job to restart each process, especially in the correct order.  The quickest way of doing this is by rebooting your server.

Our advice regarding this matter is:

  1. Ensure the latest glibc packages are installed.
  2. Reboot your server (or restart all processes that use glibc)

Feel free to get in touch if we can help with this.

CVE-2014-3566 – POODLE

What is POODLE

The POODLE (Padding Oracle On Downgraded Legacy Encryption) vulnerability allows an attacker to obtain data transferred with the SSL 3.0 protocol.  An attacker acting as a man in the middle can downgrade a TLS connection to SSL 3.0 and then use a padding-oracle attack to access sensitive information such as cookies.  Since stealing a user’s cookies will allow an attacker to login as that user, they are the most likely target of a POODLE attack.

Prevention

This vulnerability can be fixed either on the server or in the client.

Site owners can protect their users against POODLE attacks by disabling TLS fallback or SSL 3.0 (Note that disabling SSL 3.0 will break the site for IE6 users):

  • For Apache: SSLProtocol all -SSLv2 -SSLv3
  • For Nginx: ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

Browsers are rolling out fixes but for users the quickest fix is to disable SSL 3.0:

  • In Firefox this is done by going to about:config and setting security.tls.version.min to 1
  • Chrome users have to use the command line flag --ssl-version-min=tls1

Going deeper

This attack is possible because SSL pads requests to fill the last block before encryption.  SSL 3.0 only requires the last byte to be checked by the server; it must have a value equal to the number of bytes that have been used for padding.  The values of the other padding bytes are not validated, this allows an attacker to move the block they want to decrypt to the the last block and try all 256 possible values until the server accepts the request, allowing them to decode one byte of the cookie.  An attacker in a privileged network position (or sharing public WiFi) just needs to downgrade the SSL connection from TLS to SSL 3.0 and then use JavaScript to quickly obtain a cookie one byte at a time.

For more technical information I would recommend this article by ImperialViolet.

Feature image made by Koji Ishii licensed CC BY 2.0

CVE-2014-6271 – Shellshock

Shellshock is a bug in the bash shell.  The main issue comes from the fact that commands can be executed if they are crafted into environment variables.  This means anyone who can send a user agent to Apache can run commands as the user running Apache.

Am I affected?

You can test if your server is vulnerable by logging in and running

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If it outputs vulnerable there are a few steps you can take to try to prevent it being exploited.

Prevention for website owners

The easiest solution is to update to a version of bash that isn’t vulnerable however if one has not yet been released on your distribution you will have to consider other prevention methods.

Since an attacker needs to exploit a vulnerable service two likely targets being SSH and Apache you can mitigate most attack vectors by stopping these services.
As long as you have another way to login it is worth stopping SSH since it is likely to be running as root it could allow an attacker to gain root access to the server.
Stopping Apache is a more difficult decision since it will prevent customers from accessing your site however if you are very concerned then it may be the best cause of action.

A more complex solution is to switch to a different shell instead of bash but this is more complex and may have unexpected consequences to how applications run so we don’t recommend doing this blindly.

If you have a maintenance agreement with us then you don’t need to worry because we are updating bash whenever possible.

Feel free to get in touch if we can help with this.

Feature image – “Shellshock” by Linux Screenshots is licensed under CC BY 2.0

CVE-2014-0160 – Heartbleed

A vulnerability has been discovered that allows anyone over the internet to read data straight off of your server.

“Catastrophic” is the right word. On the scale of 1 to 10, this is an 11.
– Bruce Schneier

Labelled “Heartbleed” this vulnerability leaves your servers memory vulnerable and accessible to be read by anyone. A lot of private information is at risk, everything from passwords to SSL certificate keys are loaded into memory so often it is only a matter of time until a malicious user gets them.

The affected software, OpenSSL is a library that provides tools for encryption. OpenSSL is installed by default on many Linux systems as many core tools depend on it for SSL. It is widely used by servers for web, email, remote shell, VPN, file transfer and much more…

Test your website for the Heartbleed vulnerability.

The following command lists all services using libssl:

sudo lsof | grep libssl

The only fix is to upgrade OpenSSL to a non-vulnerable version and restart all services using it. Since it is used by so many services it can quickly become a large job to restart each process, especially in the correct order. The quickest way of doing this is by rebooting your server.

For more reading see the official Heartbleed website.

Our advice regarding this matter is:

  1. Ensure a fixed OpenSSL package is installed.
  2. Reboot your server (or restart all processes that use OpenSSL)

Feel free to get in touch if we can help with this.

Feature image by Alan O’Rourke under the CC 2.0 license.