Wednesday, May 14, 2014

OpenSSL Heartbleed Vulnerability and Implications

An extremely critical defect in the cryptographic software library (OpenSSL) has been found, the vulnerability is named Heartbleed and it affects the heartbeat implementation in OpenSSL version 1.0.1 up to version 1.0.1f.

This vulnerability can be used to get the user Ids, passwords, SSL private keys (and pretty much anything in your servers memory) so it is important to update your affected servers immediately.

The bug has been fixed in OpenSSL 1.0.1g. All Major Linux Distributions have released updates to the vulnerability.


  • Vulnerable OpenSSL libraries have been running in production for over two years
  • OpenSSL is used in the Apache and nginx Web server applications (and most apps that use SSL encryption) on most Linux systems
  • OpenSSL is also used in many serial devices that are connected to the internet
  • Exposed private keys enables eavesdroppers to decrypt any of your site's SSL traffic
  • No data on your server is "safe" until you fix this vulnerability ***

*** Will your customers will be willing to shop at a site that can't keep their passwords, credit card numbers or other personal and financial information safe?

*** What will your HIPPA penalty (or legal settlement) be if patient medical data is exposed from one of your servers or connected serial devices?

Breach Notification

Breach notice laws typically define, “personal information” as, "A user name or email address, in combination with a password or security question and answer that would permit access to an online account."

The data that a Heartbleed attack could capture would trigger a breach notification; However, the problem (gift?) for most health based organizations is that they have no idea whether a breach has occurred, unless they have advanced monitoring solutions in place, see How to Safeguard Against Future Vulnerabilities section below.

HIPAA Security Rule

The Heartbleed vulnerability does not constitute unauthorized access or acquisition of personal information subject to most state and federal data breach notification requirements, including the HIPAA Data Breach Notification Rule. However, the HIPAA Security Rule contains a number of provisions that require covered entities and business associates to maintain procedures to monitor system activity for potential security incidents and investigate any such potential security incidents.

The HIPAA Security Rule requires covered entities and business associates to “regularly review records of information system activity, such as audit logs, access records, and security incident tracking reports.” 45 C.F.R. § 164.306(a)(1)(ii)(D). HHS guidance materials further state that this specification “should also promote continual awareness of any information system activity that could suggest a security incident.” See CMS, HIPAA Security Series Vol. 2 Security Standards: Administrative Safeguards

The HIPAA Security Rule requires covered entities and business associates to create and maintain appropriate records of system activity. See 45 C.F.R. 164.312(b). However, covered entities and business associates have significant discretion to create and maintain activity records based upon the formal assessment of their security risks.

Action Plan

  1. Update every system, application and device that needs an OpenSSL update
  2. Restart all services and devices
  3. Install new SSL certificates (revoke old certs and replace encryption keys)
  4. Reset all passwords (force password resets for users)
  5. Request users to reset passwords to 3rd party applications, e.g. DropBox, Google, etc.
  6. Evaluate the effectiveness of your audit and security incident tracking systems
  7. Consider installing IPS or NGFW

Reality Check

Many sites have reported that more than two thirds of the sites on the internet were affected.

It was likely less than half of the sites on the internet, since it was not every *nix server and also because the OpenSSL software had to be built in the last two years.

Today, however, it appears that less than 3% of the sites on the internet are vulnerable.

So, web site admins were responsive.

Regardless, if your site is in the 3% you are in dire need of an OpenSSL upgrade.

Who's Affected?

The most obvious potential victims are commerce web sites.

Do you shop at any site looking for the best deal?

If so, consider installing a web browser plugin that will indicate whether the site you are about to share your confidential information is affected.

Check out Chromebleed

More alarming is that fact that there are many more devices with internet connectivity that may never get patched.

Some of those devices are simple routers other devices control industrial and medical equipment.

Had medical diagnostics performed lately? How secure are the devices at your doctor's office or local hospital?

Have one of those fancy home automation systems to control your garage door or security cameras?

Hacking into one of those devices and controlling it may not be easy, but gathering enough information from it to determine when you are home is.

How to Safeguard Against Future Vulnerabilities

Since the Heartbleed bug exposes a seemingly random chunk of 64KB of data, odds are that it would take quite a few requests before a private key could be recovered. Recovering working user Ids and their passwords would not take near as many tries.

Sites protected by intrusion protection devices (IPS) can be configured to block traffic from an end point that is sending an inordinate amount of requests.

Sites that have monitoring tools, like Splunk, in place to look for traffic anomalies and security admins to interpret the logs are in a better position to detect such a breach.

However, not many sites are equipped with such sophisticated security devices and traffic monitoring tools and even fewer have trained eyes that can interpret the traffic.

Sites that have valuable data to secure typically not only have high performance intrusion detection devices, but also hire a third party security team to analyze and act on attacks.

I'm partial to the IBM product line: IBM Security Network Intrusion Prevention System

What CTO ever went wrong hiring IBM security services? IBM Intrusion prevention and detection services

NextGen Firewall Solutions

The NGFW has a focus on application awareness that integrates packet inspection and threat detection on the the firewall.

A NGFW can look into HTTP(S) traffic and determine if it contains confidential data like credit card numbers or social security numbers.

A NGFW can be configured to allow specified user groups access to Facebook, but disallow Facebook chatting.

The placement and configuration of a NGFW takes thought, but can be well worth the investment.

IBM and Palo Alto NGFWs are the best on the market, IMHO.

This is what I worked on a few years ago: IBM Next Gen IPS

Check this out to see a video of IBM Next Gen IPS: IBM Demos Some of My Work on


For the do-it-yourself companies, there is Splunk.

Splunk Enterprise is a data collection and analysis platform for real-time operational intelligence. It's the easy, fast and secure way to search, analyze and visualize the massive streams of machine data generated by your IT systems (event logs, devices, services, TCP/UDP traffic, etc.)

Set up alerts to notify you when something is wrong. Troubleshoot application problems and investigate security incidents in minutes instead of hours or days, avoid service degradation or outages, deliver compliance at lower cost and gain new business insights.

How to Fix the Heartbleed Vulnerability

Simply upgrade your OpenSSL library to version 1.0.1.g or later of OpenSSL to fix the vulnerability.

Check Your OpenSSL Version

# openssl version
OpenSSL 1.0.1b 10 May 2012

Ubuntu and Debian

# apt-get update
# apt-get upgrade

Centos and Fedora

# yum update


# zypper update

Verify Upgrade Was Successful

Ubuntu and Debian

Here's an example of a successful upgrade to OpenSSL:

#dpkg-query -l 'openssl'
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version           Architecture   Description
ii  openssl             1.0.1e-3ubuntu1.3 amd64          Secure Socket Layer (SSL) binary and related

Confusing Version Number

Don't be confused by the fact that the version number listed is 1.0.1e.

The official Ubuntu SSL page lists that version number as the correct, patched version. For details see:

CentOS and Fedora

Use rpm to verfiy your version of OpenSSL.

# rpm -qa | grep openssl


After upgrading your OpenSSL library, don't forget to restart services that use it such as:
  • httpd
  • postgres
  • mysql
  • cpanel
  • dovecot
  • postfix

Also, you'll need to reissue your certs and revoke any certs created before Apr 7 2014.

Verify Your New OpenSSL Fix

You can use the following site to verify your server:

Warning: Do not use this site to check sites that you do not own.

The technique used to check the SSL flaw is an exploit that requests extra bytes from the target server's buffer. Viewing those extra bytes would be tantamount to unauthorized access to data, which is illegal in most countries.

Unauthorized Access

"Unauthorized access" entails approaching, trespassing within, communicating with, storing data in, retrieving data from, or otherwise intercepting and changing computer resources without consent. These laws relate to either or both, or any other actions that interfere with computers, systems, programs or networks.

For details see Computer Crime Statutes

Ethical Hacking

On April, 11, 2014, a cloud service provider, CloudFlare, set up an nginx server with a vulnerable version of OpenSSL and challenged the community to steal its private key.

Several people independently retrieved private keys using the Heartbleed exploit.



Better Safe Than Sorry

Stephen Arthuro Solis Reyes I was dismayed to read about this young man. A computer science student that failed to think through the consequences of his actions.

The Royal Canadian Mounted Police have arrested a 19-year-old London, Ontario, man for his alleged role in exploiting the Heartbleed vulnerability to steal data from the Canada Revenue Agency website.

See 19-Year-Old Teenager Arrested for Exploiting Heartbleed Bug to Steal Data

Like I said in the Unauthorized Access section above, accessing data that's not yours without consent is a crime and if don't know everything about how you can be tracked, don't do it. Even if you do, don't do it. Don't limit your future by making snap decisions.

Earlier Version of OpenSSL

Here's a case where an older operating system, running an old version of OpenSSL, is not affected by the Heartbleed bug.

Determine Currrent openssl Version

# apt-cache policy openssl
  Installed: 0.9.8g-4ubuntu3.20
  Candidate: 0.9.8g-4ubuntu3.20
  Version table:
 *** 0.9.8g-4ubuntu3.20 0
        500 hardy-updates/main Packages
        500 hardy-security/main Packages
        100 /var/lib/dpkg/status
     0.9.8g-4ubuntu3 0
        500 hardy/main Packages

# openssl version -a
OpenSSL 0.9.8g 19 Oct 2007
built on: Mon Feb 18 21:40:29 UTC 2013
platform: debian-amd64
options:  bn(64,64) md2(int) rc4(ptr,char) des(idx,cisc,16,int) blowfish(ptr2)
OPENSSLDIR: "/usr/lib/ssl"

# openssl version -b
built on: Mon Feb 18 21:40:29 UTC 2013

Attempt upgrade

# apt-get update
Hit hardy Release.gpg
Ign hardy/main Translation-en_US
Ign hardy/restricted Translation-en_US
Ign hardy/universe Translation-en_US
Hit hardy-updates Release.gpg
Ign hardy-updates/main Translation-en_US
Ign hardy-updates/restricted Translation-en_US
Ign hardy-updates/universe Translation-en_US
Hit hardy-security Release.gpg
Ign hardy-security/main Translation-en_US
Ign hardy-security/restricted Translation-en_US
Ign hardy-security/universe Translation-en_US
Hit hardy Release
Hit hardy-updates Release
Hit hardy-security Release
Hit hardy/main Packages
Hit hardy/restricted Packages
Hit hardy/universe Packages
Hit hardy/main Sources
Hit hardy/restricted Sources
Hit hardy/universe Sources
Hit hardy-updates/main Packages
Hit hardy-updates/restricted Packages
Hit hardy-updates/universe Packages
Hit hardy-updates/main Sources
Hit hardy-updates/restricted Sources
Hit hardy-updates/universe Sources
Hit hardy-security/main Packages
Hit hardy-security/restricted Packages
Hit hardy-security/universe Packages
Hit hardy-security/main Sources
Hit hardy-security/restricted Sources
Hit hardy-security/universe Sources
Reading package lists... Done

# apt-get upgrade openssl
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Verify Installed Version

# openssl version -a
OpenSSL 0.9.8g 19 Oct 2007
built on: Mon Feb 18 21:40:29 UTC 2013
platform: debian-amd64
options:  bn(64,64) md2(int) rc4(ptr,char) des(idx,cisc,16,int) blowfish(ptr2)
OPENSSLDIR: "/usr/lib/ssl"

# openssl version -b
built on: Mon Feb 18 21:40:29 UTC 2013

# dpkg-query -l 'openssl'
| Status=Not/Installed/Config-f/Unpacked/Failed-cfg/Half-inst/t-aWait/T-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                                       Version                                    Description
ii  openssl                                    0.9.8g-4ubuntu3.20                         Secure Socket Layer (SSL) binary and related cryptographic tools


This old OpenSSL 0.9.8 branch is not vulnerable.

The Heartbleed bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug (Mon Apr 7 20:31:55 UTC 2014).


No comments:

Post a Comment