Cloudflare, Keyless SSL and the selling of 'secure'.

Cloudflare is an amazing content-delivery-network (CDN) platform that powers a lot of the web. They make the web faster and more stable by distributing content around the globe so that it is nearer visitors, blocking abusive users and offering a school-of-fish approach to web security. They also have a unique reputation in the security field, both for making predictions about security that turned out to be wrong, and for having the openness and transparency to make their assertions publicly and to admit mistakes. Their Heartbleed challenge and their general approach to security helped the community take a theoretical problem in TLS (the technology that tries to secure the web) and prove that it was an actual issue deserving of immediate mitigation and attention.

Yesterday, Cloudflare announced a new product offering called Keyless SSL. They held the technical details back from the press until today. Now with the full technical detail release, I write this post as my excitement for a claimed innovation turns again into the disappointment of overhyped marketing and a product that doesn't deliver on the promise of secure end-to-end computing in the cloud.

The marketing material boasts that you can have "secure" TLS operation without divulging your Private Keys. The key claim made in their blog post yesterday ( ) was:

"The lock appeared and the connection was secured, end-to-end." [emphasis mine]

There's only one problem -- its simply not accurate, and the connection is not secure end-to-end. That is, the connection isn't secured between the user and the organization they're doing business with. The reality is that the connection is being intercepted, in the middle, by a third party and with all the inherent security implications native to MITM proxies.

To their credit Cloudflare has put up a great technical description post of what they have built and you'll find the technical blog post at

Here's where the claims of security and end-to-end encryption break down: the Session Keys. In the Keyless SSL architecture, the Session Key is shared with Cloudflare. Whether its the confidentiality of user input (think accounts and passwords), the content of the website (think bank balances), the security policy of web scripting (think same origin), persistent storage access (think cookies), and so on and so forth -- its the Session Keys that form the foundation of the web's security and confidentiality model. Without confidentiality of the Session Keys, you don't have any security.

With the Session Key you have a read/write ability to view and modify any of the data flowing between the user and the organization that they are doing business with. This gives rise to a host of legal problems that come from the /ability/ to break TLS connections, but the short of it is, if you can read and write the confidential data in the middle of the connection (like at Cloudflare), then the session is no longer end-to-end secure and at some point, if you're big enough, you should probably expect a government will come knocking (or even hacking).

If you're an organization and you willingly give those Session Keys to a third party, you're just deputizing them for your entire online business. The product is anything but "Keyless" and involves significant confidential data disclosure to Cloudflare. While the service is running (read, the normal state of things) they have all the same authorities and problems that you do as it relates to the session security, and the problems of maintaining the confidentiality of your user information.

So lets go back to banking. Banking's hugely complicated but one of the key obstacles to their adoption of cloud technology is that there are strong privacy protection rules that govern banking confidentiality. This prohibits a third party from receiving confidential information about account holders -- and it makes it impossible to share a Private Key or break encryption to allow for the use of a CDN service. Such prohibitions apply despite the benefits to the bank of using a CDN service, such as improved fault tolerance and service resiliency. Kicking the proverbial can down the road from the SSL certificate Private Key to the Session Key, doesn't change anything as it relates to the confidentiality of banking information -- Like seeing the users account username, password, or account balances. In both public and Session Key scenario's Cloudflare's service can read your client's information -- (compelled, accidentally or hackerishly) -- and thats the rub, its just not end-to-end secure.

Law Enforcement and Political Risk

There's also the issue of the Law Enforcement Agencies (LEAs) using the Cloudflare service to break into user accounts. In effect, a Cloudflare-like system expands the range of actors that can be forced to disclose sensitive key material to government agencies. This includes ongoing and logged access to the generated Session Keys and TLS parameters, which could be compelled by government order.

We all know about the cheeky spy-agency 'SSL added and removed here' slide, but even outside the clandestine, we have witnessed the willingness of American authorities to compel American companies to disclose this type of information before. The case in point is Lavabit. In the Lavabit case the company, which was strongly marketing privacy and security as key service selectors, was forced by legal process to provide cryptographic material to facilitate the installation of interception devices against the entire Lavabit service. The technical design of the system did not protect against such a brazen, service-wide approach to user-account access by law enforcement. Thankfully Ladar Levinson, after a long legal fight, shuttered the Lavabit service as a result of the political damage.

This is all to say, that technical security to guard against oneself as a bad actor is extremely important. It takes more than rhetoric and the hope of doing good to build truly secure services that are immune from the political risks we saw in Lavabit. If you want to known more about the Lavabit nightmare, Moxie Marlinspike has an absolute must-read on the subject.

The lesson here is that it is important that Cloudflare and other services trying to implement TLS services truly understand the scope of the political risk they are creating when they start managing keys, even Session Keys, on behalf of others.

They must be up-front with partners about these very real risks.

Practical Abilities - Private Keys vs Session Keys

The only significant difference between the Session Key and the Private Key in a TLS setup is that the Private Key comes first in the sequence. The end goal of the handshake is to derive a shared Session Key, and its this Session Key that provides the abilities and confidentialities expected of TLS. The TLS Handshake/Private Key process can be thought of abstractly like a Session Key minting machine, but its these Session Keys that actually work the locks. Once you have the Session Key for a connection, the Private Key is no longer relevant for that connection.

With access to a service's Session Keys you can:

  • - Intercept confidential user information. (usernames, passwords)
  • - View and change the content of user web pages. (MITM attack, see bank balances, etc)
  • - Identify and isolate specific website users. (User profiling)
  • - Be subject to government requests for data interception, manipulation.
  • - Suffer a security breach the same as a bank could.
  • - Access user information like cookies and browser storage.
  • - Script against the user's browser as the domain.
  • - Cryptographically impersonate the service and authenticate with an end user.
  • - (Future? Bind the service to a specific TLS key.)
  • - Operate an app firewall/CDN service and inject captcha's and similar into secure sessions

Only one very small part of the TLS operation is performed by the certificate Private Key in question, and its none of the things we really care about -- like the ability to maintain communications confidentiality and respond to law enforcement as a first party. Really, the only significant new ability the Keyless solution offers over a shared Private Key solution is the ability to turn off the creation of new session keys; like in the case of Cloudflare's service becoming compromised, its a better kill-switch. If thats the only contingency you're planning for, Keyless is right for you.

If thats not your contingency, and you have practical issues with third party data access; like ones of legal, policy and malicious attackers -- well then, its like Moxie described of Lavabit:

"Unfortunately, their primary security claim wasn't actually true."

To translate to Cloudflare Keyless SSL, I'll posit:

Unfortunately, their end-to-end security claim wasn't actually true.

Reinventing the wheel

The really bad news though is that, what they "invented" using raspberry pi's and fabled stories of skunkworks development was already largely found in commercial off the shelf products known as networked hardware-security-modules (Net-HSMs).

Thales, Amazon, among others make network HSMs -- You put your Private Keys in them, they stay in the datacenter, and then you point a webserver (or group of webservers like cloudflare's CDN) at the them using an OpenSSL engine (among other methods). The HSM handles the Private Keys, signs the secrets and, in effect, provides a similar kind of service as what Cloudflare is doing with their signing daemon. The webserver just offloads the cryptographic signing operations to the HSM via OpenSSL directly.

So really, there's nothing all that new to see here, networked HSM's have been around for a long time, and they do practically the same job, all-be-it at a high cost. However, due to the problems inherent in trusting keys (even Session Keys or oracles) to third parties, they have never really been popular for third party access or use. They're primarily used within an organization as a defense-in-depth technique to limit the damage caused by network intruders.

New security risks

Contrary to claims made about the service providing equivalent to on-premise security for SSL keys, I find there are a number of entirely new risks presented by the model.  Just a few of the newly expanded security vulnerabilities include:

- Trusted Cloudflare staff compromising the service. (Third party employee risk)
- Government agencies "hunting sysadmins" at Cloudflare. (LEAs, NSA, CSEC, GCHQ, etc)
- Hacker risk both at Cloudflare and with oracle access control. (unauthorized use)
- Technical downtime risk. (more points of failure)
- Oracle attack risk.
-- This is where the Oracle is tricked into revealing something about its internal keys. The post covers a couple of these known attacks, padding oracles and timing attacks, but at least for the latter it doesn't solve them, it just pushes responsibility off to the OpenSSL library that, while the best we have, suffers regularly from new attack vectors and zero-day vulnerabilities. I would be curious to see how the signing oracle design will stand up over time to the increasing sophistication of adaptive chosen plaintext/ciphertext attacks, among others.
- Confused deputy risk. (attacking the oracle to sign malicious data)
--This is where something goes wrong at the service and you sign bad data anyway. For example, the signing of specially crafted data. There are a number of TLS security concerns regarding the signing of crafted data, and this presents an entirely new risk when a third party is involved and able to get you to blindly sign data.

and the list goes on and on. In the world of TLS security, extra eyes and extra legal organizations (possibly in different jurisdictions) with access to data creates massive risk. Oracles that blindly sign or decrypt arbitrary cryptographic data are generally frowned upon as an architecture, as are deputized daemons that cannot verify the source of their inputs; as a result I have to question the idea that the the Cloudflare Keyless SSL solution provides the same level security as on-premise keys.

So with the Keyless SSL architecture, in my opinion, thoroughly debunked, we're still left with the fairly 'classic' E2E problem; how can we leverage the cloud's benefits while maintaining the confidentiality of the communications?

Cracking that E2E security nut to work with intermediaries will be one of the key research projects of our generation. Its one of the reasons I was so excited when Cloudflare announced they'd managed to do an E2E-secure CDN and optimistic that if anyone could solve it, it was them.

Sadly, with the announcement of the Keyless SSL technical details, it seems that future is still over the horizon.

As always, I welcome a response from the vendor and will happily update this post to include their response if provided.