SSL Proxy Failing To Decrypt The Handshake, Fixing Connection Reset Issue in New Browsers
There was a new update couple of months ago affecting web servers and web browsers introducing a new TLS extension (Extended master secret) that changes the way master_secret is generated. This update broke the devices that were doing SSL inspection or proxy because they wouldn’t be able to decrypt the handshake and ultimately would reset the connection. In this post we’re going discuss the issue and look for workarounds and fixes. These fixes would be either an update in the device which is decrypting the SSL traffic or temporarily disabling this feature on servers and clients.
Handshake Failure Alert In Client Certificate Authentication Using F5 BIG-IP LTM & ASM
If you’ve been following my blog posts you’re already familiar with what client certificates are, how they work and how to setup a scenario with IIS and F5 BIG-IP (LTM/ASM). The setup would perfectly work fine until around 9 months ago (November 2015) but something in newer version of browsers broke things leading to handshake failure errors and connection reset at BIG-IP.
Thomas Pornin, the famous bear at Security.Stackexchange.com once said about client certificate based authentication:
3. Certificates are complex
Deploying user certificates is complex, thus expensive:
Issuing and managing certificates is a full can of worm, as any PKI vendor can tell you (and, indeed, I do tell you). Especially the revocation management. PKI is about 5% cryptography and 95% procedures. It can be done, but not cheaply.
In addition to what Thomas said about the procedures of certificate management, the implementation side of the story is also quite challenging. Now let’s get back to the issue which was connection reset and handshake failure.
At the time I faced this issue I made sure I did setup everything correctly, but the connection was still being reset. The settings shown in the previous post would happily work until several months ago. There had to be some new feature or an update either on client side or on BIG-IP’s firmware (12.1 with hotfix was used) that’s messing things up. The issue we ran into was that IE 10, older versions of Chrome and Firefox 42 were working, hence the server configuration was fine but IE 11, Chrome 52 and Firefox 47 were throwing errors and resetting connections when going through BIG-IP while using Client Certificate Authentication and ProxySSL feature was enabled. Let’s take a peek at error pages displayed.
IE 11 Error Page as you would expect, said basically “nothing” :
Firefox 47 Error Page said SSL_ERROR_HANDSHAKE_FAILURE_ALERT :
Chrome 52 Error Page said ERR_SSL_PROTOCOL_ERROR :
Ok so my guess was that somehow LTM was failing to decrypt the handshake but I needed to gather more information. I decided to collect as much log as I could so I enabled debug mode logging on BIG-IPs LTM to produce more verbose logs, in order to to this:
- Log in to the Traffic Management Shell (tmsh) by typing the following command:
- Enable SSL debug logging by typing the following command:
modify /sys db log.ssl.level value Debug
– Once done remember to revert this value to default using the following command:
modify /sys db log.ssl.level value Warning
After enabling this option and running tail -f /var/log/ltm the following error showed up upon client connection initiation:
I ran tcpdump on BIG-IP via ssh which came pre-installed and I did the same on my webserver to see what was happening. (BIG-IP also ships with ssldump pre-installed which can come in handy to look into the SSL handshake process).
After the initial Client and Server Hello and key exchange, this is what happened from BIG-IP’s point of view :
4 2 0.0010 (0.0004) S > C Handshake ServerHello Version 3.2 session_id= ac 35 00 00 2d ca 6c 90 a6 b8 b3 31 47 4a d5 aa 21 6a 22 ca 2f 1d 92 42 e2 5e 5b 04 ab 18 d6 14 cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA compressionMethod NULL Certificate ServerHelloDone 4 3 0.0022 (0.0012) C > S Handshake ClientKeyExchange 4 4 0.0022 (0.0000) C > S ChangeCipherSpec 4 5 0.0022 (0.0000) C > S Handshake 4 6 0.0036 (0.0014) S > C Alert level fatal value handshake_failure 4 0.0036 (0.0000) S > C TCP FIN 4 0.0040 (0.0003) C > S TCP FIN
From web server’s point of view the WAF was resetting the connection in the middle of the handshake. Since old web browsers were doing the handshake successfully this had to do something with an update or a new feature present in newer versions of browsers. I started comparing the packet captures between the working and failing browser samples. I filtered the packets by “ssl” keyword.
The following packet capture belongs to IE 10.0920016521 (result: Successful) :
Comparing IE 10 SSL traffic with IE 11:
In the table below I compared IE 10 with IE 11 and tried to figure out the difference between their requests.
|IE 10.0920016521 (result: Successful)||IE 11.420.105.86 (result: Fail)|
|Client Hello (C > S)|
|Version||TLS 1.0||TLS 1.2 Fails and switches to TLS 1.0|
|12 Cipher Suites||TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, … TLS_ECDHE_RSA…, TLS_DHE_DSS_…||28 Cipher Suites|
|Server Hello, Certificate, Server Hello Done (S > C)|
|Version||TLS 1.0||TLS 1.0|
|Certificate & Server Hello Done (S > C)|
|Client Key Exchange (C >S)||Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message (C > S)|
|RSA Encrypted PreMaster Secret||Encrypted Handshake Message|
|Change Cipher Spec (C > S)||Alert(Level: Fatal, Description: Handshake Failure)|
|Encrypted Handshake Message (C > S)|
|Change Cipher Spec, Encrypted Handshake Message (S > C)|
|Application Data (C > S)|
As depicted, all the steps look similar until a certain point at which the BIG-IP sends a handshake failure. I compared the packets and their fields one by one and they looked exactly similar, I couldn’t figure out anything in the beginning. Let’s look at Client Hello more closely and see if there’s something suspicious that we were missing:
|IE 10.0920016521 (result: Successful)||IE 11.420.105.86 (result: Fail)|
If you look closer, on the right there are two unknown extensions which I ignored initially. These extensions aren’t defined in my version of wireshark on windows, I decided to keep working on this issue in my free time so I opened the packet captures at home on my mac and surprisingly enough OS X version of wireshark did show what the unknown 23 extension was.
Extension 23 turned out to be “Extended Master Secret” extension as defined in RFC7627.
What is Extended Master Secret and what is it designed for?
Throughout my research I didn’t come across many documents apart from ietf RFCs which happened to be really helpful. Going back to them let’s take a look at RFC 7627.
Generating Master Secret in TLS (Without Extended Master Secret)
Before extended master secret extension was introduced, parties communicating under TLS protocol (RFC 5246) would compute master secret with the following formula:
master_secret = PRF(pre_master_secret, "master secret", ClientHello.random + ServerHello.random) [0..47];
Explaining the paramters
As defined in RFC 5246 The Transport Layer Security (TLS) Version 1.2:
The master secret is always exactly 48 bytes in length. The length
of the premaster secret will vary depending on key exchange method.
When RSA is used for server authentication and key exchange, a 48-byte pre_master_secret is generated by the client, encrypted under the server’s public key, and sent to the server. The server uses its private key to decrypt the pre_master_secret. Both parties then convert the pre_master_secret into the master_secret, as specified
A conventional Diffie-Hellman computation is performed. The negotiated key (Z) is used as the pre_master_secret, and is converted into the master_secret, as specified above. Leading bytes of Z that contain all zero bits are stripped before it is used as the pre_master_secret.
Note: Diffie-Hellman parameters are specified by the server and may be either ephemeral or contained within the server’s certificate.
pre_master_secret is the value generated by the client using a CSPRNG and some other variables, it is then transferred to the sever in encrypted format using server’s public key. Using the equation above both client and the server can generate the same master_secret which is then used to encrypt the transmitted data using symmetric encryption. The reason why pre_master_secret is transferred and master_secret isn’t sent directly has to do with modularity of the SSL protocol. There are several cipher suites that can be used (RSA…, DH…, ECDH…) and these algorithms generate different format of pre_master_secrets . Using master_secret we can take any type of pre_master_secret (suppose it’s the PSK here) and come up with a common format for master_secret such that it’s totally agnostic about the underlying key exchange algorithm.
ClientHello.random and ServerHello.random are parameters that prevent replay attacks (an attacker can use same client random but server random won’t be the same). They are generated by both parties using secure random number generators.
Generating Master Secret in TLS Using Extended Master Secret
This extension was introduced to prevent a certain type of attack as explained by tripwire blog. The attack is done by an evil/hacked CA that generates a rogue certificate for a website, the attacker then uses this rogue certificate which is trusted by clients to implement a MITM scenario, RFC 7627 explains the attack:
As described in [TRIPLE-HS], in both the RSA and DHE key exchanges, an active attacker can synchronize two TLS sessions so that they share the same “master_secret”. For an RSA key exchange where the client is unauthenticated, this is achieved as follows. Suppose a client C connects to a server A. C does not realize that A is malicious and that A connects in the background to an honest server S and completes both handshakes. For simplicity, assume that C and S only use RSA ciphersuites.
1. C sends a “ClientHello” to A, and A forwards it to S.
2. S sends a “ServerHello” to A, and A forwards it to C.
3. S sends a “Certificate”, containing its certificate chain, to A. A replaces it with its own certificate chain and sends it to C.
4. S sends a “ServerHelloDone” to A, and A forwards it to C.
5. C sends a “ClientKeyExchange” to A, containing the “pre_master_secret”, encrypted with A’s public key. A decrypts the “pre_master_secret”, re-encrypts it with S’s public key, and sends it on to S.
6. C sends a “Finished” to A. A computes a “Finished” for its connection with S and sends it to S.
7. S sends a “Finished” to A. A computes a “Finished” for its connection with C and sends it to C.
At this point, both connections (between C and A, and between A and S) have new sessions that share the same “pre_master_secret”, “ClientHello.random”, “ServerHello.random”, as well as other session parameters, including the session identifier and, optionally, the session ticket. Hence, the “master_secret” value will be equal for the two sessions and will be associated both at C and S with the same session ID, even though the server identities on the two connections are different. Recall that C only sees A’s certificate and is unaware of A’s connection with S. Moreover, the record keys on the two connections will also be the same.
This attack is possible because Master Secret only depends on ClientHello.random and ServerHello.random, the proposed fix was to include the certificate and other session dependent parameters in the formula to generate the master secret.
We define a session hash after completion of the handshake as follows:
session_hash = Hash(handshake_messages)
The extended master secret that’s extension type 0x0017 signals both the client and server that Master Secret is generated with this formula:
master_secret = PRF(pre_master_secret, "extended master secret", session_hash) [0..47];
The “session_hash” depends upon a handshake log that includes “ClientHello.random” and “ServerHello.random”, in addition to cipher suites, key exchange information, and certificates (if any) from the client and server. Consequently, the extended master secret depends upon the choice of all these session parameters.
This changes the way that Master Secret is calculated, if BIG-IP or some other proxy that has the server’s certificate and keys is going to decrypt the handshake is unaware of this extension calculates the master secret in the old way and fails to decrypt the handshake and ultimately has to reset the connection. Now in order to fix this issue we either have to update the device so that it understands extended master secret extension or disabled this feature temporarily.
Disabling Extended Master Secret in SChannel (IIS)
If updating the device in order to understand extended master secret is not an option we’d have to disable it at our web server temporarily. To do this we have to add a key in windows registry, using Microsoft’s MS15-121 Security update for Schannel to address spoofing: November 10, 2015 document:
To make these registry changes, follow these steps:
- Click Start, click Run, type regedit in the Open box, and then click OK.
- Locate and then click the following subkey in the registry:
- On the Edit menu, point to New, and then click DWORD Value.
- For the computer that is receiving the connection request, type DisableServerExtendedMasterSecret: REG_DWORD for the name of the DWORD, and then press ENTER.
For the computer that is initiating the connection request, type DisableClientExtendedMasterSecret for the name of the DWORD, and then press ENTER.
- Right-click the new DWORD entry, and then click Modify.
- Type 1 (or any non-zero value) in the Value data box to disable the TLS extension.
Note You do not have to restart the computer after you make changes to the DisableClientExtendedMasterSecret registry settings. However, you must restart the computer if you delete the DisableClientExtendedMasterSecret registry settings.
At this state finally BIG-IP manages to decrypt the handshake and client certificate authentication completes successfully while ASM and LTM are actively doing their job.
Edit (18 Nov 2016): F5 reacted to the issue probably from the forum post, you can see their solution here which tells you to disable extended master secret on server side.
Excellent resource, just excellent. This is exactly what I needed. After 2 weeks of trying to work through an extremely similiar problem, the Extended Master Secret being the issue, I finally had enough clues from my Wireshark capture to get the correct search terms which led me here. Well done Babak.
I’m really glad that sharing my experience saved the head ache of getting around this issue from you sir 🙂
I reported this to F5 and they made a note in their AskF5 Support articles. I referenced your blog for more information.
Saved me a ton of time! Thank you sir.
I had almost the same issue and it turns out that F5 added support for extended master secret from version 13: https://support.f5.com/csp/article/K66202244. The issue was the ids was not able to decrypt/analyze the traffic after upgrading to version 13 and after 1 month we finally fixed it by disabling on the F5 (client) and not on the server side (IIS 2012 r2).
So .. OK … we disabled that on the F5 and now ids is working … but what is not clear to me is that we did the upgrade to version 13 — 2 months before the issue started !! so the extended master secret started to be used suddenly after 2 months and ids stopped “working” !!
And another fact is that the windows 2012 r2 is always kept updated / monthly updates – so it definitely has 2015 update ms15-121 installed.
Anybody any thoughts? Could a be an Schannel bug/issue? Or some other change on the IIS?
I have disabled TLS Extended Master Secret in our ADFS server it does not solve the problème.
After this action ADFS is no reachable and in client side i get SSL/TLS error, any idea on this ?
Can you elaborate on your setup? What error are you getting exactly? What led you to believe that extended master secret is the issue?
This is an amazing post. I am running 12.1.2 and was stumped with why we were having this issue. After reading the details here, and confirming our setup was similar, I made disabled EMS on the server and it worked. When I upgrade to v13+ I’ll remove the registry setting and confirm it still works.
Thank you for taking the time to post to the F5 forums and for posting this amazingly detailed explanation.