As I told yesterday, I was not very satisfied with the updated NSS remediation guide concerning the TCP Split Handshake issue, published after the second round of testing on Cisco and Fortinet devices.
In particular, in case of Cisco, in my opinion the report was poor on details, considering Cisco’s ACL approach suboptimal and definitively coming to the discouraging conclusion that:
Our original results are unchanged, and ultimately Cisco did offer some mitigation steps.
This is clearly in contrast with what stated in the official Cisco post, which declares Cisco ASA firewall not susceptible to the issue, even if, in my opinion, the most disappointing aspect of the story consists in the fact that no other detail is provided on the NSS document, leaving many unresolved questions about the real nature of the issue and the level of vulnerability of Cisco devices.
Since I was really curious to discover were the truth resides, I decided to ask to Cisco Engineers to provide more details on the testing results, and after few hours it is exactly what they kindly did with an accurate and detailed description of the events posted by Joe Karpenko and Omar Santos, the two engineers who took part to the joint session with NSS Labs.
There are 2 connection establishment handshakes associated with this topic, they are as follows:
* Split Handshake (primary concern/issue)
* Simultaneous Open
By default, the Cisco ASA accelerated security path (asp) prevents both the “Split Handshake” and “Simultaneous Open” using the “tcp-dual-open” connection check. The Cisco ASA firewall drops the TCP SYN segment sent from the server (eg: fakestack.rb) when there is an embryonic TCP connection already open between two endpoints.
However, NSS created and demonstrated a brand new test-case which deviates from the 2 connection establishment handshakes mentioned above along with the most commonly used 3-way handshake. This new test-case is not compliant with the TCP connection establishment equirements defined in RFC 793.
For the “Split Handshake”, the first TCP segment sent by the server (fakestack.rb) in response to the clients TCP SYN segment is a TCP ACK segment (also described in the paper, The TCP Split Handshake: Practical Effects on Modern Network Equipment, pg. 200). However, for the new test-case a TCP RST/ACK segment is sent instead. At this point the client would be in a state called SYN_SENT and the server in the SYN_RCVD state.
The protocol specifications for TCP (defined in RFC 793) define how to process TCP segments received in certain states. When an endpoint is in a SYN_SENT state and it receives a TCP RST/ACK segment the endpoint aborts and closes the connection.
During our testing, the client ignores the TCP RST/ACK segment sent by the server (fakestack.rb) and does not abort the connection. Upon seeing the TCP RST/ACK segment sent by the server (fakestack.rb) the Cisco ASA firewall tears down the connection slot. Immediately following the TCP RST/ACK segment sent by the server (fakestack.rb) it sends a TCP SYN segment which initiates a *new* connection establishment and completes a 3-way handshake that complies with the TCP specifications defined in RFC 793.
For the new test-case, access control list rules can be applied using an access-group and used as additional countermeasures to mitigate and prevent unsolicited connection attempts between the endpoints for a TCP conversation when the client does not abort the connection as defined in the RFC protocol specification for TCP.
Given this description of the events, I completely agree with Cisco’s interpretation and I definitively believe there is nothing strange about the behaviour of the ASA firewall, since it immediately tears down the connection slot upon receiving a TCP RST/ACK (how it should be), and immediately allocates a new connection after receiving the new TCP SYN from the server.
Moreover, in the testing scenario the client behaviour does not fit with the TCP RFC. As a matter of fact, page 32 of the TCP RFC 793 states that:
The principle reason for the three-way handshake is to prevent old duplicate connection initiations from causing confusion. To deal with this, a special control message, reset, has been devised. If the receiving TCP is in a non-synchronized state (i.e., SYN-SENT, SYN-RECEIVED), it returns to LISTEN on receiving an acceptable reset. If the TCP is in one of the synchronized states (ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT), it aborts the connection and informs its user. We discuss this latter case under “half-open” connections below.
Not only. Page 37 of the same RFC, on the paragraph “Reset Processing” states that:
In the SYN-SENT state (a RST received in response to an initial SYN), the RST is acceptable if the ACK field acknowledges the SYN.
Which is exactly the occurrence in the above scenario when the client receives the TCP RST/ACK.
The sum of the two assertions definitively means that the tested scenario is probably not fully compliant to RFC 793 since, as stated by Cisco Engineers, upon receiving the TCP RST/ACK from the server, the client should reset the connection, free the socket and revert to LISTEN state, which corresponds, according to RFC 793, to a state waiting for a new connection request from any remote TCP and port.
And even if could be acceptable to perform a test in similar conditions not covered by the RFC 793, similarly I do not find anything strange or suboptimal in deploying ACLs to prevent unsolicited connection attempts between the endpoint. As I told yesterday, a firewall should protect critical assets from unsolicited connections independently from the risk of a TCP Handshake attack…
Again, I would like to thank the Cisco Engineers for their kindness and the transparency with which they quenched my curiosity thirst.
P.S.: A final thought from my youth
Now the big picture is clear! Few years ago, when I was younger and at the end of my short and shining system engineer life, I stumbled upon the curious case of a custom application which suddenly stopped to work after an upgrade of the firewall. Deeper analysis showed that each session of the application used the same TCP port for source and destination (the port number was used to identify the customer sigh!). Moreover the server used to terminate the connection with a TCP RST/ACK and to immediately open a new connection with a SYN packet with the same source and destination port number of the previous session. Does it sound familiar to you after reading the post? Yes it does! At that time we spent many hours on insulting the dangerous mind of the programmer and his strange interpretation of the TCP-IP RFC (but the client port should not be allocated on the random Ephemeral Port Range from 1024 to 65535?). After many years I got it!: he was a precursor of the TCP Split Handshake attack.
You will be asking which was the firewall that since then proved not to be susceptible to TCP Split Handshake… I will never say it! Not even under torture. I only may say that in order to fix the problem we had to perform a very unlikely tuning on the timeout parameters of the firewall queues…
Update May 12: TCP Split Handshake: Why Cisco ASA is not susceptible
On May, the 9th 2011, nearly in contemporary, Cisco Systems and Fortinet, the last two security vendors involved in the TCP Split Handshake affair, which had not yet released a fix for the encountered issue, released two separate posts indicating the result of a second session of tests performed with NSS Labs.
As you will probably know, Cisco Systems was not able to reproduce the issue on its labs and decided to perform a second joint session with NSS Labs on April, the 21st 2011 promising a definitive, resolutive post for the same day. I must confess I have been waiting for a while for the promised post, eager to know the outcome and the likely happy ending of the story. I still did not know I would have had to curb my hunger for knowledge for nearly 20 days (much more than initially expected), and (unfortunately) I would also have had to renounce to the happy ending as well (at least for Cisco).
Analyzing singularly (and in alphabetical order) the two vendors:
In an update to its initial post, dated May, the 9th 2001, Cisco stated that after a thorough investigation of the TCP Split Handshake issue raised by NSS Labs, the company has confirmed that the Cisco ASA firewall is not susceptible to this issue. In all test cases examined, the ASA operates as expected, providing protection in its default configuration against the Split-Handshake as defined in the original TCP Split Handshake paper. As a result, the Cisco PSIRT (Product Security Incident Response Team) closed this investigation on May 4th.
Moreover, during the two recent visits to NSS Labs, Cisco was presented with a number of scenarios, including new test cases that deviated from the original Split-Handshake scenario. The Cisco PSIRT collected traces and provided feedback to NSS Labs on all scenarios. In each case, Cisco demonstrated successful network protection through the default ASA configuration or the implementation of firewall policies that are fully supported, documented and used pervasively in enterprise deployments.
Similarly, in a nearly contemporary update to its initial post, Fortinet announced the release, on April the 20th 2011, of an update for their FortiGate platform to correctly handle and block the TCP split handshake attack technique. This fix was subsequently tested by NSS Labs, which recognized its effectiveness on permanently addressing the TCP split handshake issue with just the FortiGate firewall function enabled. The patch applies to FortiOS 4.0 MR2 and is available for download, for customers with a forticare contract, on the Company FortiCare support portal. An update to FortiOS 4.0 MR3 is scheduled in the near future.
All’s well that ends well?
Not really! NSS labs has released an update to their remediation guidance freely available, upon registration, at this link. If the document states that, after the update, the Fortinet platform is no more vulnerable to the TCP Split Handshake:
Update: On April 21, 2011 Fortinet provided NSS Labs FortiOS 4.0 MR2 Patch 6. NSS Labs has confirmed that with the patch applied, Fortinet provides protection against the TCP Split Handshake.
In case of Cisco the situation is not so univocally resolved:
Update on May 6: Over the past several weeks, NSS Labs has worked with Cisco, providing numerous configurations, PCAPs and live demonstrations of the TCP Split Handshake getting past a Cisco ASA. Our original results are unchanged, and ultimately Cisco did offer some mitigation steps. Unlike every other tested vendor, Cisco’s approach to defend against the TCP Split Handshake is based upon Access Control Lists (ACLs). An ACL centric approach is suboptimal since it requires firewall administrators to follow best practices as well as have a low-level understanding of how the TCP Split Handshake works in order to avoid an accidental “misconfiguration” that enables the attack. And there are some firewall configurations in which using an ACL will not be possible.
In practice, according to NSS Labs, it looks like (but this is my personal interpretation since the above phrase does not provide enough details), Cisco devices block the TCP Split Handshake if a proper Access Control List is in place. Unfortunately it is not specified if the ACL must permit the traffic (that is an allowed connection showing the TCP Split Handshake pattern is blocked) or must deny it (that is a blocked connection showing the TCP Split Handshake pattern is blocked as it should be). In any case the ACL-based approach is not considered optimal since it requires direct intervention (and configuration) from the Administrators (and a good knowledge of how TCP Split Handshake).
I must confess, if both assumptions are correct, that in any caseI do not completely agree with NSS Labs conclusions. Firstly in my ideal world firewall should be managed by skilled administrators knowing what they do and moreover which could be the impact of configuration changes on possible attack vectors (ok I did not know the occurrence of TCP Split Handshake before the NSS Labs affair, but nobody’s perfect!). Secondly if a critical resource should lack an ACL (or should be the unintended victim of an accidental “misconfiguration”), this could potentially be more dangerous than a “simple” TCP Split Handshake attack since in that case the target resource could be exposed to a pretty much wider range of threats…
Meanwhile I was too curious and I kindly asked to Cisco to provide more details… I will update the post as soon as I will have any…
During these days I enjoyed speaking with many colleagues about the results of the tests and definitively, I must confess that firewalls were not the only entities unaware the TCP Split Handshake, as a matter of fact, none of the professionals I discussed with (of course including me the first time I read about it) were familiar with this method of establishing TCP connections.
Nevertheless the show must go on: professionals must study to stay up-to-date (and learn what TCP Split Handshake is), firewalls (if susceptible to attack) must be fixed in order to learn how TCP Split handshake is correctly handled.
After the surprising findings of the test vendor are running for cover, so I spent half an hour to check the state-of-the-art after some communications from NSS Labs (unfortunately I was not able to attend the webinar of today) and some rumors on the Infosec arena.
Among the manufacturers found susceptible to TCP Split handshake attack during the first round, Palo Alto Networks has released an update (4.0.2) to fix the TCP Split Handshake Evasion, after the fix the manufacturer was able to pass the TCP handshake attack test.
As far as Juniper Networks is concerned, today a communication sent by E-mail by NSS Labs has indicated that this vendor is working on a fix as well: a configuration setting which will be enabled by default for new customers.
But probably the most interesting piece of news is the fact that today some Cisco representatives today went to NSS Labs to participate in the vulnerability-assessment on site and sort out any issues directly. Cisco refused to accept the results of the tests since was not able to reproduce the issue on any tested platform (ASA, IOS Firewall, IPS Appliances). An updated blog post about the findings is expected later today. NSS Labs also expects to publish updated findings related to what firewalls it tested have completed remediation to protect against the TCP Split Handshake attack.
Just for fun…
(But not only!), I gave a look individually to other vendors not involved in the tests to see if they had analyzed the behavior of their technologies on this issue.
Some McAfee representatives indicated me that their Enterprise Firewall platform is not prone to TCP Split Handshake attack. I looked for some information and I found this post from the vendor. Would be interesting if the security manufacturer from Santa Clara could release a more detailed documentation (maybe they already released but I did not find it J).
Stonesoft issued a blog post with the result of the test performed individually on its Stonegate Devices with the same BreakingPoint method pointed out in the original document describing the attack. The finding is that with the only firewall function the security device is not vulnerable if the “strict mode” is enabled in the advanced properties of the node. In normal or loose mode the traffic is permitted (even if Stonesoft indicates that the firewall does not get spoofed, that is correctly recognizes the origin of the session). With the antivirus function enabled the firewall is not vulnerable in any mode.
Astaro except some tweets indicating that the technology is not vulnerable. Would be interesting, also in this case, if the vendor could release some detailed document on the necessary configurations to be implemented to avoid the spoof (or if they are enabled by default).
At this point I look forward to read the result of Cisco/NSS joint tests…
Update May 12: TCP Split Handshake: Why Cisco ASA is not susceptible
Update May 11: The Never Ending Story
Update April 21: Other Considerations on TCP Split Handshake
Few days ago, independent security research and testing NSS Labs, issued a comparative report among six network security technologies. The controversial results created a comprehensible turmoil among the security vendors involved in the tests, and more in general inside the infosec landscape. As a matter of fact it turned out that that five of the six tested platforms were susceptible to TCP Split handshake attack.
As a security professional, I am pretty much involved with at least five of the six tested technologies, consequently, although I never heard about TCP Split Handshake before, I must confess I was really curious to learn which was the only platform capable of surviving the test (the answer is indirectly provided by the vendor – Checkpoint – missing from the list contained on the remediation report subsequently released). Fortunately the scientific side of me took over and instead of making judgments and drawing conclusions about the results, I decided to learn more about TCP Split Handshake and the reasons why a security equipment may be vulnerable.
TCP Split Handshake in RFC 793
Since TCP is a connection-oriented protocol, every connection begins with a “handshake” defined in RFC 793. The handshake defines three well defined steps and for this reason it is called “TCP Three Way Handshake.”
The host initiating the connection, referred as the client, send to its peer, referred as the server, a synchronization packet, or SYN. In order to correctly identify the beginning (and the subsequent “state” of the session, the SYN packet contains an initial Sequence Number (ISN) which corresponds to a pseudo-random number.
Upon reception of the SYN packet, the server acknowledges that, and generates its own SYN. This “SYN/ACK” packet contains both the server’s Initial Sequence Number, as well as an acknowledgment number equal to the client’s Sequence Number plus 1. The fact that the server sends a single packet to initiate the connection on its side and to acknowledge the initial SYN sent from the client is known as piggy-backing and, as explained later, is the fundamental aspect in which TCP Split Handshake differs from Three Way Handshake.
At this point, in order to establish the session, the client concludes the Three Way Handshake and acknowledges the server’s SYN/ACK, sending a packet with its own ISN incremented by one, as well as its acknowledgement number equal to the server’s ISN plus 1.
As mentioned above, in the second phase of the handshake, the piggy-backing allows the server to use a single packet to send its own SYN and to acknowledge the SYN packet received from the client (ACK). However, let us assume that the server could decide to split the second phase of the handshake and send a dedicated ACK packet to acknowledge the client SYN, and a further dedicated packet with its own SYN. This is exactly what is stated at section 3.3, page 27, of RFC 793, which introduces an intriguing four-step process:
1) A --> B SYN my sequence number is X 2) A <-- B ACK your sequence number is X 3) A <-- B SYN my sequence number is Y 4) A --> B ACK your sequence number is Y
As a consequence, one might expect that an RFC 793 perfectly compliant client be capable to silently accept packet two, explicitly ACK packet 3, and hence complete the handshake more-or-less normally. At least in theory…
In reality, in such similar circumstances, NSS test have shown that some network security devices, with the sole firewall function enabled, get confused and behaves in a stateless manner. In few words, if even the client behaves as stated in the RFC, that is it is able to correctly establish the session even if it accepts separated ACK and SYN packets from the server, the network security device, on receiving the SYN from the server (packet 2), loses the awareness of the session and lets the traffic flow without enforcing any security control as if it belongs to an uncontrolled session (in theory an unknown or out-of-state session should be blocked). This means that a malicious payload conveyed through a TCP Split Handshake intiated session could go through the firewall and as a consequence, an attack scenario is quite straightforward: an attacker could think to use a server-side malicious script to establish the session by mean of a TCP Split Handshake and consequently install an executable on the client (a very fashionable event in the last days), for instance, by mean of an ActiveX Buffer Overflow on the target client browser.
The bad news is that this kind of attack is not new, and a similar attack scenario was reported for the first time approximately one year ago (with different behaviours reported for clients and security devices). The strange side of the story relies on the fact that this behaviour may not be considered a real vulnerability, but rather an occurrence covered by RFC not correctly implemented or not enabled on the default configuration by security vendors (please consider that RFC 793 also includes a further method for establishing a TCP connection dubbed “TCP Simultaneous Open” in which two TCP hosts simultaneously attempt to open a connection to each other via a SYN packet).
Last but not least…
For the record, as previously stated, NSS Labs released a remediation report containing the indications needed to mitigate (where necessary) the occurrence of the TCP Split Handshake for the affected technologies. Moreover two vendors (Cisco and Fortinet) added some indications as reported in the following:
- According to an official blog post, Cisco was not able to reproduce the issue occurred in NSS Labs Test and is further investigating the TCP Split Handshake attack on its devices.
- According to an official response in a blog post, Fortinet is not susceptible to TCP Split Handshake attack if IPS and Antivirus protections are enabled. A special IPS signature has been developed and a firmware update is scheduled for May in order to block TCP Split Handshake attack with only firewall enabled:
- For Juniper devices the line “set security flow tcp-session strict-syn-check” must be inserted into configuration (this option affects all the traffic, so it must be set with caution);
- Palo Alto is working to release an official fix between mid-April and early May;
- For Sonicwall devices, the option “Enforce Strict TCP Compliance” must be enabled (also in this case this option affects all the traffic and must be set with caution).
- Other Considerations On TCP Split Handshake (paulsparrows.wordpress.com)