As I told yesterday, I was not very satisfied with the updated NSS remediation guide concerning the TCP Split Handshake issue, published after the second round of testing on Cisco and Fortinet devices.
In particular, in case of Cisco, in my opinion the report was poor on details, considering Cisco’s ACL approach suboptimal and definitively coming to the discouraging conclusion that:
Our original results are unchanged, and ultimately Cisco did offer some mitigation steps.
This is clearly in contrast with what stated in the official Cisco post, which declares Cisco ASA firewall not susceptible to the issue, even if, in my opinion, the most disappointing aspect of the story consists in the fact that no other detail is provided on the NSS document, leaving many unresolved questions about the real nature of the issue and the level of vulnerability of Cisco devices.
Since I was really curious to discover were the truth resides, I decided to ask to Cisco Engineers to provide more details on the testing results, and after few hours it is exactly what they kindly did with an accurate and detailed description of the events posted by Joe Karpenko and Omar Santos, the two engineers who took part to the joint session with NSS Labs.
There are 2 connection establishment handshakes associated with this topic, they are as follows:
* Split Handshake (primary concern/issue)
* Simultaneous Open
By default, the Cisco ASA accelerated security path (asp) prevents both the “Split Handshake” and “Simultaneous Open” using the “tcp-dual-open” connection check. The Cisco ASA firewall drops the TCP SYN segment sent from the server (eg: fakestack.rb) when there is an embryonic TCP connection already open between two endpoints.
However, NSS created and demonstrated a brand new test-case which deviates from the 2 connection establishment handshakes mentioned above along with the most commonly used 3-way handshake. This new test-case is not compliant with the TCP connection establishment equirements defined in RFC 793.
For the “Split Handshake”, the first TCP segment sent by the server (fakestack.rb) in response to the clients TCP SYN segment is a TCP ACK segment (also described in the paper, The TCP Split Handshake: Practical Effects on Modern Network Equipment, pg. 200). However, for the new test-case a TCP RST/ACK segment is sent instead. At this point the client would be in a state called SYN_SENT and the server in the SYN_RCVD state.
The protocol specifications for TCP (defined in RFC 793) define how to process TCP segments received in certain states. When an endpoint is in a SYN_SENT state and it receives a TCP RST/ACK segment the endpoint aborts and closes the connection.
During our testing, the client ignores the TCP RST/ACK segment sent by the server (fakestack.rb) and does not abort the connection. Upon seeing the TCP RST/ACK segment sent by the server (fakestack.rb) the Cisco ASA firewall tears down the connection slot. Immediately following the TCP RST/ACK segment sent by the server (fakestack.rb) it sends a TCP SYN segment which initiates a *new* connection establishment and completes a 3-way handshake that complies with the TCP specifications defined in RFC 793.
For the new test-case, access control list rules can be applied using an access-group and used as additional countermeasures to mitigate and prevent unsolicited connection attempts between the endpoints for a TCP conversation when the client does not abort the connection as defined in the RFC protocol specification for TCP.
Given this description of the events, I completely agree with Cisco’s interpretation and I definitively believe there is nothing strange about the behaviour of the ASA firewall, since it immediately tears down the connection slot upon receiving a TCP RST/ACK (how it should be), and immediately allocates a new connection after receiving the new TCP SYN from the server.
Moreover, in the testing scenario the client behaviour does not fit with the TCP RFC. As a matter of fact, page 32 of the TCP RFC 793 states that:
The principle reason for the three-way handshake is to prevent old duplicate connection initiations from causing confusion. To deal with this, a special control message, reset, has been devised. If the receiving TCP is in a non-synchronized state (i.e., SYN-SENT, SYN-RECEIVED), it returns to LISTEN on receiving an acceptable reset. If the TCP is in one of the synchronized states (ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT), it aborts the connection and informs its user. We discuss this latter case under “half-open” connections below.
Not only. Page 37 of the same RFC, on the paragraph “Reset Processing” states that:
In the SYN-SENT state (a RST received in response to an initial SYN), the RST is acceptable if the ACK field acknowledges the SYN.
Which is exactly the occurrence in the above scenario when the client receives the TCP RST/ACK.
The sum of the two assertions definitively means that the tested scenario is probably not fully compliant to RFC 793 since, as stated by Cisco Engineers, upon receiving the TCP RST/ACK from the server, the client should reset the connection, free the socket and revert to LISTEN state, which corresponds, according to RFC 793, to a state waiting for a new connection request from any remote TCP and port.
And even if could be acceptable to perform a test in similar conditions not covered by the RFC 793, similarly I do not find anything strange or suboptimal in deploying ACLs to prevent unsolicited connection attempts between the endpoint. As I told yesterday, a firewall should protect critical assets from unsolicited connections independently from the risk of a TCP Handshake attack…
Again, I would like to thank the Cisco Engineers for their kindness and the transparency with which they quenched my curiosity thirst.
P.S.: A final thought from my youth
Now the big picture is clear! Few years ago, when I was younger and at the end of my short and shining system engineer life, I stumbled upon the curious case of a custom application which suddenly stopped to work after an upgrade of the firewall. Deeper analysis showed that each session of the application used the same TCP port for source and destination (the port number was used to identify the customer sigh!). Moreover the server used to terminate the connection with a TCP RST/ACK and to immediately open a new connection with a SYN packet with the same source and destination port number of the previous session. Does it sound familiar to you after reading the post? Yes it does! At that time we spent many hours on insulting the dangerous mind of the programmer and his strange interpretation of the TCP-IP RFC (but the client port should not be allocated on the random Ephemeral Port Range from 1024 to 65535?). After many years I got it!: he was a precursor of the TCP Split Handshake attack.
You will be asking which was the firewall that since then proved not to be susceptible to TCP Split Handshake… I will never say it! Not even under torture. I only may say that in order to fix the problem we had to perform a very unlikely tuning on the timeout parameters of the firewall queues…
During these days I enjoyed speaking with many colleagues about the results of the tests and definitively, I must confess that firewalls were not the only entities unaware the TCP Split Handshake, as a matter of fact, none of the professionals I discussed with (of course including me the first time I read about it) were familiar with this method of establishing TCP connections.
Nevertheless the show must go on: professionals must study to stay up-to-date (and learn what TCP Split Handshake is), firewalls (if susceptible to attack) must be fixed in order to learn how TCP Split handshake is correctly handled.
After the surprising findings of the test vendor are running for cover, so I spent half an hour to check the state-of-the-art after some communications from NSS Labs (unfortunately I was not able to attend the webinar of today) and some rumors on the Infosec arena.
Among the manufacturers found susceptible to TCP Split handshake attack during the first round, Palo Alto Networks has released an update (4.0.2) to fix the TCP Split Handshake Evasion, after the fix the manufacturer was able to pass the TCP handshake attack test.
As far as Juniper Networks is concerned, today a communication sent by E-mail by NSS Labs has indicated that this vendor is working on a fix as well: a configuration setting which will be enabled by default for new customers.
But probably the most interesting piece of news is the fact that today some Cisco representatives today went to NSS Labs to participate in the vulnerability-assessment on site and sort out any issues directly. Cisco refused to accept the results of the tests since was not able to reproduce the issue on any tested platform (ASA, IOS Firewall, IPS Appliances). An updated blog post about the findings is expected later today. NSS Labs also expects to publish updated findings related to what firewalls it tested have completed remediation to protect against the TCP Split Handshake attack.
Just for fun…
(But not only!), I gave a look individually to other vendors not involved in the tests to see if they had analyzed the behavior of their technologies on this issue.
Some McAfee representatives indicated me that their Enterprise Firewall platform is not prone to TCP Split Handshake attack. I looked for some information and I found this post from the vendor. Would be interesting if the security manufacturer from Santa Clara could release a more detailed documentation (maybe they already released but I did not find it J).
Stonesoft issued a blog post with the result of the test performed individually on its Stonegate Devices with the same BreakingPoint method pointed out in the original document describing the attack. The finding is that with the only firewall function the security device is not vulnerable if the “strict mode” is enabled in the advanced properties of the node. In normal or loose mode the traffic is permitted (even if Stonesoft indicates that the firewall does not get spoofed, that is correctly recognizes the origin of the session). With the antivirus function enabled the firewall is not vulnerable in any mode.
Astaro except some tweets indicating that the technology is not vulnerable. Would be interesting, also in this case, if the vendor could release some detailed document on the necessary configurations to be implemented to avoid the spoof (or if they are enabled by default).
At this point I look forward to read the result of Cisco/NSS joint tests…
In the same hours in which I was writing the original article concerning the growing attention of utilities and security vendors versus SCADA security holes; an anonymous hacker put in practice the lesson and broke into wind turbine systems. He was able to break a 200 megawat wind turbine system owned by NextEra Energy Resources, a subsidiary of Florida Power & Light, claiming revenge for an “illegitimate firing”. Having said that it is not yet known whether or not it is an hoax (Wind power company sees no evidence of reported hack), the data was posted to the Full Disclossure security mailing list Saturday anonymously, by someone using the name “Bgr R.” In the post, the author of the hack wrote:
Here comes my revenge for illegitimate firing from Florida Power & Light Company (FPL)
… ain’t nothing you can do with it, since your electricity is turned off !!!
Secure you SCADA better! Leaked files are attached …
In an e-mail interview, Bgr R said he’s a former employee who discovered a vulnerability in the company’s Cisco security management software. He used that vulnerability to hack into the SCADA (supervisory control and data acquisition) systems used to control the turbines.
Even if the screenshots of the Wind Turbine management interface look legitimate, there are some big question marks. In his interview Bigr R didn’t say much about how he broke into the SCADA systems themselves and he didn’t demonstrate much insider knowledge of Florida Power & Light (FPL) systems.
Hoax or not, this event renews the attention on SCADA Security Issues… For my part I promise I will no longer write down Security Predictions :-)
Update May 12: TCP Split Handshake: Why Cisco ASA is not susceptible
Update May 11: The Never Ending Story
Update April 21: Other Considerations on TCP Split Handshake
Few days ago, independent security research and testing NSS Labs, issued a comparative report among six network security technologies. The controversial results created a comprehensible turmoil among the security vendors involved in the tests, and more in general inside the infosec landscape. As a matter of fact it turned out that that five of the six tested platforms were susceptible to TCP Split handshake attack.
As a security professional, I am pretty much involved with at least five of the six tested technologies, consequently, although I never heard about TCP Split Handshake before, I must confess I was really curious to learn which was the only platform capable of surviving the test (the answer is indirectly provided by the vendor – Checkpoint – missing from the list contained on the remediation report subsequently released). Fortunately the scientific side of me took over and instead of making judgments and drawing conclusions about the results, I decided to learn more about TCP Split Handshake and the reasons why a security equipment may be vulnerable.
TCP Split Handshake in RFC 793
Since TCP is a connection-oriented protocol, every connection begins with a “handshake” defined in RFC 793. The handshake defines three well defined steps and for this reason it is called “TCP Three Way Handshake.”
The host initiating the connection, referred as the client, send to its peer, referred as the server, a synchronization packet, or SYN. In order to correctly identify the beginning (and the subsequent “state” of the session, the SYN packet contains an initial Sequence Number (ISN) which corresponds to a pseudo-random number.
Upon reception of the SYN packet, the server acknowledges that, and generates its own SYN. This “SYN/ACK” packet contains both the server’s Initial Sequence Number, as well as an acknowledgment number equal to the client’s Sequence Number plus 1. The fact that the server sends a single packet to initiate the connection on its side and to acknowledge the initial SYN sent from the client is known as piggy-backing and, as explained later, is the fundamental aspect in which TCP Split Handshake differs from Three Way Handshake.
At this point, in order to establish the session, the client concludes the Three Way Handshake and acknowledges the server’s SYN/ACK, sending a packet with its own ISN incremented by one, as well as its acknowledgement number equal to the server’s ISN plus 1.
As mentioned above, in the second phase of the handshake, the piggy-backing allows the server to use a single packet to send its own SYN and to acknowledge the SYN packet received from the client (ACK). However, let us assume that the server could decide to split the second phase of the handshake and send a dedicated ACK packet to acknowledge the client SYN, and a further dedicated packet with its own SYN. This is exactly what is stated at section 3.3, page 27, of RFC 793, which introduces an intriguing four-step process:
1) A --> B SYN my sequence number is X 2) A <-- B ACK your sequence number is X 3) A <-- B SYN my sequence number is Y 4) A --> B ACK your sequence number is Y
As a consequence, one might expect that an RFC 793 perfectly compliant client be capable to silently accept packet two, explicitly ACK packet 3, and hence complete the handshake more-or-less normally. At least in theory…
In reality, in such similar circumstances, NSS test have shown that some network security devices, with the sole firewall function enabled, get confused and behaves in a stateless manner. In few words, if even the client behaves as stated in the RFC, that is it is able to correctly establish the session even if it accepts separated ACK and SYN packets from the server, the network security device, on receiving the SYN from the server (packet 2), loses the awareness of the session and lets the traffic flow without enforcing any security control as if it belongs to an uncontrolled session (in theory an unknown or out-of-state session should be blocked). This means that a malicious payload conveyed through a TCP Split Handshake intiated session could go through the firewall and as a consequence, an attack scenario is quite straightforward: an attacker could think to use a server-side malicious script to establish the session by mean of a TCP Split Handshake and consequently install an executable on the client (a very fashionable event in the last days), for instance, by mean of an ActiveX Buffer Overflow on the target client browser.
The bad news is that this kind of attack is not new, and a similar attack scenario was reported for the first time approximately one year ago (with different behaviours reported for clients and security devices). The strange side of the story relies on the fact that this behaviour may not be considered a real vulnerability, but rather an occurrence covered by RFC not correctly implemented or not enabled on the default configuration by security vendors (please consider that RFC 793 also includes a further method for establishing a TCP connection dubbed “TCP Simultaneous Open” in which two TCP hosts simultaneously attempt to open a connection to each other via a SYN packet).
Last but not least…
For the record, as previously stated, NSS Labs released a remediation report containing the indications needed to mitigate (where necessary) the occurrence of the TCP Split Handshake for the affected technologies. Moreover two vendors (Cisco and Fortinet) added some indications as reported in the following:
- According to an official blog post, Cisco was not able to reproduce the issue occurred in NSS Labs Test and is further investigating the TCP Split Handshake attack on its devices.
- According to an official response in a blog post, Fortinet is not susceptible to TCP Split Handshake attack if IPS and Antivirus protections are enabled. A special IPS signature has been developed and a firmware update is scheduled for May in order to block TCP Split Handshake attack with only firewall enabled:
- For Juniper devices the line “set security flow tcp-session strict-syn-check” must be inserted into configuration (this option affects all the traffic, so it must be set with caution);
- Palo Alto is working to release an official fix between mid-April and early May;
- For Sonicwall devices, the option “Enforce Strict TCP Compliance” must be enabled (also in this case this option affects all the traffic and must be set with caution).
- Other Considerations On TCP Split Handshake (paulsparrows.wordpress.com)
How many times, stuck in traffic on a hot August day, we hoped to have a pair of wings to fly through the clouds and free from the wreckage of burning metal.
Unfortunately, at least for me (even if my second name in English would sound exactly like Sparrows) no wing so far, miraculously, popped up to save me, nevertheless I am quite confident that, in a quite near future, I will be saved by the clouds even if I will not be able to fly, or better said, I will be saved by cloud technologies that will help me, and the other poor drivers bottled between the asphalt and the hot metal, under the ruthless August sun to avoid unnecessary endless traffic jams on Friday afternoons.
Some giants of Information Technology (Cisco and IBM in primis) are exploring and experimenting such similar solutions, aimed to provide Car Drivers with real time information about traffic and congestions in order to suggest them the optimal route. In this way they will provide a benefit to the individual, avoiding him a further amount of unnecessary stress, and to the community as well, contributing to fight pollution and making the whole environment more livable and enjoyable.
The main ingredients of this miraculous technological recipe consist in Mobile Technologies and cloud technologies and the reasons are apparently easy to understand: everybody always carries with him a smartphone which is an incommensurable real time probe source of precious data necessary to model a traffic jam (assuming that it will be ever possible to model a traffic jam in the middle of the Big Ring of Rome): as a matter of fact a smartphone allows to provide real-time traffic information correlated with important parameters such as GPS position, average speed, etc.
Cloud technologies provide the engine to correlate information coming from mobile devices (and embedded devices) belonging to many different vehicles, providing the computational (dynamically allocated) resources needed to aggregate and make coherent data from many moving sources in different points of the same city or different interconnected cities. Cloud technologies may act a a single, independent, point of collection for data gathered on each device, dynamically allocating resources on-demand (let us suppose to have, in the same moment, two different jams, one of which is growing to an exponential rate and requires, progressively more and more computational resources), providing, to the individual (and to the City Administrators) a real time comprehensive framework, coherent and updated (nobody would hope to be led, by his navigator, to a diversion with a traffic-jam much worse than the original one which caused the diversion.
Of course, already today many consumer navigators offer the possibility to provide real-time traffic information, anyway the huge adoption of cloud technologies will offer an unprecedented level of flexibility together with the possibility to deal with a huge amount of data and to correlate the collected information with other data sources (for instance the V2V Veichle2Veichle e V2I Veichle2Infrastructure). From the city administrations perspective, the collected data will be invaluable for identifying the more congested points (and drive the subsequent proper targeted corrective actions), and moreover for supporting a coherent and sustainable development of the city.
Cisco and IBM are working hard to make this dream become true in few years with different approaches converging to the cloud: Cisco is leveraging the network intelligence for a project pilot in the Korean City of Busan (3.6 million of inhabitants). Cisco vision aims, in the third phase of the project scheduled before the end of 2014, to provide the citizens with many different mobile services in the cloud, with a Software-As-A-Service approach. Those services are dedicated to improve urban mobility, distance, energy management and safety. A similar project has recently been announced also for the Spanish City of Barcelona.
The IBM project, more focused on applications, is called The Smarter City and aims to integrate all the aspects of city management (traffic, safety, public services, etc..) within a common IT infrastructure. Few days ago the announcement that some major cities of the Globe, for instance Washington and Waterloo (Ontario), will participate to the initiative.
Even if the cloud provides computing power, dynamicity, flexibility and the ability to aggregate heterogeneous data sources at an abstract layer, a consistent doubt remains, and it is represented by security issues for the infrastructure… Apart from return on investment considerations (for which there are not yet consistent statistics because of the relative youth of the case studies depicted above), similar initiatives will succeed in their purpose only if supported by a robust security and privacy model. I already described in several posts the threats related to mobile devices, but in this case the cloud definitely makes the picture even worse because of the centralization of the information (but paradoxically this may also be an advantage if one is able to protect it well.) and the coexistence of heterogeneous data, even though logically separated, on the same infrastructure. As a consequence compromising the only point that contains all the data coming from heterogeneous sources that govern the physiological processes of a city, could have devastating impacts since the system would be affected at different levels and the users at different services. Not to mention, moreover, in case of wider use of this technologies, the ambitions of cyberterrorism that could, with a single computer attack, cripple the major cities around the globe.
In the wake of the infamous LizaMoon which has flooded an impressive number of databases all over the world with SQL Injection, infecting more than 1,500,000 URLs according to Google Search, the next frontier of Information Security to which security vendors are likely to move, is the branch of application security. The last vendor in order of time to make an acquisition (just a couple of days before LizaMoon was detected) was
Intel McAfee, which decided to enter the database security market (estimated more than $ 600 million in 2012) acquiring Sentrigo, a Santa Clara based company focused on database security, former member of the SIA Technology Partnership Program (McAfee Security Innovation Alliance) and currently linked to McAfee by an OEM partnerships.
The red Infosec Colossus of Santa Clara is just the latest player to enter this market, following the example of IBM, which offers a complete Application Security solution since 2009, thanks to the acquisitions (in rigorous chronological order) of DataPower (Web Application/XML Security), Ounce Labs (Code Analysis) and Guardium (Database Security). A set of solutions which form respectively the Websphere, Rational and InfoSphere Guardium Security Solutions.
McAfee and IBM are accompanied by Fortinet, another important security player which has been active in this field for some years. Fortinet has been investing in database and application security since 2006, and even if it lacks a code analysis solution, it offers a portfolio which extends up to the database (scanning and monitoring) level, through the acquisition of IP-Locks, and up to XML /Application Firewall, thanks to its offer of FortiWeb appliances.
As you may notice the three examples above are particularly meaningful of how the security is now converging towards application security. Although McAfee, Fortinet and IBM have very different backgrounds, they are converging to a comparable application security offer: McAfee approached the problem from the endpoint security, which is its historical strength, IBM from the content security, since its adventure in security has started from the acquisition of ISS, and finally Fortinet from the network security, well represented by its Fortigate appliances.
According to my personal model, the complete cycle of application security moves on four levels: training of developers is the first level and the necessary foundation upon which an application security project is built. Where the ability (and security awareness) of developers does not arrive, Vulnerability Assessment/Penetration Test (second level) may be enforced to check the level of security of the applications. If we move to a more “technological” plane there are two more levels: they consist respectively in Code Analysis (a preventive measure) and XML/Application/Database security solutions implemented with dedicated software or appliances (an infrastructural measure). Please consider that (an aspect which is not secondary) these kindw of solutions are also driven by increasingly stringent regulations, such as PCI-DSS, and emerging “De Facto” standards such as OWASP (Open Web Application Security Program).
If IBM is currently the only vendor to cover the three areas of application security (code analysis, XML/Web application security and database security), in addition to McAfee and Fortinet, there are other vendors at the door, looking at this market with great interest starting from Cisco Systems, provided with a great ability to execute, but currently focused primarily on its network-centric approach by mean of its ACE family of XML Firewalls, and HP, which, even if currently leaks an XML/WAF or Database Security solution) is approaching the world of Application Security starting from code analysis thanks to the acquisition of Fortify, considered by many observers the market leader in this field.
Actually, alongside these vendors there are more players which, even if more focused on network security, however, look carefully in this market by offering niche solutions, as is the case, for instance, with Checkpoint, which combines its traditional firewall modules (or software blades according to the current terminology) with Web Security functions specifically tailored for application threat, or also Citrix which approaches the problem from the Application Acceleration/Distribution perspective.
It is likely that the market value and the emotional drive of LizaMoon will soon bring furher earthquakes in this market. In my honest opinion, the next to fall will be two more important partners in the McAfee SIA focused in Application Security: Imperva (Web Application/XML Firewall) and VeraCode (Code Analysis) are well advised…