Since the early days of TCP, port scanning has been used by computer saboteurs to locate vulnerable systems. In a new set of top secret documents seen by Heise, it is revealed that in 2009, the British spy agency GCHQ made port scans a "standard tool" to be applied against entire nations (Figure 1, see the picture gallery). Twenty-seven countries are listed as targets of the HACIENDA program in the presentation (Figure 2), which comes with a promotional offer: readers desiring to do reconnaissance against another country need simply send an e-mail (Figure 3).
The HACIENDA Programm
The documents do not spell out details for a review process or the need to justify such an action. It should also be noted that the ability to port-scan an entire country is hardly wild fantasy; in 2013, a port scanner called Zmap was implemented that can scan the entire IPv4 address space in less than one hour using a single PC.  The massive use of this technology can thus make any server anywhere, large or small, a target for criminal state computer saboteurs.
The list of targeted services includes ubiquitous public services such as HTTP and FTP, as well as common administrative protocols such as SSH (Secure SHell protocol – used for remote access to systems) and SNMP (Simple Network Management Protocol – used for network administration) (Figure 4). Given that in the meantime, port scanning tools like Zmap have been developed which allow anyone to do comprehensive scans, it is not the technology used that is shocking, but rather the gargantuan scale and pervasiveness of the operation. The next section gives background on how port-mapping tools work and what information is gained by using them, making it clear what becomes possible when a state actor uses them at scale.
Background: The TCP Three-Way Handshake
The most commonly-used protocol on the Internet is TCP | the Transmission Control Protocol. Every time an email is sent or a web page is browsed, TCP is the protocol that is used to move data reliably between clients and servers. Port-mapping tools take advantage of a structural problem in TCP in order to determine what services are running on a system. Since the early days of TCP, port scanning has been used by attackers to locate vulnerable systems. Whenever a TCP client wants to communicate with a TCP server, the two parties perform what is called a TCP three-way handshake. The flawed design of this handshake is the foundation for port mapping tools, as during the handshake, the server leaks information about the availability of a service without checking the client's authorization.
Figure 5 illustrates the sequence of TCP packets which are sent to establish a connection. The establishment of the connection works as follows: the host which wants to initiate a connection first sends out a TCP SYN ("synchronize") packet. If the destination host accepts the connection request, it sends a SYN/ACK ("synchronize/acknowledge") packet. After receiving a positive reply, the initiating host sends out an ACK ("acknowledge") packet, which finalizes the TCP three-way handshake. This TCP three-way handshake allows an adversary to easily determine if some TCP service is offered at a given port by a host on the Internet: if the TCP port is closed, the server reacts differently to the TCP SYN packet (Figure 6), sending a RST ("reset") packet instead of the SYN/ACK it would send were the port open. Thus, an adversary can easily map Internet services by considering the differences in the server's replies in the packet flows depicted in Figure 5 and Figure 6 respectively.
- Julian Kirsch is finishing his Master's degree at the Technische Universität München, where he will soon join Prof. Eckert's chair for computer security to pursue a doctorate degree. His research interests include reverse engineering and counter-espionage.
- Christian Grothoff is funded by the Deutsche Forschungsgemeinschaft (DFG) under ENP GR 3688/1-1 until the end of August 2014. He is now moving from the Technische Universität at München to Inria Rennes, where he will start a research team in the area of secure decentralized networks. His research interests include compilers, programming languages, software engineering, networking and security.
- Monika Ermert is a freelancer for heise online and has written on DNS, DNS security and more issues for a many years.
- Jacob Appelbaum is an investigative journalist.
- Laura Poitras is a documentary film maker and journalist living in Berlin.
- Henrik Moltke is an investigative journalist.
The Enemy Online
In addition to simple port scans, GCHQ also downloads so-called banners and other readily available information (Figure 4). A banner is text sent by some applications when connecting to an associated port; this often indicates system and application information, including version and other information useful when looking for vulnerable services. Doing reconnaissance at the massive scale revealed in the documents demonstrates that the goal is to perform active collection and map vulnerable services ubiquitiously, not to go after specific targets.
By preparing for attacks against services offered via SSH and SNMP, the spy agency targets critical infrastructure such as systems used for network operations. As shown in the past with the penetration of Belgacom and Stellar, when an employee's computer system or network credentials may be useful, those systems and people are targeted and attacked.
The database resulting from the scans is then shared with other spy agencies of the Five Eyes spying club (Figure 7), which includes the United States, Canada, United Kingdom, Australia and New Zealand. MAILORDER is described in the documents as a secure transport protocol used between the Five Eyes spy agencies to exchange collected data.
Every device a target
The process of scanning entire countries and looking for vulnerable network infrastructure to exploit is consistent with the meta-goal of "Mastering the Internet", which is also the name of a GCHQ cable-tapping program: these spy agencies try to attack every possible system they can, presumably as it might provide access to further systems. Systems may be attacked simply because they might eventually create a path towards a valuable espionage target, even without actionable information indicating this will ever be the case.
Using this logic, every device is a target for colonization, as each successfully exploited target is theoretically useful as a means to infiltrating another possible target. Port scanning and downloading banners to identify which software is operating on the target system is merely the first step of the attack (Figure 8). Top secret documents from the NSA seen by Heise demonstrate that the involved spy agencies follow the common methodology of online organized crime (Figure 9): reconnaissance (Figure 10) is followed by infection (Figure 11), command and control (Figure 12), and exfiltration (Figure 13). The NSA presentation makes it clear that the agency embraces the mindset of criminals. In the slides, they discuss techniques and then show screenshots of their own tools to support this criminal process (Figure 14, 15 and 16).
The NSA is known to be interested in 0-day attacks, which are attacks exploiting largely unknown vulnerabilities for which no patch is available. Once an adversary armed with 0-day attacks has discovered that a vulnerable service is running on a system, defense becomes virtually impossible. Firewalls are unlikely to offer sufficient protection, whether because administrators need remote access or because spy agencies have already infiltrated the local network (siehe: Barton Gellman and Ashkan Soltani. Nsa infiltrates links to yahoo, google data centers worldwide, snowden documents say. The Washington Post, October 2013). Furthermore, adding additional equipment, such as firewalls administered via SNMP, into an internal network may also open up new vulnerabilities.
Figure 8 points to a particular role that HACIENDA plays in the spy club's infrastructure, namely the expansion of their covert infrastructure. The top secret documents seen by Heise describe the LANDMARK program, a program by the Canadian spy agency CSEC which is used to expand covert infrastructure (Figure 17).
The covert infrastructure includes so-called Operational Relay Boxes (ORBs), which are used to hide the location of the attacker when the Five Eyes launch exploits against targets or steal data (Figure 18). Several times a year, the spy club tries to take control of as many machines as possible, as long as they are abroad. For example, in February 2010 twentyfour spies located over 3000 potential ORBs in a single work day (Figure 19). However, going over the port scan results provided by HACIENDA was considered too laborous (Figure 20), so they programmed their OLYMPIA system to automate the process (Figure 21). As a result, the spies brag that they can now locate vulnerable devices in a subnet in less than five minutes (Figure 22).
The Canadians are not the only ones using HACIENDA to locate machines to compromise and turn into ORBs. At GCHQ, the hunt for ORBs is organized as part of the MUGSHOT program (Figure 23). The GCHQ has also automated the process and claims significant improvements in accuracy due to the automation (Figure 24). Again the information obtained from HACIENDA plays a prominent role (Figure 25). A key point is that with MUGSHOT the GCHQ integrates results from active scans (HACIENDA) as well as passive monitoring (Figure 26), to "understand everything important about all machines on the Internet".
Thus, system and network administrators now face the threat of industrial espionage, sabotage and human rights violations created by nation-state ad- versaries indiscriminately attacking network infrastructure and breaking into services. Such an adversary needs little reason for an attack beyond gaining access and is supported by a multi-billion dollar budget, immunity from prosecu- tion, and compelled collaboration by companies from Five Eyes countries. As a result, every system or network administrator needs to worry about protecting his system against this unprecedented threat level. In particular, citizens of countries outside of the Five Eyes have, as a result of these programs, greatly reduced security, privacy, integrity and resilience capabilities.
Spy agencies are using their powers to commandeer Internet systems for power projection. Their actions follow the standard template of cyber-criminal behav- ior, using reconnaissance through active and passive port scanning to identify potential victims. Given this serious threat, system administrators need to improve their defensive posture and, in particular, reduce the visibility of non- public services. Patching services does not help against 0-day attacks, and firewalls may not be applicable or suffcient. In the second part of our article, we will introduce another option for system administrators to make non-public system administration services less visible for reconnaissance operations. By standardizing such techniques, the Internet community may be able to dampen the ability of security services to master the Internet.
Knocking down the HACIENDA
In this article, we will describe a new port knocking variant that uses the nation-state adversary model, and thus offers some protections against the HACIENDA program, thereby possibly stopping the spy agencies at the reconnaissance stage.
While defending against undisclosed vulnerabilities in public services is rather difficult, minimizing one's visible footprint and thus one's attack surface for administrative services is much easier. Port knocking  is a well-known method for making TCP servers less visible on the Internet. The basic idea is to make a TCP server not respond (positively) to a TCP SYN request unless a particular "knock" packet has been received first. This can be helpful for security, as an attacker who cannot establish a TCP connection also cannot really attack the TCP server.
However, traditional port knocking techniques  generally do not consider a modern nation-state adversary. Specifically, port scans are not the only method an attacker may use to learn about the existence of a service; if the service is accessed via a network where the adversary is able to sniff the traffic, the adversary may observe the connection and thereby deduce the existence of a service. A nation-state attacker may even be able to observe all traffic from the TCP client and perform man-in-the-middle attacks on traffic originating from the client. In particular, with compromised routers in the infrastructure, it is possible to execute a man-in-the-middle attack to take over a TCP connection just after the initial TCP handshake has been completed. An advanced attacker in control of routers may also try to identify the use of insufficiently stealthy port knocks by detecting unusual patterns in network traffic. However, it may still be safe to assume this adversary does not flag a standard TCP handshake as suspicious, as this is way too common.
TCP Stealth is an IETF draft (Julian Kirsch, Christian Grothoff, Jacob Appelbaum, and Holger Kenn: Tcp stealth, August 2014. IETF draft) which describes an easily-deployed and stealthy port knocking variant. TCP Stealth embeds the authorization token in the TCP ISN, and enables applications to add payload protections. As a result, TCP Stealth is hard to detect on the network as the traffic is indistinguishable from an ordinary 3-way TCP handshake, and man-in-the-middle attacks as well as replay attacks are mitigated by the payload protections. TCP Stealth works with IPv4 and IPv6.
TCP Stealth is useful for any service with a user group that is so small that it is practical to share a passphrase with all members. Examples include administrative SSH or FTP access to servers, Tor Bridges, personal POP3/IMAP(S) servers and friend-to-friend Peer-to-Peer overlay networks. The easiest way to use TCP Stealth is with operating system support. TCP Stealth is available for Linux systems using the Knock patch (siehe: Julian Kirsch. Knock, August 2014).. For kernels that include this patch, TCP Stealth support can be added to applications via a simple setsockopt() call, or by pre-loading the libnockify shared library and setting the respective environment variables.
As the mainline Linux currently does not yet offer support for Knock, the kernel of the machine which should be using Knock needs to be patched. Patching the kernel is straightforward:
1. First, obtain the sources of the desired kernel version from https://www. kernel.org if you intend to use a vanilla running kernel. Note that many distributions make adaptations to the kernel and therefore provide custom kernel sources, so one might want to check for the customized kernel sources.
2. Once the kernel sources are available, download the appropriate Knock patch from https://gnunet.org/knock. Note that if you intend to run a kernel version which is not explicitly listed on the Knock website, the best option is to try out the patches of the closest version provided.
3. Change to the directory where the kernel sources reside (replace the <your-version>-part according to your selection of the kernel- and the patch-version) and apply the patches (you can f ind more information on how to apply and revert patches on the kernel source in the kernel.org archives):
$ cd linux-<your-version>/
~/linux $ patch -p1 < /path/to/knock/patch/tcp_stealth_<your-version>.diff
4. Get the configuration of the currently running kernel. There are two widely used methods which can be used interchangeably:
(a) Debianoids maintain a copy of the kernel configuration parameters in the /boot directory. You can copy the config to your current kernel sources using the following command:
~/linux $ cp /boot/config-$(uname -r) .config
(b) Many other distributions compile the kernel with the possibility to read the running kernel's configuration from the /proc/ file system:
~/linux $ zcat /proc/config.gz > .config
(c) If none of the cases above applies for your distribution, you can try to use the default kernel configuration by entering
~/linux $ make defconfig
However, do not expect a convincing kernel to result from this in terms of performance and stability.
5. Choose the defaults for all configuration parameters which are not in your current configuration. A different kernel version might introduce new compile configuration options:
~/linux $ yes "" | make oldconfig
6. Enable Knock in your current configuration by selecting Networking Support > Networking Options > TCP/IP networking > TCP: Stealth TCP socket support in the inter-active menu:
~/linux $ make menuconfig
7. The kernel is now ready for compilation. Enter
~/linux $ make bzImage && make modules
to compile the kernel and all additional modules. Be prepared for the fact that this step can take a long time. If you have a machine with more than one processor core, you can adjust the number of build threads using the -j option to both make commands.
8. If compilation succeeds, install the new kernel and all modules. Afterwards, automatically create a new initramdisk for your newly compiled kernel. If you have sudo installed, enter
~/linux $ sudo make modules_install && sudo make install
otherwise enter the these commands into a root prompt leaving out both sudos.
9. Reboot the machine and instruct your boot manager to boot into the new kernel. You now have a Knock aware machine.
Enabling Knock Using LD PRELOAD
Knock can be used without having to modify the source code of the program. This can be useful in cases where the source code is not available or when inserting the needed libc calls is infeasible (for example due to restrictions imposed by the application logic).
In order to use Knock in existing applications, a dynamic library libknockify is provided. The basic usage of the libknockify shared object to enable Knock for program example program is as follows:
Afterwards, if the application example program communicates via TCP, libknockify will set the respective socket options to enable the use of Knock in the kernel. In the example, the shared secret is derived from the text "shared secret", and the content integrity protection is limited to the first 42 bytes of payload in the TCP stream. If the KNOCK INTLEN variable is not set, content integrity protection is disabled.
Using TCP Stealth via setsockopt()
Application developers can integrate support for TCP Stealth directly into their code. This has the advantage that it is possible to control which TCP connections have TCP Stealth enabled, and it might further improve usability. To enable basic port knocking with a Knock-enabled kernel, the application only needs to perform a single setsockopt() call after creating the TCP socket:
char secret = "This is my magic ID.";
setsockopt (sock, TCP_STEALTH, secret, sizeof (secret));
For content integrity protection, TCP clients need to additionally specify the first bytes of the payload that will be transmitted in a second setsockopt() call before invoking connect():
char payload = "1234";
setsockopt(sock, IPPROTO_TCP, TCP_STEALTH_INTEGRITY, payload, sizeof(payload));
connect (sock, ...);
write (sock, payload, sizeof (payload));
Servers expecting content integrity protection merely need a second setsockopt() call to specify the number of bytes that are expected to be protected by TCP Stealth:
int payload_len = 4;
setsockopt(sock, IPPROTO_TCP, TCP_STEALTH_INTEGRITY_LEN,payload_len, sizeof(payload_len));
Nowadays, most end-user devices access the Internet from behind a gateway router which performs network address translation (NAT). While TCP Stealth was designed to avoid the use of information that is commonly altered by NAT devices, some NAT devices modify TCP timestamps and ISNs and may thus interfere with the port knocking mechanism.
Table 1 summarizes experiments by Honda et al. showing how common ISN modification by NAT devices is in practice. In terms of security, TCP Stealth is limited to the 32 bits of the TCP ISN field; hence, a persistent adversary may still succeed by luck or brute force. However, we believe that TCP Stealth will provide adequate protections against indiscriminate attackers performing untargeted attacks (such as HACIENDA). Moving administrative services to non-standard ports can further decrease the chance of accidental discovery by active port scanners.
While the use of integrity protection with TCP Stealth is technically optional, port knocking without integrity protections offers little security against an adversary that observes network traffic and hijacks connections after the initial TCP handshake. Thus, future network protocols should be designed to exchange key material at the beginning of the first TCP packet. Sadly, this is not the case for SSH, which instead exposes a banner with version information to an attacker well before the cryptographic handshake. Hence, design flaws in the SSH protocol currently require the use of an additional obfuscation patch  to effectively use TCP Stealth integrity protections with SSH.
Technical solutions such as TCP Stealth are one way for administrators to harden their systems by protecting internal TCP services against the attacks by criminals, be they private, commercially motivated or state parties. However, as Linus Neumann of the CCC recently stated in an OpEd for Heise, it may not be possible to win the race in the long run solely through technical means. Without the necessary political will to legally protect, promote and fund secure communication systems, this one-sided battle will continue | and users will lose. Neumann underlined that secure communication systems were possible, but that governments are much more concerned about loss of control than about hardened (and less controllable) networks. Much more political work lays ahead; however, operating system vendors and administrators can improve the situation today by deploying modern security solutions. (jk)