Abstract: Despite the conventional wisdom that proactive security is superior to reactive security, we show that reactive security can be competitive with proactive security as long as the reactive defender learns from past attacks instead of myopically overreacting to the last attack. Our game-theoretic model follows common practice in the security literature by making worst-case assumptions about the attacker: we grant the attacker complete knowledge of the defender’s strategy and do not require the attacker to act rationally. In this model, we bound the competitive ratio between a reactive defense algorithm (which is inspired by online learning theory) and the best fixed proactive defense. Additionally, we show that, unlike proactive defenses, this reactive strategy is robust to a lack of information about the attacker’s incentives and knowledge.

Discussion: This 2010 paper, a collaboration between security and machine learning researchers, makes the bold claim that rather invest your resources in making your system as secure as possible up-front, in many scenarios it’s just as good — or even preferable — to fix security problems based on what systems attackers target, a paradigm they call reactive security. It justifies this with a simple game-theoretic model in which a defender with finite resources, typically a high-level security manager, must allocate them among particular lower-level security tasks.

The primary model used by the paper, of interest on its own, is a two-player game taking place on a graph: the attacker begins at a start vertex and moves through the graph by executing successful attacks. Each vertex has a payoff value that the attackers receives once that vertex is reached, and each edge has a cost, indicating how much the attacker must pay to cross it (representing effort invested in an attack). The initial cost of an edge is based on the surface area of the system being attacked, and the defender can invest in defending an edge, temporarily increasing its cost for one round, but has only finite resources to go around during each round. Attackers and defenders alternate in rounds: the defender picks a set of edges to invest in, then the attacker gets to execute a series of attacks, always beginning at the start vertex. Finally, edge costs are hidden to the defender until the attacker uses the edge; this models how it’s difficult to determine a priori where security weaknesses in a system are.

Although the examples in the paper have vertices representing system components like machines on a network, I think vertices in the graph are best thought of not as particular systems being exploited, but rather a set of resources controlled by the attacker, or more generally, the current attacking capabilities of the attacker. At the start vertex, they control nothing but their own system; as each edge is traversed, they add new attacking resources to their collection. This allows systems like the one shown to the right, where an attacker may or may not have full control over a front-end system before attacking a back-end system, to be modelled.

The main result of the paper is this: under the assumption that the defender’s investment in an edge is linearly reflected in the attacker’s cost for that edge, a specific machine learning algorithm for reactive security (based on placing more weight on recently attacked nodes and exponentially decaying the weight over time) performs comparably to a pure proactive approach, in which the defender knows all edge costs a priori and picks a fixed, optimal strategy. This is done essentially by reducing the problem to a standard online learning problem and using known results. Moreover, they argue that in cases where the edge costs are estimated incorrectly or the attacker acts unexpectedly due to incomplete knowledge, the reactive security algorithm is superior to the proactive approach. Besides providing support for reactive security, this model also provides formal support for other pragmatic measures like defense in depth, which is investing in defense measures that are never needed unless some other defensive measure is first overcome.

Although the model is simple, it is quite general: not only can vertices represent machines on a LAN, but also components of an application, or a hybrid thereof. Traversing an edge can correspond to an attack from outside, a privilege elevation on a single machine, or to an attack between machines on a LAN. Two different attacks on the same system can be expressed by distinct edges with distinct costs, if the attacker possesses a different set of resources in each case, which makes sense. Even social engineering can be modelled: once the attacker has invested in tricking or bribing the employee, they can add the employee to their set of resources, lowering the cost of attacking new systems.

On the other hand, the model also has a number of weaknesses. A couple were made explicit in the paper:

• The model assumes that the mapping from defender investment in an edge to attacker cost to overcome it is linear. This is justified heuristically with the claim that the rate of discovering new security defects in software is roughly constant. Experience shows however that exploits that are easy to exploit can be hard to fix (e.g. DoS attacks), and exploits that are hard to exploit can be easy to fix (e.g. a buffer overflow in code that doesn’t directly handle user input). Moreover, the same “investment” in a system can be spent many different ways, and the model offers little insight into which way is most effective. One solution is to successively use the model itself to decompose vertices into subgraphs.
• Reactive security is unsuitable for dealing with situations in which an attack is so devastating to the company that it cannot recover. For example, a company that suffers a high-profile blow to its reputation in the press may see a drop in sales that ultimately drives it into bankruptcy, or may have major investors pull out. Even an optimal response in such a scenario is too little, too late.

Some other weaknesses are less obvious:

• The assumption that edge costs become known as soon as an attacker exploits that edge. In reality, the detection of a single exploit of a system gives very little information about the overall security of a system, particularly after that exploit is repaired. The frequency of exploits of an edge might be a better indicator (the learning may already effectively take this into account).
• The assumption that edges have a fixed cost which only increases temporarily if the defender invests in them. In reality, the cost of an attack depends on many dynamic factors, including the skill set and resources available to the attacker marketplace, and the behavior of the system under attack. As new attackers enter the attacker marketplace, as attackers learn new techniques, as transaction costs among attackers change (or as they form teams), or as new hacking tools become available, the cost of attack can go up or down. Patches, upgrades, installation of new applications, or even changes in load can expose new vulnerabilities or fix old ones (in particular, patches implemented in reaction to particular attacks do not go away after the round is over). All these factors can change quickly and are largely invisible to the defender.
• Similarly, payoff is assumed to be fixed from round to round. Payoff changes by the minute based not only on what new information the attacked resources are storing, but also the marketplace value of the information, which can be rapidly shifting and unpredictable. Reactive investment in defense of a system that, tomorrow, is of little interest to attackers is a bad idea.
• Edge costs are assumed to be independent, in the sense that investing in one edge does not affect the cost of any other edge. In reality, as is easy to see in the diagram above, improvements to one edge may quite directly affect the cost of another related edge as well. Another common example would be rolling out an operating system upgrade in the data center, which could increase the cost of all edges simultaneously.
• Attackers are assumed, after each round, to lose control over all their attacked resources. Long-lived attacks such as rootkits can go undetected during attempts to clean up after an attack, allowing the attacker to start in the next round with more resources in the bag at the beginning. In the worst case, the rootkit itself delivers a positive payoff to the attacker, and the attacker doesn’t need to take any additional action at all!

Although the system provides formal evidence that reactive security can be beneficial, it also provides formal evidence that extreme reactive security, pouring all your resources into the system that was just attacked last week, is a terrible strategy. Simple attack strategies which alternate between systems can exploit this kind of reactionary tactic.

Despite the many weaknesses and limitations enumerated above, as I would expect to find in any nascent research area, I find this work exciting and think it opens up a range of possibilities for software security management and challenges the intuition that reacting to attackers is impulsive or short-sighted. Perhaps future work in this area may provide richer models that will offer more new and surprising strategies to defenders.

Abstract: We introduce the notion of a conditioned-safe ceremony. A “ceremony” is similar to the conventional notion of a protocol, except that a ceremony explicitly includes human participants. Our formulation of a conditioned-safe ceremony draws on several ideas and lessons learned from the human factors and human reliability community: forcing functions, defense in depth, and the use of human tendencies, such as rule-based decision making. We propose design principles for building conditioned-safe ceremonies and apply these principles to develop a registration ceremony for machine authentication based on email. We evaluated our email registration ceremony with a user study of 200 participants. We designed our study to be as ecologically valid as possible: we employed deception, did not use a laboratory environment, and attempted to create an experience of risk. We simulated attacks against the users and found that email registration was significantly more secure than challenge question based registration. We also found evidence that conditioning helped email registration users resist attacks, but contributed towards making challenge question users more vulnerable.

Discussion: This paper from NDSS 2009 introduces conditioned-safe ceremonies, an informal model for security protocols that explicitly models the actions of users. Rather than conservatively considering users to be unpredictable agents capable of any action, it takes advantage of their properties as creatures of habit to help facilitate the desired, secure outcome.

Many of us are familiar with the problem of security warnings sometimes known as click fatigue: when a user is asked to dismiss a security warning frequently during normal operation, they begin to disregard it in all situations. This was for example the primary criticism of Windows Vista’s User Account Control (UAC) feature. There are two reasons for this: one is that security is not the primary concern of users who are focused on completing their primary task; the other is that humans asked to perform a process repeatedly will naturally begin to streamline the process by omitting optional steps and completing mandatory steps using rapid rule-based processing and simple pattern matching. If a situation called for a particular response in the past, visually similar stimuli will encourage users to perform the same task nearly automatically. In psychology this kind of decision-making strategy that settles upon an adequate solution is known as satisficing, and is very difficult to reverse.

Unfortunately this is precisely the type of user behavior exploited by phishers: a typical user presented with a log-in form resembling one they have used many times in the past will thoughtlessly enter their credentials, having long since eliminated the optional steps of carefully examining the security indicators that would expose it as a fraudulent replica.

A cryptographic protocol – such as SSL – is usually described in terms of a number of nodes representing participating machines which exchange messages over channels. A ceremony, coined by Intel’s Jesse Walker, extends the concept of protocols by incorporating nodes for the human users themselves and explicitly representing communication between users and their machines via I/O devices (Carl Ellison. Ceremony Design and Analysis. Cryptology ePrint Archive, Report 2007/399, 2007). This model opens up opportunities for modelling user behavior.

A conditioned-safe ceremony is one designed under the assumption that users will satisfice and behave according to habit; it operates by conditioning the user to follow certain rules during the ceremony. A simple example of conditioning is the Windows log-in screen, which asks the user to press CTRL+ALT+DEL before logging in. This key signals to the operating system that the information entered in the log-in dialog should not be made available to applications or keyboard interception drivers. Because users are always asked to do this, and are not permitted to skip this step, they develop a consistent habit of doing so. The inability of a user to log in without first pressing this key is called a forcing function: forcing functions encourage conditioning and discourage omitting steps which may seem unimportant.

A conditioned-safe ceremony should satisfy several important properties:

• It should only condition safe rules – “rules that are harmless to apply in the presence of an adversary.”
• It should condition at least one immunizing rule – “a rule which when applied during an attack causes the attack to fail.”
• Conditioned rules should be safe to follow under all circumstances, without any complex decision-making.
• It should not assume users will reliably perform any action that is not conditioned by the ceremony.

There are two different types of errors a user can make during a ceremony which may threaten its security:

• An error of omission: The user was expected to apply a rule but took no action.
• An error of commission: The user took an unexpected action not conditioned by the ceremony.

An attacker may attempt to induce either type of error. For example, if an application pops up a Windows log-in box, a user may unthinkingly enter their password without pressing the protective CTRL+ALT+DEL hotkey, because they were not instructed to do so in this instance. An error of commission is usually induced by the attacker giving the user specific instructions, such as “visit this URL in your web browser” or ” Users tend to be suspicious of unfamiliar instructions, making attacks of this type more difficult. An ideal conditioned-safe ceremony should protect against as many errors of both types as possible.

The paper presents an example conditioned-safe ceremony for machine validation: the first time a user logs into a site from a particular machine, they must validate their identity. To do this, the site sends them an e-mail containing a link that they must click; after the link is clicked, a cookie is installed and the user has full access to the site from their current machine. The link only works once. The goal of the attacker is to trick the user into not clicking on the link in the e-mail, instead giving it to the attacker; to accomplish this, they display a phishing web page giving specific instructions on how to do this.  This involves both an error of omission (not clicking on a link that they usually click on), and an error of commission (pasting the link into the website, an action they do not normally take). The expectation is that, if users tend to perform actions they are accustomed to, they will ignore or fail to complete the attacker’s instructions, and the attack will fail.

Sure enough, the experiment bears this out: although as many as 40% of users fall for the attack described above, an alternative design that does not follow the principles of conditioned-safe ceremonies leads to attack  success rates of over 90%. Interviews with the subjects who didn’t fall for the attack show that over half of them didn’t notice the attacker’s special instructions, or thought they were unimportant – the same inattentive attitude that makes security warnings useless now benefits users!

The most exciting thing about this work to me is that it’s one of the first to adopt and exploit a successful model of human behavior in the design of security protocols – a critical step, as humans all too often remain the weakest link in any secure system. Previous efforts have given great insight into the types of errors people make, but not into how designs can work around these limitations in human behavior.

On the other hand, the informal model presented in this work stands in stark contrast to the mathematical models used in cryptography, where cryptographic protocols are routinely subjected to formal verification techniques such as model checking and theorem proving – an important future direction is to generalize these same tools to ceremonies. Moreover, although satisficing behavior is evidently an important component in user behavior, it is obviously not the only such component: 40% of users were persuaded to commit multiple errors in the ceremony. It should come as no surprise that humans are complex creatures that cannot be adequately modelled by a simple set of conditioned rules. In this case, what processes underlie these divergent behaviors, and how can they be modelled? Another important question involves modelling of errors, or divergence of users from the model: can we empirically predict the likelihood of certain sets of errors occurring, and then formally validate that attacks are not possible  in the most likely scenarios? The attacks in this work were relatively ad hoc and don’t seem to rule out the possibility of another attack involving only a single user error.

In short, the area of ceremonies is fertile ground for the development of new models that can effectively predict the behavior of the system as a whole, facilitating the development of protocols that will subtly push users towards making all the right security decisions, even when security is the last thing on their mind.

The author releases all rights to all content herein and grants this work into the public domain, with the exception of works owned by others such as abstracts, quotations, and WordPress theme content.

Abstract: One way to provide fault isolation among cooperating software modules is to place each in its own address space. However, for tightly-coupled modules, this solution incurs prohibitive context switch overhead. In this paper, we present a software approach to implementing fault isolation within a single address space. Our approach has two parts. First, we load the code and data for a distrusted module into its own fault domain, a logically separate portion of the application’s address space. Second, we modify the object code of a distrusted module to prevent it from writing or jumping to an address outside its fault domain. Both these software operations are portable and programming language independent.

Our approach poses a tradeoff relative to hardware fault isolation: substantially faster communication between fault domains, at a cost of slightly increased execution time for distrusted modules. We demonstrate that for frequently communicating modules, implementing fault isolation in software rather than hardware can substantially improve end-to-end application performance.

Discussion: This 1993 paper by Wahbe and Lucco (now at Microsoft), Thomas E. Anderson (now at the University of Washington), and Susan L. Graham describes a software-based method for isolating modules on RISC machines, with immediate application to fine-grained privilege separation. Address spaces, implemented in hardware, are used to isolate processes in modern commodity OS’s, and software fault isolation (SFI) is an alternative with several advantages: most notably, it requires no hardware support, and communication cost between protection domains is much lower.

Background

Suppose a system is divided into modules (components) – for example, a multitasking operating system may have a kernel with various applications running on top of it. A system is said to provide fault isolation if it can recover from the failure of a module without risking the integrity of the rest of the system. For example, if you’re playing a game and it crashes, that shouldn’t cause your web browser – or your entire computer – to crash, or even to malfunction. Fault isolation is valuable for mitigating failures in large software systems where failures are inevitable.

Fault isolation also forms the foundation for a valuable security technique called privilege separation: if each module is only permitted to perform a limited set of certain operations, then even if a module is compromised by an attacker, the attacker only gains access to the privileges held by that module, rather than the entire system. For example, if you’re playing a game it should only have access to the game data files, not the files containing your bank information, which it doesn’t need; then, even if an attacker hijacks your game, they still can’t access your bank information. Privilege separation allows security vulnerabilities in large systems to be mitigated and allows the effort of security verification and auditing to be concentrated on the modules that hold the most dangerous privileges.

In a typical fault isolation system, each module owns some subset of memory where it stores its code and private data structures. To ensure that a misbehaving module does not compromise the integrity of other modules, modules must not be able to write to memory owned by other modules. Moreover, modules must communicate in a controlled manner, usually through an explicit interface, so that other modules can’t trick a module into modifying its own memory maliciously on their behalf.

Today, the primary mechanism used to enforce these properties is dynamic address translation, which maps virtual addresses (memory locations accessed by the application) to physical addresses (locations on the memory device itself) on-the-fly. This functionality is implemented by a specialized piece of hardware called a memory management unit (MMU). By introducing this layer of indirection, the operating system can ensure that a process has no ability to read or write memory belonging to other processes simply by configuring the MMU to provide no mapping from any virtual address to these locations. Because only the operating system kernel can reconfigure the MMU, this protection is secure enough to use with malicious code.

The downside to MMU-based protection is that communication is expensive. Suppose module A wants to send a message to module B. Normally, if both modules had the same view of memory, it would just call module B and pass it a pointer to the message. This takes only a few instructions and is very fast. When two modules have different address space mappings, however, the cost rises dramatically: just switching from running one module to running the other requires a context switch, which involves saving and restoring the complete register set, reconfiguring the MMU, and flushing the MMU’s cache (the translation lookaside buffer). On top of that, it can’t just pass a pointer to the message, because for module B that same pointer may map to a different location in physical memory. There are many different ways to send messages from one process to another, but these all involve overhead.

In applications where the number of messages sent is large, this overhead rapidly becomes prohibitive. An example is a Gigabit Ethernet driver, which has to send a new packet to the network stack about every 12 microseconds. Considering the CPU time needed to parse the headers, this leaves very little time for expensive context switches. As a consequence, in practice components like this are simply not modularized; if they crash, the system crashes.

More generally, any system that is decomposed into fine-grained modules will tend to exhibit a lot of communication between modules – the performance disadvantage of doing this under the MMU model outweighs the security and robustness advantages of having smaller modules.

Software fault isolation (SFI)

Software-based fault isolation, or SFI, aims to provide a substantially different model for fault isolation emphasizing fast communication between modules. In this model, all modules have the same view of memory (run in the same address space), but different subsets of memory are owned by different modules, and every write to memory is checked to make sure the current module owns that memory. Additionally, function calls between modules are controlled so that each module can only be entered at a predetermined set of entry points described in its interface specification.

Despite the name, the important thing here is not that these mechanisms are implemented in software – this offers the convenience of deploying the solution on existing commodity platforms, but is not essential, and indeed most SFI systems rely on some creative combination of software mechanisms and (existing) hardware mechanisms.

In its simplest form, this model is straightforward to provide: imagine you have a trusted compiler and you use it to compile an application consisting of multiple modules.  Each module is assigned a fixed range of memory for its code and data. Whenever the compiler emits a write instruction, it also emits a check to make sure that the address being written to is in the range of the module being executed. Likewise, whenever the compiler emits an indirect branch or jump, it inserts a check to make sure that that jump either lies within the current module, or is a valid entry point of some other module. This simple solution has two main problems:

1. It’s really slow.
2. It’s possible for a malicious module to circumvent the checks. For example, it could insert an indirect branch to one of its own write instructions, skipping the check in front of it.

To deal with the circumvention problem, Wahbe et al use a dedicated register that holds an address. When the compiler emits code, it maintains two invariants:

1. All writes must be performed to the address stored in the dedicated register;
2. The dedicated register should almost always point to a valid address inside the current module; if any instruction invalidates this invariant, the program must either fail or restore the invariant before the next write or indirect branch instruction.

Now, a module is free to jump to any instruction it wants within its own bounds, without risking writing to another module’s memory. This does imply that the dedicated register can’t be used for any other purpose, but on a RISC machine with 32 registers this is not a problem. The same trick can be used to protect indirect branches (jumps to data must be excluded; this can be done either by using a second dedicated register for code, or by marking all data non-executable).

This leaves open the question of how to allow calls between modules (called cross-fault-domain RPCs by Wahbe et al). The scheme implemented in this work stores a jump table inside each module, with one entry for each possible cross-module call. The trusted compiler permits this jump table (and only this jump table) to contain branches to points outside the module. This allows the checks on indirect jumps to be very simple (they just have to make sure modules only jump to their own code region). Rather than transferring control directly to another domain, these jump tables transfer control to call stubs which perform several important operations before invoking the actual call:

• Because each module needs to access its own data on the call stack, each module is given a private stack in its own data region. When transferring control to another module, we have to switch to its stack and copy across any stack-based arguments. A similar mechanism is used by commodity OSs when trapping in and out of the kernel.
• Standard calling conventions include callee-save registers, registers which must not be altered by the function being called. Since the called module may be malicious, the call stub saves these registers instead.
• The dedicated register invariants must be restored upon return.

Finally, we return to the problem of performance: inserting complex checks before every write and indirect jump is expensive, especially if those checks involve branches. The insight of Wahbe et al is that if the code and data region for each module is the set of all addresses with some common bit prefix (say, addresses 0x1f00000 through 0x1fffffff) then we can make sure that all writes and jumps go into this region by merely bitmasking the target address to have the correct prefix. If the original address lies outside the region, this will cause some random part of the module to get overwritten or jumped to – but only malicious or invalid code will encounter this behavior, so it’s not a problem (except, perhaps, during debugging).

Another important optimization involves the stack: writes to the stack are considerably more common than writes to the heap in typical programs, and are usually made at a small fixed offset from either the stack pointer or the frame pointer. Rather than check every one of these, they take advantage of the stack’s locality by maintaining the invariant that the stack pointer (or frame pointer) points inside the module’s data region, and only checking it when it’s modified. Since writes through the stack pointer may have a small offset attached, they create “guard regions” containing nothing useful before and after each module’s data region; even if the stack pointer is at the beginning or end of the region and the offset is set to the minimum or maximum value, it still can’t write past the guard regions.

To actually implement privilege separation with SFI, a little something extra is needed – if all modules in an application could make the same system calls, they would effectively have the same privilege. As described in section 3.4, Wahbe et al’s scheme uses a simple but flexible model in which one module is permitted to make system calls freely and no other module is permitted to make any; it acts as an arbitrator and can implement arbitrary fine-grained policies by observing which module invoked it.

Evaluations show that with all the optimizations described above, the overall scheme is quite efficient – runtime overhead ranged from 0 to 12%, with an average of 4.3%. However, this is only checking writes – checking reads, which is done in the MMU model and is important for protecting sensitive module-private data such as passwords, requires a much higher overhead due to the larger number of reads (21.8% on average).

One practical issue with Wahbe et al’s scheme as a security solution is that it depends on a large, trusted compiler for correctness – later works such as MIT’s PittSFIeld mitigated this problem by using a separate, smaller verifier that is run on machine code when loading it.

Practical impact and later work

Despite the apparent utility of Wahbe et al’s scheme, sometimes termed classical SFI, it is not widely used in practice. This may be because the scheme depends critically on a number of features of RISC machines, and RISC machines are not widely deployed in the PC market (they are, however, quite common in the mobile and embedded market). It may be because it was never effectively developed into a robust product or integrated well with other tools. More recent work, such as MIT’s PittSFIeld (2006) and Vx32 (2008), have effectively extended SFI to CISC platforms such as the x86 and put more effort into promulgating a robust prototype.

Although SFI is a hot research area now, I remain skeptical of approaches like Vx32 that depend on creative exploitation of legacy hardware that is in the process of being phased out. The way I see it, SFI presents a useful fault isolation model that is independent of its software-based implementation, and deserves explicit hardware support to mitigate its performance costs. Doing an efficient check for each read and write is exactly the sort of thing hardware is good at (and software is really bad at). The hard question is exactly what hardware support should be implemented to support typical applications – SFI is so poorly deployed, that few applications have ever been architected with fine-grained modularization and privilege separation in mind, and the array of possible design choices is overwhelming. Some proposed schemes include Mondriaan Memory Protection (2002), and the Hard Object (2009) project I worked on at UC Berkeley. Just as important is more work on programming tools to describe and implement module isolation and interfaces in a generic way that can be implemented using a variety of fault isolation schemes, ranging from no protection to dynamic address translation to SFI to new hardware schemes – this will help to bootstrap the process of getting large enough applications built that new platforms for isolation can evaluated. Regardless, this is going to be an important area of research in the near future and I’m excited to see what techniques will gain adoption.

The author releases all rights to all content herein and grants this work into the public domain, with the exception of works owned by others such as abstracts, quotations, and WordPress theme content.