How to Assess the Impact of Non-Lethal Weapons

Introduction

Non-lethal weapons (NLWs) represent a diverse set of systems whose common feature is that they are intended to incapacitate rather than kill or destroy. For example, they include laser dazzlers that cause targets to experience intense glare, the Active Denial System (ADS) that emits millimeter-wave energy to cause a temporary heating sensation, pepper balls that irritate eyes and airways, blunt-impact munitions such as rubber bullets and bean bags, and vessel-stopping technologies that entangle propellers. Generally, their effects are intended to be reversible. NLWs represent a subset of intermediate force capabilities (IFCs), which also encompass cyber, electronic warfare, and information operations. The term “IFC” is not doctrinal but is gaining traction in NATO circles. In this article, we focus on the NLW subset of IFCs.

At a time of increasing competition below the threshold of full-scale conflict, NLWs can play a role in addressing gray-zone operations: situations in which an adversary seeks to coercively change the situation without instigating a war.^[1] They can be used to demonstrate resolve and counter coercion without inflicting casualties in ways that could cause unwanted escalation. NLWs can also be valuable in other contexts, such as clarifying individuals’ intent in ambiguous situations or dispersing civilian crowds deliberately impeding military operations without causing permanent harm.

To inform decisions about how to acquire and employ NLWs, it is important to be able to measure their tactical, operational, and strategic impact. However, measuring the impact of NLWs requires a different methodology from more traditional approaches that do the same for lethal weapons. Lethal weapons are often assessed in terms of their ability to inflict a certain level of damage, whereas NLWs are valued for their ability to circumscribe it. Given this challenge, the U.S. Joint Intermediate Force Capabilities Office (JIFCO) asked a team from the RAND Corporation to conduct a study on how best to evaluate the impact of IFCs at multiple levels. In the remainder of this article, we describe that study, which we led, and the findings from it. While this analysis was centered on NLW usage within the U.S. Department of Defense (DoD), much of it can readily be applied in a NATO context.

Methodology

We began by reviewing over 150 documents and conducting 36 interviews with a variety of experts on NLWs. Based on this, we developed a structure called a “logic model” that linked the activities of NLWs with U.S. strategic goals via a series of intermediate steps. We refined the logic model based on expert feedback, then identified metrics that could be used to measure each item within the logic model. Next, we developed varied vignettes for NLW usage and evaluated the relative merits of the various metrics in the contexts of those vignettes. In addition, we further analyzed data from interviews and documents to identify broad themes and then developed a set of findings and recommendations regarding how best to evaluate and communicate the impact of NLWs.

Developing a Logic Model on the Impact of NLWs

Logic models can provide a structured way to relate specific processes or programs with high-level goals.^[2] The logic model that we developed to characterize NLWs described how the following five categories related to one another:

• Inputs – items that are required for NLWs to be used, such as the systems themselves, doctrine, and training
• Activities – what NLWs actually do
• Outputs – the direct results of NLW usage
• Outcomes – higher-level effects of NLW usage
• Strategic goals – ultimate goals of the DoD.

The logic model consists of a series of elements distributed across each of these five categories (see Figure 1).

The inputs, listed in the leftmost column, include the systems themselves, the tactics, techniques, and procedures (TTPs) and concepts of operation (CONOPs) for using them, as well as doctrine, training, and sustainment capabilities. They also include the laws of war (LOW) and rules of engagement (ROE) that shape how NLWs may be used. 

Activities, listed in the second column from the left, consist of things NLWs do. For example, these elements include hailing to communicate with other parties, disorienting them, impeding their mobility, or temporarily incapacitating them. Some NLWs can perform more than one activity at once, e.g., hailing can also help to reveal another party’s intent based on how that party responds.

Outputs represent the direct results of NLW employment. Examples of these direct results include increasing time for decisions, impacting costs to US and adversaries, and minimizing collateral damage. The outputs are listed in the center column in Figure 1. Outcomes, listed in the column second from the right, are another level up and relate more to higher-level impacts of NLWs, such as managing escalation, enhancing perceptions of U.S. forces, and managing relationships with partner nations. Finally, strategic goals, listed in the rightmost column in Figure 1, are wide-reaching goals established by DoD leadership – specifically pulled from the 2018 National Defense Strategy unclassified summary.[3] While NLWs cannot be entirely responsible for the achievement of these higher-level goals, their use can contribute towards their fulfillment.

Figure 1: NLW Logic Model (Source: Krista Romita Grocholski et al., How to Effectively Assess the Impact of Non-Lethal Weapons as Intermediate Force Capabilities, Research Report RR-A654-1 (Santa Monica, CA: RAND Corporation, 2022), https://doi.org/10.7249/RRA654-1).

Connectivity Between Logic Model Elements

After constructing the logic model, we mapped the connectivity between individual elements of the logic model, which helps illuminate the ways in which the use of NLWs can create higher-level impacts. This also highlights which portions of the logic model are the most important to examine when it is applied to specific scenarios or goals. Figure 2 shows the completed connectivity mapping for the NLW logic model. In the figure, strong connections are indicated by thicker, darker lines than weaker connections (e.g., strong connection = bold line). Arrow colors are for clarity – all arrows coming from a particular element are the same color. Elements surrounded by a dark blue box are linked to strategic goals via strong connections.

By considering the mapping as a whole, we were able to identify patterns and develop some key insights. The density of the linkages between the logic model elements decreases as we move from left to right in Figure 2. For example, most activities have strong connections to most outputs, but fewer outcomes have strong connections to multiple strategic goals). This holistic view also allows us to see which elements of the logic model contribute most to the strategic goals, both via direct connections and through a series of strong linkages. All seven of the activities, nine of the thirteen outputs, and five of nine outcomes have strong links to the strategic goals. In assessing the impact of NLWs on the fulfillment of DoD-wide strategic goals, the elements encased by blue boxes are the most important.

Identifying Metrics to Evaluate the Logic Model

Having developed the logic model, we used it as a basis for identifying metrics that could be used to evaluate the impact of NLWs. We identified 97 unique metrics that collectively measured all 29 elements at the activity, output, and outcome levels. Some of the identified metrics were applicable to more than one logic model element, so we used those metrics multiple times, giving us an effective set of 115 metric-element pairings. We did not develop metrics for the inputs because those metrics would not relate to the effects of NLWs. We also did not develop metrics for the strategic goals, whose assessment is determined at a DoD-wide level and goes far beyond the scope of our study. Examples of the metrics we identified for three of the logic model’s elements are shown in Table 1.

Overall, we found that:

• Activity metrics primarily related to which people or systems were affected by NLW usage and how they responded to it
• Output metrics generally related to providing the user with more time options, curtailing the adversary’s options, and reducing tactical risks
• Outcome metrics most often related to reducing strategic and operational risks, influencing perceptions, maintaining morale, and reducing costs.

Figure 2: NLW Logic Model with Connectivity Between Elements (Source: Grocholski et al., How to Effectively Assess the Impact of Non-Lethal Weapons as Intermediate Force Capabilities).

Table 1. Examples of Metrics Associated with a Subset of Elements of the Logic Model (Source: Grocholski et al., How to Effectively Assess the Impact of Non-Lethal Weapons as Intermediate Force Capabilities).

Developing Vignettes

To ground the logic model and the associated metrics in the real world and to evaluate our metrics in a range of scenarios, we created and examined a total of thirteen vignettes featuring the use of NLWs. The vignettes encompass a range of circumstances and conditions. We ensured that they collectively included all combinations of possibilities with respect to the following criteria:

• Whether the adversary sought to escalate the situation. This provides some insights into the extent to which NLWs may be de-escalatory in situations where an adversary deliberately seeks to escalate the situation. It also provides comparisons between the de-escalatory capabilities of NLWs in situations with both escalatory and non-escalatory situations.
• Whether withdrawal was feasible. U.S. withdrawal can contribute to de-escalation of a situation, so we gauged both situations in which withdrawal was not possible and those in which it was.
• Whether the narrative surrounding the incident was stable (i.e., whether disinformation could radically change the narrative). Given that NLW usage can play an important role in shaping narratives, and those narratives can shape their ultimate impact in turn, it was important to explore both cases in which narratives from incidents were highly malleable and those in which they were not.

We assessed the values of each of these using a binary (yes/no) distinction and ensured that the thirteen vignettes included all eight possible combinations. We also designed the thirteen vignettes so that they collectively included all of the U.S. military services, took place in a range of locations around the globe, and spanned the air, sea, and land domains. Where possible, vignettes were based partly upon past events to enhance their realism. For example, one vignette involved a U.S. aircraft being intercepted and harassed by two military aircraft, so it sought to use NLWs to get them to back away without causing crashes or escalating the situation. This was based on a real-life incident in 2000, in which two Chinese aircraft intercepted a larger, slower U.S. aircraft, resulting in an accidental collision. A vignette involving U.S. marines securing an embassy against a rioting mob also reflected actual events in Bahrain in 2002, with the proviso that in the vignette, the marines could employ a range of NLWs. Similarly, an incident in which boats with unknown intent approached a U.S. destroyer was loosely based on the suicide boat attack that damaged the USS Cole in 2000. Still, in the vignette, NLWs provided additional options to protect the ship.

Our analysis of these vignettes confirmed that advanced NLWs (particularly directed energy) could have a substantial impact in a range of situations beyond their typical applications generally associated with law enforcement and crowd control (such as pepper spray). For example, in a gray-zone maritime standoff, advanced NLWs could help to demonstrate resolve without escalating the situation.

The vignettes also revealed the relative versatility of different classes of NLWs. We found that three types of systems were particularly versatile, with applicability in a majority of the vignettes, across a variety of contexts and domains. Both acoustic systems and laser dazzlers could be used to hail, deceive, distract, disorient, or confuse individuals. In addition, ADS could provide focused, discriminating effects to tactically deter the other side, deny access, or compel movement. While these NLWs were especially versatile, a number of other NLWs also played important roles in specific vignettes. Having a panoply of NLWs available can ensure that the right ones are used for a particular situation. 

Evaluating Metrics in the Context of Vignettes

We explored the vignettes using our logic model and metrics. The first step in this analysis was to determine which NLWs were applicable to the vignette. We then determined which elements of the logic model were relevant to the vignette itself and evaluated the associated metrics in the context of the vignette. The qualities of each metric (not the value of the metric) were evaluated using four standard criteria [4]:

• Validity – how well the metric measures the element
• Reliability – the degree to which multiple measurements will be consistent
• Feasibility – how easily the measurement can be made
• Timeliness – how quickly a measurement can be made.

This evaluation showed that most metrics were strongly applicable to the logic model elements and relatively straightforward to measure; however, only about half of the developed metrics were applicable to any particular vignette.

Themes Identified in Interviews

As part of our analytical process, our team conducted 36 interviews with experts and stakeholders from 25 organizations. Four broad themes came out of our analysis of these interviews:

1. The two biggest barriers to NLW integration within DoD are cultural reticence and resource limitations. Potential NLW users often have limited experience with their usage, contributing to limited confidence in them. They also sometimes do not understand the effects of these systems and/or perceive them as less useful than lethal systems. Competing training requirements often result in NLW training being de-emphasized. 
2. NLWs are often seen as logistically burdensome in terms of space, power, and other requirements, so they are often not brought to locations where they could be useful.
3. Opportunities for NLW usage beyond military policing and crowd control are not widely perceived. The utility of NLWs in the competition below the threshold of war and many other contexts was not well-recognized.
4. The above challenges are interrelated and mutually reinforcing. For example, a lack of NLW usage due to training shortfalls and an aversion to supporting them logistically contributes to a lack of awareness and confidence regarding these systems, which lowers their priority in terms of both training and logistics. 

Recommendations and Closing Remarks

Based on the results of our study, we made a series of recommendations to the U.S. Joint Intermediate Force Capabilities Office and DoD that could be potentially applicable to NATO and individual nations. First, the logic model, or a similar NATO-focused variant, can be used in a range of forums, including in discussions with senior leaders, in order to illustrate how NLWs can impact strategic goals. Second, to evaluate the impact of NLWs, it is necessary to gather data that can be used to calculate values for the metrics. This could be done using real events, wargames, and live exercises. Metrics that are associated with logic model elements with strong links to strategic goals and that are easy to measure should be assessed first.

Additionally, our study found that NLWs are often perceived negatively, which inhibits their larger adoption and use. To address and overcome this, we recommend that those seeking to leverage NLWs establish consistent and clear policies, concepts of operations, standardized training, and protocols to integrate non-lethal capabilities into tactics, techniques, and procedures. Additionally, the logic model, metrics, vignettes, and technology demonstrations can be used to inform non-specialists about the utility of NLWs. Finally, future NLW capabilities should be designed to reduce perceived and actual burdens on operators. Specifically, in order to appeal to potential users, future NLW development should prioritize making NLWs that are easy to carry, easy to maintain, and easy to learn how to use, even at the expense of other design tradeoffs. Moreover, the advanced NLWs that we had identified as particularly versatile in our vignette analysis—notably acoustic systems, laser dazzlers, and the ADS—are capabilities that should also be prioritized for future development.

Disclaimer

The views expressed are solely those of the authors and do not represent official views of the PfP Consortium of Defense Academies and Security Studies Institutes, participating organizations, or the Consortium’s editors.

Acknowledgment

Connections: The Quarterly Journal, Vol. 21, 2022, is supported by the United States government.

About the Authors

Krista Romita Grocholski is a physical scientist at the RAND Corporation and serves as co-program manager for the NOAA-funded Mid-Atlantic Regional Integrated Sciences and Assessments (MARISA) center. Her research at RAND has covered a wide range of topics, including emerging technologies, force generation and readiness modeling, assessing commercial space capabilities, climate adaptation and resilience, and assessing the impact of non-lethal weapons. Prior to RAND, Romita Grocholski completed her doctoral and master’s degrees in observational astronomy at the University of Florida.

E-mail: kristarg@rand.org

Dr. Scott Savitz is a senior engineer at the RAND Corporation. Much of his research focuses on how to improve the effectiveness and resilience of operational forces through the use of new technologies and modified tactics. Recently, he has led analytical efforts to assess capability gaps in the European Arctic, the impact of non-lethal weapons, intelligence on threats to U.S. ports, and how the U.S., Japanese, and Brazilian militaries can invest in emerging technologies. Savitz has also led analyses on testing infrastructure requirements for autonomous systems, how to improve maritime domain awareness, how to measure the impact of efforts to counter hostile networks, and how the Coast Guard can prepare for future Arctic operations. He has previously led studies on how to effectively use unmanned surface vehicles, how to counter naval mine threats, and how the Coast Guard can make more informed asset allocation decisions. Previously, Savitz provided on-site analytical support for the Navy’s mine warfare command and the U.S. Naval Forces Central Command. He has led exercise observation teams around the globe and supported the Navy in Bahrain from 2001-2003, addressing political-military, counter-terrorism, and chemical/ biological/ radiological defense issues. Savitz earned his bachelor’s degree in chemical engineering from Yale University and master’s and Ph.D. degrees in the same field from the University of Pennsylvania.

E-mail: ssavitz@rand.org