Of Methodology and Men

By Kyleanne Hunter for Denver Dialogues

With headlines like these: “Marine Study Finds All-Male Infantry Units Outperformed Teams with Women,” “Marine Experiment Finds Women Get Injured More Frequently, Shoot Less Accurately than Men”, “Marine Corps Study says Units with Women Fall Short on Combat Skills,” it’s not surprising that emotionally charged statements are shaping the debate surrounding women in combat. Since then-Secretary of Defense Leon Panetta eliminated the ground combat exclusion rule for women in the military in 2013, the services have embarked on various studies and experiments aimed at determining the feasibility of integrating women into frontline ground combat units.

What is striking is that the two primary infantry services—the Army and the Marine Corps—went about their evaluation in very different ways. The difference in their methodologies resulted in very different conversations. Given that the results of these evaluations will have a direct impact on national security, it is essential that they be as unbiased and systematic as possible, and not based on passionate emotion.

The Army approached the question of integrating women into ground combat units by essentially creating a natural experiment. By using Ranger School, the premier school aimed at advanced training for “close combat and direct-fire battles,” as a testing ground for women’s capabilities, they utilized a long-established and transparent institution with set standards and expectations. Introducing women into Ranger School allowed Army leadership to evaluate women against a standard proven for decades to forge elite soldiers physically, mentally, and tactically. Women were judged against this standard, where “a 5-mile run is still a 5-mile run. … A 12-mile march is still a 12-mile march,” and succeeded against it. Knowing that women could perform against an preestablished and proven standard eliminated gender as a variable in combat effectiveness.

The graduation of women from Ranger School, while greatly celebrated by female soldiers and veterans across the globe, was largely void of passionate or emotional rhetoric. And while it has been acknowledged that few women will likely meet these standards in the future, the success of the first two has led the Chief of Staff of the Army to open the school to “all soldiers who are qualified and capable” of meeting the standard.

The Marine Corps, however, had no equivalent to Ranger School in which to test the feasibility of integrating women into ground combat units. Prior to Secretary Panetta’s announcement, the only requirement to be eligible for an infantry military occupational specialty was to be a male who has successfully completed boot camp or officer training. Without any established measurable physical or tactical benchmarks, the Marine Corps embarked on a year-long Force Integration Plan. This plan was intended to mimic an infantry unit’s pre-deployment training and subsequent combat deployment in order to evaluate the “physical performance and psychological characteristics” necessary for combat. The findings of this study were released in a four-page summary.

Unlike the Army’s pragmatic response to women being able to meet already established standards, the Marine Corps’ findings elicited great amounts of passionate discourse and speculation. This emotional response can largely be traced to faults in both the way the Marine Corps went about its study and presented the analysis.

Methodological Missteps

1. Unlike Initial Conditions. The Marine Corps’ first challenge was determining how to establish a performance baseline for women’s entry into the study. However, the disparity in initial training between male and female recruits (both physical and tactical) means that the initial conditions were not equal. Comparing women with less tactical training and lower physical expectations to men with more initial training is not a legitimate comparison.

2. Inconsistent Unit of Analysis. The long-form memo released about the study states that the its purpose was to “essentially deconstruct many collective ground combat arms tasks [and] identify what individual tasks and standards an individual Marine must achieve…to be a fully contributing member of that unit.” However, despite this focus on the individual Marine, women were judged as a group based on their average performance, while men were judged as individuals. This is not only bad science, but denying female Marines their individual agency is both counter to the original guidance by Secretary Panetta to set gender neutral standards for all specialties and reinforces the hyper-masculine culture that largely fuels the emotional response to this study. Note that buried deep in the Marine Corps memo it is acknowledged that individuals (men or women) of similar initial fitness levels performed at the same level and sustained similar rates of injury.

3. Changing Measurements. Prior to the start of the Corps’ Force Integration Plan, there was no preestablished standard for entry into combat arms specialties. Because of this, standards and measurements were adopted as the exercise was in progress. Without a predetermined set of measurements identified as the standard for success, this experiment is not objective nor can it be replicated.

Misleading Analysis

These methodological missteps are compounded by a non-transparent data analysis. Rather than allowing for outside observers during the evaluation or providing access to the raw data collected, the results were released first on September 10^th via a four-page summary, and later via a fourteen-page memo (leaked to the press on September 23^rd). The manner in which the results have been presented are largely responsible for emotional response.

1. Subjective ranking rather than objective analysis. The four-page summary states, “All-male squads, teams and crews demonstrated higher performance levels…compared to gender-integrated squads, teams and crews.” The inclusion of a term such as “higher” tells us nothing about whether an objective performance standard was met. Groups or individuals can be infinitely ranked against one another, and someone will always come out as “best.” However without the actual performance data, there is no way to determine what “higher” means. What words like “higher” and “better” do, however, is offer fodder for emotionally charged headlines.

2. Missing Information. Both the four-page summary and the fourteen-page memo allude to 134 “tasks” units were asked to perform. However, there is no record of what these tasks are, or their relationship to combat effectiveness. Reporting one group is “better” at a “task,” yet not mentioning what the task is, is a hollow statement, offering little insight on relative performances. Furthermore, there was no discussion as to the composition of the teams judged against one another. “Mixed gender” can relate to large range of group composition. Without knowing the actual breakdown of the teams, an objective judgment is difficult to render.

3. No Discussion of Military Effectiveness. At root, the purpose of Secretary Panetta’s lifting of the combat exclusion rule was to increase military effectiveness by harnessing the talents of the broader population. While the Marine Corps mentioned participants of the study being judged against the performance of specific “tasks,” there was no overall mention of combat effectiveness. Military effectiveness is more than just running fast or shooting straight.

4. Emotionally Charged Language. In the prelude to any discussion on the study, the author of the fourteen-page memo, Brigadier General George Smith, writes about women as a “self-selected population with much to prove.” This sort of bias, especially when devoid of any actual data, does not contribute to objectivity, but fuels hyperbole and misinformation.

The current popular and political debate surrounding women in combat integration highlights the importance of methodology, analytical rigor, and careful framing. It appears that the top decision makers recognize the flaws in the Marine Corps study and in the way the study has been presented to the public—but this hasn’t prevented great political theater from playing out and further confusing the issue. The men and women of our armed services, and the nation as a whole, deserve more and better evidence, not ill-formed methodologies and emotional drama.

Kyleanne Hunter is a PhD student and Research Fellow at the Sié Chéou-Kang Center for International Security and Diplomacy at the University of Denver, and former U.S. Marine Corps Cobra pilot.