Proceedings of a Workshop—in Brief

Integration Readiness Levels: Assessing Their Potential for Speeding Technology into the Soldier’s Hands

Proceedings of a Workshop—in Brief

On April 3–5, 2024, the National Academies of Sciences, Engineering, and Medicine’s Board on Army Research and Development (BOARD) convened a workshop to examine the potential use of integration readiness levels (IRLs) to speed technology transition in U.S. Army acquisition. The workshop planning committee consisted of experts in systems integration, systems engineering, military operations, and experts in specific technology areas relevant to Army systems and as detailed in the statement of task (see below). While the Army does not formally use IRLs in acquisitions, recent discussions across the national security community and the Department of Defense (DoD) prompted the Office of the Deputy Assistant Secretary of the Army for Research and Technology (DASA(R&T)) to request that the National Academies convene a workshop to explore the origin of IRLs, their current use (or lack thereof) in government and industry, and their potential to support Army acquisitions.

STATEMENT OF TASK

The workshop convened leaders from government, industry, and academia to explore the following:

1. Review the current Army approach to IRLs across the major technology areas articulated in the National Defense Strategy.
2. Discuss common definitions and metrics that could form the basis of a future IRL framework and how their applications could improve technology transition.
3. Identify what unique definitions and metrics for IRLs may be required for the emerging technologies of artificial intelligence (AI)/machine learning (ML)/autonomous weapons, hypersonic weapons, quantum sensing, and directed energy weapons.

The information summarized in this Proceedings of a Workshop—in Brief reflects the opinions of individual workshop participants and should not be viewed as a consensus of the workshop’s participants, BOARD, or the National Academies. The workshop was designed to facilitate a candid conversation among participants and to foster effective collaboration between academic and industry professionals with the Army’s science and technology (S&T) community of interest.

SPONSOR REMARKS

Matthew Willis, DASA(R&T), spoke about the criticality of integration in research, development, test, and evaluation. Currently, several major acquisition programs efforts are broken down into multiple component or subcomponent development projects. He stated, “I don’t necessarily know if there’s an appreciation across the

department for the level of complexity and certainly the criticality of integrating all of these different, varying, components into the final defense system that the government is going to acquire.” He wondered if IRLs can truly measure the variableness of systems integration.

Vice Admiral, U.S. Navy (ret.), Paul Gaffney, planning committee chair, asked if the Army plans on making any acquisition policy changes to accommodate new integration practices. Willis responded that they do not want to approach acquisition and integration differently from the other services, but they do want to become more informed about how to assess capability integration within their major acquisition programs. Jane Pinelis, planning committee member, asked if the Army is planning on using IRLs as a project management tool or as an assessment tool for the government. Willis responded that it would not be used for project management but instead as a holistic assessment of a capability’s transition potential into the Army’s broader portfolio. William “Bruno” Millonig, BOARD director, asked if there are any thoughts about moving the technological maturation initiative toward integration aspects of technology rather than maturity. Willis responded that a lot of what the program is doing is focused on technology integration within the Army S&T program. James Thomsen, planning committee member, asked if the Army expects IRL metrics and descriptions to be clear enough so that they can be entered into future contracting language with industry. Willis responded that while it is an interesting thought, currently, there are no plans to include it in any contracting language due to the Army’s own lack of understanding of IRLs.

ARMY INTEGRATION READINESS

Paul Terzulli, deputy director, Program Executive Office (PEO) Intelligence, Electronic Warfare and Sensors (IEW&S) Integration Directorate, U.S. Army, spoke about portfolio level integration. Information dominance, for example, integrates cyber, electronic warfare (EW), and other portfolios together. Millonig asked about the distinction between product and portfolio integration. Terzulli responded that there has been greater demand for cross-functional program integration. He added that to achieve specific capabilities, it is going to require the integration of multiple programs that by themselves will not be able to deliver what the Army requires. Terzulli stated that he sees broad applicability with both program and product integration, as IRLs can act as a common lexicon in both applications.

Terzulli spoke about the criticality of integration for the future Army division, which will leverage integrated capabilities to support decision making and data visualization for operators and commanders. He also commented that there are still many point solutions in different domains that need to be integrated and that the taxonomies and ontologies for each of those solutions are siloed. He concluded by speaking about PEO IEW&S’s success in leveraging modular open systems architectures as a means to facilitate technology integration across its portfolio, specifically for sensors.

Gaffney asked how PEO IEW&S defines system integration success when there is no formal IRL metric to measure that. Terzulli responded that ultimately, if the customer (i.e., a commander) is satisfied, that acts as a direct reflection on the performance of the product. Gary Polansky, planning committee member, commented that having good models helps better enable systems integration. He asked, in terms of integration readiness, if modeling and simulation or digital twins are being considered. Terzulli responded that the objective is to build a digital engineering ecosystem that programs can collaborate on, eventually building that out with digital twins. The challenge with digital twins is that in some cases, depending on the size of a program, it may not be justifiable to invest in a digital twin. Thomas Giallorenzi, planning committee member, asked if PEO IEW&S has seen examples of technologies being fielded and integrated whose maturity level was either under- or overestimated. Terzulli responded that he has seen examples of both, but it mainly depended on the complexity of the system that was being integrated. He added that IRLs could help more consistently quantify risk to a program or commander. William Osborne, planning committee member, asked if there has been any consideration of incentivizing industry to compromise on functionality for the greater good of integration by modifying contracting language. Terzulli responded that it has been a challenge because the commodity lines are siloed in different com-

modity areas and contracting strategies end up optimized against the different commodity areas.

PANEL DISCUSSION: PERSPECTIVE OF THE OFFICE OF THE SECRETARY OF DEFENSE AND THE DEFENSE ACQUISITION UNIVERSITY

The first panel of the workshop brought together speakers from DoD and the Defense Acquisition University (DAU) to discuss the policy and training and education perspective on IRLs. Robert Larino, director for policy and workforce, Office of the Executive Director for Systems Engineering and Architecture, Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E)), spoke about ways in which assessing technology readiness may be different from assessing integration readiness. He explained that with technology readiness and maturity, the focus is on new technologies and identifying the highest potential risk from a technology maturity perspective as the system develops. With integration readiness, the focus tends to be more broadly focused on all of a system’s components and figuring out whether they are ready to be fully integrated. Thomsen commented that technology readiness levels (TRLs) in DoD today are used as an “up-down vote” in deciding whether a formal acquisition decision goes forward. This decision has a lot of things, such as money and careers, tied to it that may create some intended and unintended behaviors. Larino responded that TRLs are in the law and are “kind of a gate,” but they aim to provide an honest insight on a technology’s development progress and try to mitigate risk early in the engineering process.

Larino concluded by commenting that OUSD(R&E) is working with its OUSD Acquisition and Sustainment counterparts on developing guidance on operational readiness levels (ORLs), which aim to be a broader readiness assessment framework that can measure technical maturity and integration readiness, while also capturing the technical risk that is attributable to a lack of testing.

David Pearson, center director for engineering and technology, DAU, spoke about DAU and explained that its role is to take policy and train the DoD workforce on how to properly implement it. When it comes to creating courses about new policy guidance, DAU engages policymakers early to gain insight and get a jump start on determining what learning products need to be produced. This also helps prepare DAU to deploy the policy onto its numerous platform vehicles and engage in rapid deployment training. During the policy implementation phase, DAU has faculty that can engage directly with project managers (PMs) to help them understand the policy and implement it in their workplace.

Robert Raygan, professor of systems engineering, DAU, spoke about how DAU engages with policymakers and creates learning modules to help inform students of new policy through use cases. Raygan also noted that a lack of early systems engineering rigor and discipline is the number one cause of programs stalling.

Raygan also noted that, within the realm of emerging technologies, technology transition happens within a program or life-cycle process and describes the flow of moving from one phase to another. Insertion can be into single or multiple programs, depending on where it is.

Thomsen asked if there is enough consistency among IRLs, systems modeling language (SysML), and other mission engineering methods and techniques that are already in place so that there is not any potential of confusing those who are doing the modeling work. Raygan responded that there could be consistency between them and that he would like the department to move toward machine readable templates rather than document-based ones, owing to the fact that machine-readable templates can be pulled into a database or model. Gaffney pointed out that in the TRL guide,¹ integration and aspects of IRLs are mentioned numerous times. He asked why IRLs or ORLs should even be implemented when guidance on TRLs can just be modified. Larino responded that the benefit to having different readiness metrics is that they can break down a complex system into smaller, more supportable metrics. He did caution about rolling too much into one metric as that one metric can take on more technical and programmatic risks. Pinelis noted that there are many dimensions to an IRL and that may make it difficult to characterize any specific technology with an IRL. Brooks commented that the use of systems readiness level (SRL) metrics may be a better way to use IRL instead of expanding on TRL.

__________________

GOVERNMENT ACCOUNTABILITY OFFICE ACQUISITION REPORTS AND IMPLEMENTATION OF TECHNOLOGY READINESS LEVELS

Mike Sullivan, former director, Contracting and National Security Acquisitions, Government Accountability Office (GAO), spoke about GAO’s reports that dealt with TRL implementation. He discussed three different reports² that tried to capture best practices when it came to technology, engineering, and manufacturing. One of the reports, “Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes,” explained how lower TRL technologies face a greater number of risks and unknowns. As a technology hits TRL 6 and 7, the risks and unknowns reduce to the point that most technologies go into the product development phase. TRL 8 and 9 is where production normally starts. The report suggested several recommendations, such as that the Office of the Secretary of Defense (OSD) should adopt a disciplined and knowledge-based approach of assessing technology maturity, such as TRLs, DoD-wide; require that technologies reach a high-technology level (TRL 7) before making commitment; and empower PMs to refuse to accept key technologies with low levels of readiness on individual programs.

Gaffney commented that a recent GAO report on technology readiness assessment guidance stated that many experts believe that the integration of critical technologies into such systems are already captured in the readiness levels of the TRL scale.³ He was troubled by the fact that IRL was only mentioned in a minute portion of the report and was hardly expanded on to explain how technologies can move from one level to another. He asked if it would be worthwhile for a PM to set aside money for IRLs. Sullivan responded that the PM probably should if they are going to use them to build a product.

INTEGRATION READINESS LEVEL ORIGINS AND DEVELOPMENT

Brian Sauser, professor and department chair of logistics and operations management, University of North Texas, spoke about how one of his motivations for creating IRLs was to establish a common language so that the integration of two technologies of varying TRLs can be compared.

He also spoke about three challenges that continue to persist in systems development: (1) life cycles are different and changing, which poses a challenge when you try to integrate them together; (2) the ability to understand, monitor, measure, and govern a system’s life cycle is not keeping pace with the rate of change; and (3) understanding how systems integrate to provide sufficient solutions to capabilities in an agile environment is not sufficient. He stated that “we must settle for good enough, no perfect.”

Sauser spoke about the difference between hard and soft metrics. Hard metrics are measured objectively through data analysis. They are generally more difficult to derive owing to the need for data. Soft metrics are measured through a subjective judgement. They are relatively easy to derive but require a complementing rational that explains the assessment. He added that readiness levels tend to be soft metrics.

Sauser spoke about the history of IRLs and how they were originally based on the Open Systems Interconnection 7 Layer Model. According to Sauser, the three things that are required to have integration are interfaces, interactions, and compatibility. Interfaces are the connections that allow communications or relationships between two components. Interactions are the ability of the components to influence each other through their interfaces. Compatibility are the semantics of the components to orderly and efficiently integrate and operate with other elements in a system with no modification or conversion required. Figure 1 illustrates how these three concepts build the foundation for early integration work during a system’s development life cycle.

Sauser also spoke about the importance of artifacts and how they can help move readiness levels from a subjective soft metric to an objective hard metric. Planning committee member John Luginsland, Michigan State University, asked if SysML is the right framework to start creating artifacts. Sauser responded that it is one way to do so. He added that he and his team took different systems engineering architecting tools, such as Rhapsody, and used plugin modules to tie the artifacts that were

__________________

developed through those modeling tools to readiness levels.

Gaffney asked if the artifacts would be different for different technology areas. Sauser responded that artifacts will vary across different technology areas because the way they are engineered is very different from one another. Pinelis asked if any thought had been given to the fact that IRLs for a system might change during sustainment or while it is in operation, as systems today are constantly updated. Sauser responded that consideration of that was given and that measuring integration and system readiness, along with the system that is being developed, will need to be as dynamic as the environments they are going into. Giallorenzi asked about the biggest challenge that comes with readiness levels. Sauser responded that the biggest challenge is putting artifacts against the levels and communicating that down to the engineers doing the work. Gaffney asked about the cost of implementing IRLs and finding artifacts to support IRLs. Sauser responded that if the assessment is tied to artifacts, then the cost should be minimal because the work is already being done and artifacts are naturally being generated.

INTEGRATION READINESS AT NASA

Steven Hirshorn, chief engineer, Aeronautics Research Mission Directorate, NASA, introduced the origin of TRLs, stating that the concept and intent originated from the mid-1970s and was eventually infused into NASA’s institutional culture because of John Mankin’s “Technology Readiness: A White Paper”⁴ that characterized TRLs in a more instantiated form. The paper was principally intended to help decision makers characterize project risks using TRLs, particularly during the early phases of a system’s life cycle. Hirshorn added that decision makers should not just look at the TRL number as a gate or a “go, no-go” decision-making tool, but as a way to characterize risk.

Hirshorn introduced the advancement degree of difficulty (AD²) concept and how TRLs are unidimensional. They can tell you the maturity of a particular technology but say nothing about how easy or difficult it is to mature that technology. AD² adds that dimension to the conversation. Hirshorn added that TRLs and AD² go hand in hand when discussing risk. Gaffney asked if NASA uses

__________________

TRLs for procurement of specific technologies or for all of them. Hirshorn responded that as a mechanism that is used to understand risk, it is used uniformly across NASA but not mandated. He noted that NASA’s Science Mission Directorate instantiates TRLs in its requests for proposals for critical technologies.

Hirshorn spoke about the different readiness levels that exist today. He noted that there are 41 different readiness and maturity levels that are being used today, but that number could be much higher, “probably 10 times the number of that out there.” Hirshorn stated that of all the readiness and maturity levels, TRL is the only one at NASA that exists in policy. He added that IRL, while not part of NASA’s institutional culture or policy, is used in communities around the agency.

Hirshorn explained that NASA measures integration readiness through interface control documents (ICDs) and rigorous testing. At the systems level or lower, ICDs help verify which requirements can be marked closed and identify the amount of effort over time remaining on any open verifications. This tool helps inform systems integration reviews (SIRs) that are built into the systems engineering life cycle as milestones. He noted that SIRs can be used in different parts of the life cycle. NASA Science, for example, uses them at the beginning of the integration process, while the Human Spaceflight team uses SIRs at the end of the integration process. Hirshorn added that integrated performance testing goes hand in hand with ICDs, because with NASA missions, systems and components need to work perfectly or else the mission is lost. As such, there is a large emphasis on using integrated performance testing to help close ICD requirements.

Hirshorn concluded by speaking about how NASA Aeronautics does not develop any operational capabilities and that the things they do develop normally reach TRL 5 or 6 before they are handed off to industry to further mature and certify. This helps to lower risk for industry partners. Hirshorn lamented that a lot of research and development (R&D) work in NASA Aeronautics is focused on improving performance and little on understanding risk and system hazards. He explained that focusing 5 percent of the budget on understanding—for example, new hazards that the technology presents to a system—could be very beneficial for industry.

Millonig asked if there is an equation that makes an IRL or if an IRL contributes to an equation, as, for example, SRL = TRL + IRL. Hirshorn responded that he did not find having an equation practical or useful and that IRL is more of a component to assess risk. Pinelis added that she does not think there will be a math equation because the weights of the inputs will vary between missions and trying to have a one-size-fits-all approach equation could be dangerous. Gaffney asked who makes integration and readiness level decisions at NASA, because there is no trained acquisition workforce. Hirshorn responded that decisions are made either by NASA headquarters or they are left at the agency level. Where the decision happens depends on the complexity, cost, and risk of the project.

DAY 1 CLOSING DISCUSSION

Several planning committee members commented that IRLs have real value in acting as a common lexicon for systems integration. Planning committee member Peter D. Schwindt, Sandia National Laboratories, and others felt that the definitions surrounding IRLs should be more clearly defined and not overlap with TRLs. They noted that not providing additional clarity would only create confusion for program offices.

Some planning committee members were still unsure about IRLs, as they felt that additional scoping and work needed to be done before they were formally implemented. Thomsen and Giallorenzi wondered about the additional workload cost that IRLs would put on a program, because there are already various tools available to PMs that measure integration, such as ICDs. Other planning committee members noted that the workload cost would introduce scheduling delays and slow down the development process.

Schwindt and Polansky highlighted the importance of the AD² concept and noted that it adds significant value to IRL, TRL, and manufacturing readiness level (MRL) discussions and can help in characterizing program risk. Several planning committee members viewed IRLs as more of a program management tool than an oversight tool for the government. Pinelis pointed out that IRLs

seem to be divorced from the mission unless it is being done at the program level.

A couple of members of the planning committee highlighted the importance of industry and how they play a leading role in integrating systems components for delivered products. As such, Gaffney and Thomsen suggested that industry be involved in policy discussions in a larger capacity. Gaffney also commented that it is not clear where industry starts evaluating IRL before it hands something over to DoD or how the department can hold industry to any kind of integration standard.

DAY 2 OPENING AND INDUSTRY PERSPECTIVES PANEL

Osborne introduced Kory Matthews, retired vice president of program management, The Boeing Company, and Michael Drews, vice president, NC3 Advanced Capabilities and Mission Strategy, Lockheed Martin Space. Drews opened the panel by speaking about how he views readiness levels as a scheduling and maturation risk and believes IRL’s value is in helping to measure overall system readiness. He also noted that IRL could provide some best practices to help drive out early mistakes in the development process. Mathews suggested that readiness levels be used as a part of an overall suite of best practices while also building a culture that embraces the rigor that the metrics afford.

Osborne asked, between their respective companies, who is responsible for integration work. Drews responded that within Lockheed Martin, larger programs have a systems engineering integration and test (SEI&T) team that takes the lead on integration. Matthews responded that within Boeing, integration guidance was typically laid out in the systems engineering plan. Similar to Lockheed Martin, Boeing also has a SEI&T team that would lay out the process for integration.

Osborne asked, from a budgetary perspective, how much of a program’s budget would be allocated to integration activities. Matthews responded that with a SEI&T and program management office working together on integration activities, it would cost between 10–15 percent of a total budget. Drews responded that SEI&T activities were around 25–30 percent of a total budget, with labor being about 10 percent.

Osborne asked what kind of artifacts are being generated from their company’s IRL activities. Matthews responded that throughout the IRL process, they have generated ICDs, validation tests, test reports and findings, and more to use as artifacts. Drews responded that Lockheed Martin is not contractually obligated to report IRL metrics. Because of that, the IRL process tends to be more informal. He cited ICDs and early interface testing as key artifacts in the IRL process.

Osborne asked how the two companies merged the TRL and IRL process together to ensure optimization. Drews responded that successful programs often look at the program schedule and purposefully insert risk mitigation opportunities. He noted that bringing components together early, even if they are low TRL, and conducting interface and environment testing is one way to mitigate integration risk in the schedule. Matthews agreed and commented that MRL, TRL, and IRL should be fully integrated into a program’s risk issue and opportunity management approach if the program is utilizing those tools.

Osborne asked why Lockheed Martin and Boeing are implementing IRL and if the programs using it are deriving good value from implementing IRL practices. Drews responded that they do not view IRL as metric in the same way they do with TRL but are focused on integration readiness as a principal design risk. Pinelis asked the panel if IRL is more beneficial as a program management tool or government oversight tool. Matthews responded that it is a beneficial program management tool, but only if it is not a crutch for the program. Planning committee member Laura J. McGill, Sandia National Laboratories, asked about how to verify against the ICDs, as they do not capture everything. Matthews and Drews both agreed that early testing with mature interfaces is helpful. Pinelis asked if IRLs were not required from a government oversight perspective, would people still allocate time and money to test against the artifacts. Drews responded that you can incentivize contractors to bring together integration readiness risks by establishing program milestones to show certain technologies. Matthews also commented that Boeing’s use of IRLs has helped the team identify and solve potential problems on a product’s key interfaces early in the development cycle.

HYPERSONICS TECHNOLOGY PANEL

This panel brought together experts on hypersonics systems to discuss the challenges of assessing integration readiness and the potential role of IRLs to support the development and acquisition of said systems. Polansky introduced David Van Wie, head, Air and Missile Defense Sector, Johns Hopkins University Applied Physics Laboratory, and Grant Chandler, chief engineer, Army Hypersonic Project Office, Rapid Capabilities and Critical Technologies Office (RCCTO). Van Wie opened the panel by speaking about how hypersonics is unique in that ground testing facilities can never fully replicate the conditions of flight testing, which creates a challenge for the integration testing of components. However, the gap can be filled through rigorous developmental testing of components, through the use of wind tunnels, and with digital engineering. Chandler spoke about how integration is the most challenging phase of a development program and that IRLs have the potential to be a consistent rubric. He cautioned that widespread adoption of the metric would be a daunting challenge. For IRL to be effective across the Army, according to Chandler, it needs to be better standardized, PMs need to be better trained to utilize it, and it should be adapted only as a value add to a program.

Polansky asked if IRLs, as they are currently defined, need to be tailored in any specific way for hypersonics. Chandler responded that the operational environment is much different for hypersonic missiles, and this requires extra interface considerations and a lot of specialty engineering to integrate the high-tech components of the system. Van Wie added that the challenge with IRL is that it does not speak broadly to the robustness of what is being integrated together and the potential uncertainties of the environment. Polansky asked if IRL should reflect simulation capability maturity in its assessment. Van Wie responded that there is close coupling between the testing that is done in the physical world and the integrated modeling and simulation environment that allows for the propagation and examining of things over a much broader range of conditions. Polansky also asked about the value of readiness levels and where they should and should not be used. Van Wie responded that there is some utility in them for assessing overall progress, but they should not be used as a substitute for solid systems engineering fundamentals that are necessary to build complex systems. He added that there are many ways to capture risk other than readiness levels and to be wary of readiness levels becoming gates to progress.

Gaffney asked if there are any concerns about having an independent oversight entity between the government and a customer. Chandler responded that having independent oversight is the cost of doing business. Where he gets concerned is if, from a PM standpoint, that person is putting down obstacles that impede the program from reaching certain milestones. Ted Bowlds (Lt. Gen. USAF Ret.), planning committee member, asked how test infrastructure plays a role in system development and is represented in integration readiness. Chandler responded that ground testing, in particular, is very important in being able to move a technology up the TRL and IRL scale. Without it, “you go from paper to flying, and that has a lot of risk without a whole lot of component test and subsystem tests in between.” Van Wie added that testing the technology is often as important as the technology itself. Pinelis asked about the importance of incorporating adversarial information into integration testing. Van Wie responded that it is a classic problem that the requirements of major acquisition programs are set against a threat environment. As mentioned earlier, that environment is always changing. Van Wie added that work is being done to try and close that gap.

QUANTUM TECHNOLOGY PANEL

This panel brought together experts on quantum technologies to discuss the challenges of assessing integration readiness and the potential role of IRLs to support the development and acquisition of systems based on or integrating with said technologies. Schwindt introduced Richard Overstreet, chief scientist, Atomic Clocks, Microchip, Inc., and Allen Farrington, demonstrations manager, Office of Technology, Infusion, and Strategy, Jet Propulsion Laboratory (JPL). He then spoke about the different applications of quantum in devices such as atomic clocks, inertial sensors, and more.

Regarding TRLs, Overstreet stated that his company only pays attention to TRLs when the contract requires it. Ultimately, the decision on readiness is decided by Overstreet, who consults with his team of physicists to determine whether the clock technology is ready to deploy. Regarding atomic clocks overall, they deal with

a unique set of components that operate at the limit of their specifications. Because of that, component availability is very low. Overstreet posed the question of how to judge the IRL or TRL of a component that is unique and limited in its availability. There is also a business perspective to IRLs and TRLs. Overstreet pointed out that if they create a clock that is of high TRL and IRL but nobody comes to buy it for 5 years, the IRL and TRL will eventually become zero.

Farrington spoke about how JPL, as part of NASA, suffers from many of the same issues that Hirshorn spoke about on day one of the workshop. However, as a federally funded research and development center, JPL is offered some “selective flexibility” with respect to institutional practices. He also spoke about how TRLs were first institutionalized and interpreted differently by different people within the organization. Over time, JPL stopped using the numbers for TRL and used explanations to explain where a technology was at. Farrington explained that they eventually saw what DoD was doing with TRLs and began adding tools and rubrics to help flesh out the concept. Today, they still do not use the formal numbers, but use TRL as a rubric to talk about a technology during the review process. Regarding IRLs, Farrington spoke about how the concepts of IRL are, by accident, already reflected in JPL’s review documentation. The introduction of IRL into JPL could, by Farrington’s estimate, be an enhancement for, but not replace the current review process.

Schwindt asked Farrington about his opinions on the importance of the AD² concept and about IRLs overall. Farrington responded that JPL does not use that either but relies on its review process for teasing out predictions on the future, which itself can be hit or miss. Regarding IRL, he liked the concept, but it is of little value if an organization does not go through the process of developing tools and rubrics for the engineers to use. Overstreet responded that there is some value in IRL for the program funders so that they understand what is going on with a system, but for the engineer, “it is time spent not getting stuff done…. Nothing gets done because a piece of paperwork got filed.” He continued that it is important for the government, or other oversight body, to know the state of the technology, but that implementation of IRL will come with a time cost. Farrington agreed and stated that he and his team view it as a tax on getting work done in the laboratory.

Gaffney commented that when it comes to integrating lasers into atomic clocks, some sort of integration activity took place, but with Overstreet’s team they did it without the formal IRL process. Gaffney also commented that it could be good practice for developers to communicate with the customer on what they have done to ensure integration, even if they have not been asked to do so. Schwindt asked if JPL would have any interest in implementing the formal IRL list. Farrington responded that there would be several bureaucratic hurdles if they tried to implement it. Giallorenzi asked if the panelists’ viewed their organizations’ review processes as good enough and if they considered the implementation of IRL as a net negative. Overstreet responded that he views it as not helpful, as it comes at a cost to the contract schedule, but there is some value to it from an oversight perspective. Farrington added that with the implementation of TRL, the vocation of TRL status required by a certain gate in the review process limits the innovation and new technology that is put into missions. He hoped that IRL would avoid causing the same issues with its implementation.

DIRECTED ENERGY TECHNOLOGY PANEL

This panel brought together experts on directed energy systems to discuss the challenges of assessing integration readiness and the potential role of IRLs to support the development and acquisition of systems based on or integrating with these technologies. Luginsland introduced Thomas Karr, chief scientist for sensing and directed energy, MITRE, and James Trebes, former principal director for directed energy, OUSD(R&E). Karr opened by introducing directed energy weapons (DEWs), stating that they are weapons that utilize beams of energy to destroy, damage, or disrupt a target.

Trebes spoke about the challenges of integrating DEWs. During R&D, the technology only needs to integrate with testing instrumentation and meet specific power requirements. During the testing phase, it still needs to meet power requirements and integrate with not only test instrumentation but also with test range communications, safety systems, and fire control systems. Trebes noted that the system is also subject to different environmental and weather conditions. DEWs should also

integrate with fire control systems, local kill webs, and have established operating tactics. He also spoke about the use of DEWs to intercept hypersonic, ballistic, and cruise missiles. Trebes and Karr both stated that overall, there has been very little consideration given to the integration of kinetic energy weapons and EW on the same platform and across different platforms designed to work alongside each other.

Luginsland asked if IRLs would have helped the Army with its integration of DEWs. Trebes responded that the Army did not use IRLs but implemented parallel processing, where they developed policy and tactics while simultaneously developing the laser. Trebes noted that the implementation of IRLs could have value as they may force people to think about how they go about integration. Luginsland commented that often soldiers do not know what to do with DEWs and that getting them hands on with the system can help build comfort and trust. He asked if this issue is exclusive to DEWs and if IRLs could help in that process. Trebes responded that warfighter confidence with DEWs is low, and he suggested that operational demonstrations are the only way to address that issue.

Gaffney asked if there have been wargames conducted to test the viability of DEWs. Trebes responded that they have occurred, but they were either too simplistic, or else the physics, use of the weapon, or engineering were wrong. Karr noted that the environment into which these weapons are deployed requires integration across a multi-service, multi-domain environment. Gaffney also asked about what the Army does differently with DEWs to get to a certain integration level without using the IRL checklist. Chandler responded that when RCCTO is developing systems the account of doctrine, organization, training, materiel, leadership and education, personnel, and facilities are considerations alongside the material development, which quickens overall development.

ARTIFICIAL INTELLIGENCE/MACHINE LEARNING TECHNOLOGY PANEL

This panel brought together experts on AI to discuss the challenges of assessing integration readiness and the potential role of IRLs to support the development and acquisition of these systems. Pinelis introduced Capt. M. Xavier Lugo, Task Force Lima Commander, OSD Chief Digital and Artificial Intelligence Office, and Scott Fisher, vice president, national security and intelligence, ECS Technologies. Pinelis asked Fisher if, as the lead integrator for Project Maven, he used IRLs to guide integration efforts. Fisher responded that in the beginning, he and his team had no concept of IRLs. Over time, his team developed their own system to identify key integration points across particular lines of effort. In addition, he noted that formal IRL requirements would make things difficult from a project timeline perspective, but that the idea of forming a common lexicon on integration is important. Similarly, Pinelis asked Lugo if he views IRLs as helpful. Lugo responded that he believes it helps from a thought process perspective and not a requirements perspective. He explained that IRL could help someone think through what interoperability and integration really means for a system during development, but making IRL a requirement adds another layer of bureaucracy that delays the development process. He added that the testing and evaluation (T&E) and validation process, particularly from a system of systems and autonomy perspective, should already be identifying and capturing integration issues. He questioned whether IRL would really be needed for that.

Pinelis asked Fisher about the keys to integration on Project Maven. Fisher responded that establishing an infrastructure to do integration testing is important. He also highlighted integration events, where both the government and contractors collaborate on different integration issues with their system. Additionally, he commented that establishing common protocols is important. Pinelis commented that one thing about the integration events that is not reflected in the IRLs is the importance on integration of not just technology but also people.

Pinelis asked the panelists about the major challenges in assessing AI/ML systems for integration readiness. Lugo responded that the defining the thresholds for progressing a system forward, such as defining integration success and what “good enough” looks like, is a challenge. Fisher responded that it would be helpful to have an integration metric that measured integration with not only your own system and code, but someone else’s as well. Pinelis also asked the panelists how applicable the current IRLs are to AI-enabled systems. Fisher responded that the generic IRLs do not help at all. What would help, in his opinion, would be to have a set of IRLs that are associated with particular lines of effort in AI. Lugo responded that specificity on what the scale is trying to

solve would help. He explained that current processes and checks and balances would probably catch system of systems integration issues, which makes the role and intent of IRLs uncertain.

The panelists concluded by discussing their final thoughts on IRLs. Fisher reiterated that IRL’s use as a guideline and lexicon is its real value. He stated that it would be disruptive to try and use it as a government requirement in a rapid R&D process. He noted that there are already government T&E processes that determine whether the integration works. Lugo agreed and was strongly against using it as an oversight tool that adds more bureaucracy and hurdles during system development. He added that IRL’s ability to help evaluate “apples to apples” on technology is valuable. Thomsen commented that it would be helpful for the department to find a way to better communicate and inform themselves and the vendor, in contract, about the level of integration that they are requiring in a system. Planning committee member Laura M. Haas, University of Massachusetts Amherst, asked about the cost to the program for adding specificity to the IRLs. Fisher responded that the real time cost comes from getting the government, the integrator, and the vendors all on the same page, from a standards perspective, which can take weeks. Lugo agreed that establishing standards for everyone is important. He added that, as an example, “an 8 in hardware doesn’t mean the same in AI.”

DAY 2 CLOSING DISCUSSION

Several planning committee members shared that IRLs, while a decent guide to help PMs think about integration, should not be a replacement for quality systems engineering fundamentals. McGill added that IRLs should also not be a constraint on the speed in which integration occurs, which is a risk in today’s rapid prototyping environment. Gaffney noted that a number of organizations had their own internal processes for measuring integration readiness that were built off the basic principles of IRL but were not using the formal checklist. Pinelis and others felt that IRLs did not address the integration of people, training, tactics, and more, which are important considerations for the success of a system’s integration.

A few planning committee members highlighted the 10–15 percent of labor costs that should be put toward integration activities. Bowlds cautioned that when programs begin running into technology and budget problems, they begin to take money from different areas of the budget. He added that if money is taken away from that 10–15 percent labor cost, the risk for the program begins to increase.

SUMMARY OF RECURRING THEMES FROM THE WORKSHOP

The following section highlights a few recurring themes and thoughts from the prior 2 days, as discussed by members of the workshop planning committee. Several workshop participants noted that while IRLs manage to characterize good systems engineering, they are not a replacement for good systems engineering practices and early testing. Others noted that when targeted precisely and not used as evaluation criteria, they could highlight the commander’s intent and improve overall program efficiency. Additionally, several members felt that IRLs could be a useful guide to instill a culture of integration but should not inhibit the speed of a program.

Several workshop participants, including Gaffney and Polansky, noted that implementing IRLs would work against the principle of speed and introduce unintended barriers to the agile development process. Some workshop participants commented that implementing IRL could take 5 years or more, which will add time and cost to the acquisition process for program offices. Polansky commented that if speed is the overarching goal for the Army, then it may be better to look for an alternative approach that allows for agility and rapid technological advancement.

Several members pointed out that the definition and language surrounding IRLs can be quite vague, which creates several problems. A few members commented that IRLs tend to mainly focus on technology integration, but do not lend themselves well to assessing capability integration. Other members commented that across different technical areas, IRLs must be scaled and adjusted to fit each one owing to the different needs, environments, and components that affect the technologies. A few planning committee members pointed out that research on IRLs is more than 20 years old and suggested there are gaps in the research that should be closed if there is interest in adopting IRLs institutionally.

Some planning committee members wrestled with the idea of using IRLs as either a program management tool or an oversight tool for the government. Several

members cautioned against the use of IRLs as an oversight tool or acquisition requirement, stating that IRLs could become a gate for programs trying to reach certain milestones in the development process. Some members also stated that it will delay programs and increase costs over time. Several members did see utility in IRLs being used as a program management tool by PMs to help assess integration along the system’s development cycle, act as a common lexicon for everyone to talk about systems integration, and identify and burn off risk early in the development process. Additionally, some members believed that IRLs could be useful in characterizing and progressing advanced technologies that are in the middle TRL and IRLs. Several planning committee members cautioned that IRLs are insufficient if program offices do not plan for the necessary testing resources and that IRLs would inhibit development and acquisition if made mandatory but would be best viewed as part of a suite of best practices for systems engineering.

PLANNING COMMITTEE MEMBERS VADM Paul Gaffney II (NAE) (Chair), University of South Carolina; Ted F. Bowlds, IAI North America; Nathan R. Brooks, The Boeing Company; Thomas R. Giallorenzi, L3 Harris Corporation; Laura M. Haas, University of Massachusetts Amherst; John W. Luginsland, Michigan State University; Laura. J. McGill, Sandia National Laboratories; William H. Osborne, Independent Consultant, Boeing Defense and Space (retired); Jane K. Pinelis, Johns Hopkins University Applied Physics Laboratory; Peter D. Schwindt, Sandia National Laboratories; James E. Thomsen, Seaborne Defense, LLC

STAFF William “Bruno” Millonig, Director, BOARD; Steven Darbes, Program Officer, BOARD; Ryan Murphy, Program Officer, Air Force Studies Board (AFSB); Sarah Juckett, Senior Program Officer, BOARD; Debrah Adedeji, Research Associate, BOARD; Evan Elwell, Research Associate, AFSB; Tina Latimer, Program Coordinator; BOARD, Travon James, Senior Program Assistant, BOARD; Emily Stower, Senior Program Assistant, AFSB

DISCLAIMER This Proceedings of a Workshop—in Brief was prepared by Evan Elwell as a factual summary of what occurred at the workshop. The statements made are those of the rapporteur or individual workshop participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

REVIEWERS To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Lida Beninson, National Academies of Sciences, Engineering, and Medicine; Laura Haas, University of Massachusetts Amherst; and Michael Sullivan, Government Accountability Office (retired). Katiria Ortiz, National Academies of Sciences, Engineering, and Medicine, served as the review coordinator.

SPONSORS This Proceedings of a Workshop—in Brief is based on work supported by the Office of Deputy Assistant Secretary of the Army for Research and Technology of the Army Research Office under Contract No. W911NF-23-D-0002. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the Army Research Office.

SUGGESTED CITATION National Academies of Sciences, Engineering, and Medicine. 2024. Integration Readiness Levels: Assessing Their Potential for Speeding Technology into the Soldier’s Hands: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/27790.

Division on Engineering and Physical Sciences Copyright 2024 by the National Academy of Sciences. All rights reserved.