Part 2: Predicting the Efforts to Progress on the TRL Scale
By Svetlana Klessova and Sebastian Engell
In part 1 of this contribution [1], we presented the concept of the Technology Readiness Levels (TRL) and introduced a refined definition of the TRL scale based on an extensive analysis of 38 development processes in which novel technical solutions (new devices, methods, tools, algorithms etc.) were developed starting from different initial levels to different final levels of maturity. An important characteristic of our refined definition of TRL is that the development may follow different paths with respect to the maturity level at which new components are integrated into the overall system and how much testing is performed at which state of maturity.
In order to predict the required efforts to progress on the TRL ladder, we derived “scores” for the relative efforts to progress from one TRL level to the next, normalized to a score of 1 for the progression from TRL 3 to 4. These scores are given in Table 1 below. The absolute efforts (average time and resources needed for a score of 1) are technology specific and must be adapted to the specific domain. As can be seen from Table 1, the scores are increasing when climbing up the ladder. As mentioned in part 1, the targets levels at TRL 8 are 8.2, 8.3 and 8.4.
Table 1: Scores for the progression along the refined TRL scale [1].
Figure 1 graphically represents these scores and shows alternative paths of maturation with the associated scores ranging from an initial to a final TRL. The scores must be added up along the development path and determine the overall relative effort, with 1 being the score to go from TRL 3 to TRL 4. If several new components are developed, the addition must be done for all branches in the resulting tree. The scores in Figure 1 are average scores for information technology based technical solutions in industrial environments; for other domains the increase of the effort can be higher or lower, e.g. for physical technologies such as sensing technology the increments are expected to be higher, for purely digital technologies lower. This also depends on the contextual conditions surrounding the development process [3].
Figure 1: Relative increment of the time, efforts and resources needed to progress from one TRL-sublevel to the next for incremental ICT-based innovations in an industrial environment (Source: [2], © 2022 The Authors, Reproduced with permission from Elsevier Ltd.)
We validated the scores using the available quantitative data for 38 development processes of different ICT-based innovations in industrial systems. The definition of the scores implies that the required effort should increase linearly with the sum of the scores for different development processes with different initial and final TRL This can be clearly seen in Figure 2 where the cumulated human resource efforts (person months) are shown over the increase in the scores for several novel technical solutions that were developed in two different projects.
Figure 2: Total scores for the maturation of the technologies versus absolute efforts invested for these technologies. The different technologies are represented by dots. The trendline is shown per project. (Source: [2], © 2022 The Authors, Reproduced with permission from Elsevier Ltd.)
The slopes of the straight lines indicate the difficulty of the developments or the efficiency of the projects. Different development projects in a similar domain can be compared by the effort invested divided by the increase of the scores that was achieved. Development processes in the same domain with different efficiencies need different investments to reach the same increment of the scores. A reason that the effort is larger than expected from the increase in the scores is that the validation in the lab missed important aspects of the real environment so that unexpected efforts were needed to cope with them. Also the integration in the operational environment may cause large efforts because of technical or organizational issues. So the differences can be due to the technical difficulty of the development or to organizational problems, but may also be caused by problems on the personal level. Unexpected increases in the efforts needed call for management to take a closer look.
The proposed refinement of the TRL scale and the scores enable technology leaders to assess the progress of the developments, and thus the value for money for the company or institution. The efficiency of a development process measured by the scores does not depend on the initial and targeted TRLs, as the scores account for the increase of the efforts at the higher stages. The start and end points of the developments, the durations of the development, and the budget are usually available as objective data. The scores also help to indicate how much remaining effort can be expected after a certain maturity level has been reached. This can then be compared to the expected business value and revisions of the planning can be made early if needed. The curves depicting the relationship between effort and score can serve as an early warning indicator, aiding in the identification of obstacles and problems, thereby prompting the exploration of solutions to address them.
When examining various domains, such as space projects, it may be necessary to adjust the scores assigned to maturity increments based on domain-specific knowledge. However, the underlying principle can still be applied. TRLs are used in many companies to assess the technical maturity in R&D projects, but it is difficult to estimate the levels of effort required in climbing the TRL ladder. This leadership BRIEF offers a framework to address this issue. The introduction of the refined TRLs and scores closes a gap in technology management: the quantification of the expected (average) increase of efforts when climbing the TRL ladder. The score table should be calibrated for specific sectors based on empirical data from past projects. Refined TRL levels and the scores can then provide actionable intelligence to describe the final destination of a development project, to measure its efficiency, to detect problems and to adapt management strategies where needed.
Digging Deeper:
[1] Engell, S., and Klessova, S. (2023). Climbing up the Ladder: How does the effort increase with the technical maturity in R&D projects? Part 1 – The TRL scale: the ladder to climb. https://www.ieee-tems.org/ieee-tems-leadership-briefs/climbing-up-the-ladder-how-does-effort-increase-with-technical-maturity-in-rd-projects/
[2] Klessova, S., Engell, S., Thomas, C. (2022). Assessment of the advancement of market-upstream innovations and of the performance of research and innovation projects. Technovation, 116, 102495. https://doi.org/10.1016/j.technovation.2022.102495
[3] Klessova, S., Engell, S., Thomas, C. (2023). The interplay between the contextual conditions and the advancement of the technological maturity in inter-organisational collaborative R&D projects: a qualitative study. R&D Management, https://doi.org/10.1111/radm.12598 .
About the authors
Svetlana Klessova is Director, Research and Innovation Partnerships at G.A.C. Group, France, an international consulting firm in innovation and performance. Dr. Klessova has implemented 60+ projects around the world. Her PhD thesis and publications won several awards. You can contact her via email sklessova@group-gac.com or LinkedIn Svetlana Klessova.
Sebastian Engell is Professor of Process Dynamics and Operations at TU Dortmund, Germany and IFAC Fellow. He received an ERC Advanced Investigator grant and has been involved in leading roles in numerous academia-industry R&D projects. You can contact him via his email sebastian.engell@tu-dortmund.de or LinkedIn Sebastian Engell.
