AS safety case
The key output of the SACE process is a safety case for the AS. We adopt a commonly‐used definition of a safety case as a “structured argument, supported by a body of evidence that provides a compelling, comprehensible and valid case that a system is safe for a given application in a given operating environment.” [37]. As for the safety assurance process, SACE does not provide general guidance on how to create a compelling safety case for a system. Instead, SACE focuses on what modifications, enhancements or additions are required with respect to a safety case for a more conventional system in order to address the challenges of autonomy. As the baseline for this, we consider a very simple argument structure as shown in the safety case patterns in Figures 2 and 3 below. A safety case pattern documents a reusable argument structure and types of evidence that can be instantiated to create a specific safety case instance [22].
In these figures, and throughout SACE, safety case patterns are represented using the Goal Structuring Notation (GSN) [16]. GSN is a graphical notation for explicitly capturing safety arguments that is widely used in many industries for documenting safety cases. For a detailed description of the notation, the reader is advised to consult the publicly available GSN standard [16].
Figure 2 shows what the top level of a simple safety argument for an AS may look like, documented in the form of a pattern. The intention here is to provide the general form that such an argument should take, rather than to mandate a particular structure. In practice, it is expected that a more detailed safety argument would need to be provided, but that it will retain the characteristics defined by the pattern as discussed below. The safety case for the AS also requires further argument relating to the decomposition of the design and requirements throughout the AS development process as discussed earlier. The safety argument pattern for this system decomposition is shown in Figure 3 and discussed further later in this section.
Figure 2 identifies in white, the elements of the argument structure that are required in a safety argument for an AS, but that would also need to be addressed in a safety case for any system. These are the elements that are affected much less by the autonomous nature of the system. In contrast, those elements of the argument structure that are coloured in Figure 2 represent aspects of the safety argument that are either novel to AS or which are most affected by autonomy. These correspond to the parts of the safety assurance process that are challenged by autonomy as discussed earlier. In this guidance, we provide argument patterns relating to these autonomy‐related aspects identified in Figure 2. The details of the patterns are provided as part of the relevant process activity along with descriptions of the relevant artefacts.
In the argument patterns, we make use of assurance claim points (ACPs) [16], indicated by black squares in the argument pattern, to represent points in the argument at which further argument and evidence demonstrating the confidence in particular elements is required. It can be noted in Figure 2 that many of the argument patterns we provide for AS relate to the confidence arguments for ACPs relating to the key artefacts arising from the AS safety process. This reflects the importance that must be placed on managing uncertainty when considering AS. For example in Figure 2, there can be seen to be an ACP relating to the hazards identified for the AS. This shows it is necessary to provide a confidence argument relating to the sufficiency of the identified hazards. The AS Hazard Identification Argument Pattern provides guidance on the structure of that confidence argument.
Below we describe the elements of the argument in Figure 2.
The top-level safety claim that we consider is that the AS is sufficiently safe throughout its operational life. The definition of what is considered to be sufficiently safe for an AS can be extremely challenging since it must take account of complex factors such as:
- Risk perception ‐ The tolerability of risk for an AS may be different from an equivalent non-autonomous system meaning that a direct comparison may not be an effective way to judge sufficiency.
- Risk trade‐off ‐ Autonomous operation will introduce new risks that must be traded‐off against the potential safety benefits that they may bring.
- Ethical considerations ‐ Any definition of sufficiently safe must take account of the complex ethical issues that are raised by autonomous operations (see [8], and [47] for example).
In addition to this there are many domain‐specific and legal factors that inform this definition.
These issues are the subject of extensive on‐going research that is outside of the scope of this document. For the purposes of SACE we assume that there exists an acceptable definition of "sufficiently safe”. Additional guidance in this area can be found at [3].
The strategy that is adopted is to split the safety argument into a claim regarding the safe operation of the AS when it is operating within the defined autonomous operating context (G1), and also a claim that it remains sufficiently safe when outside of that operating context (G7). This requires that the operating context has been explicitly defined (Stage 1 describes how this is done). The operating context description is provided at C1. However, it is also crucial for assurance of AS that an argument is provided, supported by evidence, to demonstrate that the operating context that is defined is sufficient to support the safe operation of the AS.
The Operating Context Assurance Argument Pattern ([G]) is provided in order to guide the development of this argument, and is discussed in detail in Stage 1. The link to the assurance argument for the operating context is established in Figure 2 using an Assurance Claim Point (ACP) [16] (indicated by the black square). ACPs represent points in the argument at which further assurance is provided.
This safety claim relates to the safe operation of the AS when it is operating within the defined autonomous operating context.
The safe operation of the AS in the defined autonomous operating context is demonstrated through an argument that all of the hazardous scenarios associated with the operation of the AS have been sufficiently mitigated. The hazardous scenarios that are identified are provided at C2. The completeness and correctness of the identified hazardous scenarios for the AS must be demonstrated. The AS Hazardous Scenarios Identification Argument Pattern ([I]) is provided in order to guide the development of this argument, and the activities and artefacts are discussed in detail in Stage 2.
The identified hazards for the AS are mitigated through ensuring that the AS operates in such a manner that the defined Safe Operating Concept (SOC) is satisfied. The SOC provides a specification for safe operation of the AS taking account of the operating context and the identified hazards. The SOC is discussed in detail in Stage 3. It must be demonstrated that the operation specified by the SOC sufficiently mitigates the identified hazards. The SOC Assurance Argument Pattern ([N]) is provided in order to guide the development of this argument.
This safety claim is supported by an argument that considers the control and mitigation of hazards and the associated safety assurance of the AS throughout the decomposition of the development lifecycle (as presented in Figure 4). This AS Decomposition Argument Pattern is presented in Figure 3 and discussed below.
This safety claim relates to the safety of the AS when operating outside of the defined operating context. Although the AS is only expected to operate autonomously within the defined operating context, there may be occasions when it is operating outside of that. This may be planned non‐autonomous operation where a human operator takes control of the system, or it may be unplanned excursions from the defined operating context. In all such cases it is important to be able to demonstrate that the system remains sufficiently safe. The Out of Context Operation Argument Pattern ([PP]) is provided in order to guide the development of this argument, and the activities and artefacts are discussed in detail in Stage 7.
Figure 3 shows the further development of the high‐level safety argument in Figure 2. This provides a pattern of argument that captures the general form that the safety argument should take for each level of decomposition in the system development process (described as a “tier” in the argument pattern). As discussed earlier, in this guidance we make no assumptions about the number of levels of decomposition in the AS development process, instead we require that a safety argument is developed that considers safety at each of those levels, irrespective of how many levels there may be. The argument pattern shown in Figure 3 is based upon patterns originally developed for the software aspects of systems [19], but which also apply more generally. The characteristics of this pattern are discussed below.
In order to demonstrate that the development of the AS satisfies the SOC, the strategy adopted in the safety case is to provide an argument and evidence that considers the safety requirements identified at each tier (level of decomposition). If we refer back to Figure 1, this will include requirements at the sub‐system and component level, but as discussed, could include requirements for additional tiers as appropriate.
The safety requirements that have been identified for the tier under consideration are provided as context at C4. It must be demonstrated that these safety requirements have been correctly decomposed, allocated and interpreted from the requirements of the previous tier of development. The AS Safety Requirements Argument Pattern ([S]) is provided in order to guide the development of this argument, and the activities and artefacts are discussed in detail in Stage 4.
Two claims are made as part of this strategy. Firstly that the defined safety requirements have been addressed by the implementation of the AS design (G5) and secondly that any potential hazardous failures that may be introduced through the design approach adopted have been managed (G6).
This claim focuses on demonstrating that each of the safety requirements defined for the tier have been addressed in the design of the AS at that tier. As indicated by the multiplicity in the argument pattern, a claim of this nature must be created for each of the defined safety requirements (this means that a specific argument is provided as to the satisfaction of each safety requirement).
These claims that the safety requirements are addressed can be supported by providing evidence to demonstrate their satisfaction (G8) as well as through further decomposition of the requirement to further tiers of the development lifecycle (G9).
The design of the AS at the current tier is provided as context at C5. It is necessary to demonstrate that the design decisions that have been made at each tier are appropriate to enable the satisfaction of the safety requirements. The AS Design Assurance Argument Pattern ([U]) is provided for this, and the activities and artefacts for design assurance of the AS are discussed in detail in Stage 5.
There will always exist the potential for hazardous failures to be introduced during the design of an AS as a side effect of design decisions. The nature of those potential failures must be correctly understood so that they can be acceptably managed. Once the design for a tier is known, the potential failures that could result from that design solution can be understood so that appropriate mitigations can be identified (such as design changes or the derivation of further requirements). This claim focuses on the management of those potential hazardous failures. The Hazardous Failures Argument Pattern ([DD]) is provided for this, and the activities and artefacts are discussed in detail in Stage 6.
This claim states that evidence is provided that demonstrates that the safety requirement is satisfied. Such evidence can be provided at the different tiers of the development arising from the verification activities performed, as illustrated on the right hand side of Figure 4. The argument must demonstrate the appropriateness and trustworthiness of the evidence provided. The AS Verification Argument Pattern ([UU]) is provided in order to guide the development of this argument, and the activities and artefacts are discussed in detail in Stage 8
Where the development of the AS continues to more detailed tiers of the design, a claim is provided that the safety requirement is also addressed through the next decomposition of the AS design. Here the safety argument for the next tier will follow the same form as has been described above (the same argument pattern from Figure 3 is applied again for the next tier). This is indicated by the loop from G9 back to S2 which indicates that the argument is repeated.
