In defining the context of operation of an AS it is important to first define the scope of the system’s autonomous capabilities. It should be noted that this will in many cases be a sub‐set of the full capability of a system where only some of the functionality is provided autonomously, or the system is only capable of performing tasks autonomously some of the time.

Autonomous capabilities of an AS may be provided entirely by the system itself or with shared responsibility between a human, or humans, and the AS. This includes systems where a human is required to monitor the autonomous operation of the AS and intervene if required. Again the limitations on such capabilities should be clearly defined along with the separation of responsibilities.

Defining capabilities

In defining the autonomous capability of an autonomous robot deployed for delivering small packages around an office environment it is specified that the robot will not be capable of autonomously loading/unloading the packages. A human operator is relied on for the safe loading of packages.

Delivery robot

The autonomous capabilities of a self‐driving car vary for different levels of driving automation. For example an SAE level 3 vehicle [[40]]() may provide autonomous traffic jam pilot capabilities under defined conditions, whereas a level 4 vehicle may provide fully autonomous driving capability under defined conditions.

Self‐driving car

The definition of autonomous capabilities of the AS will often be an iterative process with activity 2 since the capabilities must be chosen in order to match the requirements of a given ODM.

Iterative process

AS concept definition

Define autonomous capabilities of AS

The Operational Domain Model (ODM) defines the scope of the operation of an AS within which the AS can be demonstrated to be acceptably safe when carrying out its autonomous capability. When AS are developed and put into operation it is based on assumptions made about when, where, and under what conditions that AS will need to operate. The operational scope defined by those assumptions determines the scenarios that may be encountered by the AS as it interacts with its environment. It is illustrated in Figure 6 below how the ODM effectively reduces the number of possible scenarios that the AS may need to deal with when performing its autonomous capability.

medium_defining an operational domain model 2.png

small_defining an operational domain model 2.png

thumbnail_defining an operational domain model 2.png

defining an operational domain model 2.png

The assumptions must be made explicit in the form of a defined ODM that characterises the operating domain. If the ODM is insufficiently defined then the AS may encounter scenarios during its operation that were not considered during the development of the system, and which could therefore be unsafe and for which no assurance is provided. It is crucial therefore that all relevant elements of the operating domain, including those with with the AS may interact are included within the ODM. In identifying relevant elements of the ODM it is important that “non‐mission interactions” are also considered [18].

For autonomous cars, the term Operational Design Domain (ODD) [23] is often used to refer to the ODM. A number of approaches to defining the ODD for an autonomous car have been proposed. NHTSA [45] defined and categorised an ODD taxonomy based on a review of over 50 sources of literature in automotive and other domains. The taxonomy is intended to be descriptive, recognising that other organisations of the elements are possible. The report provides a set of sample baseline ODDs for different automated driving features or capabilities, such as those shown in Figure 7 below for an Automated Highway Drive (HWD) function. 

Note that the examples in Figure 7 are used to define what is asserted as being within the ODM for that particular capability (function). If the operation of the vehicle is outside of this definition then the safety of that capability is no longer assured. We return to the issue of operation outside of the defined operating context at Stage 2.

Autonomous cars

medium_table1.png

small_table1.png

thumbnail_table1.png

table1.png

One of the major challenges when defining an ODM can be identifying an appropriate level of detail at which to specify elements of the ODM. If elements are not specified with sufficient granularity then important differences in the features that may impact safety may be missed. On the other hand, too much granularity can make any analysis based on the ODM intractable due to the high number of element combinations.

It may be necessary to revisit the granularity of the ODM as a result of information learned from other activities; for example, it may be informed by the nature of the relevant operating scenarios identified at Activity 3.

Challenges when defining an ODM

In the ODM for an autonomous robot operating in an office environment, the ODM may include ‘walls’ or ‘doors’ as part of the model. However, relying only on ‘wall’ or ‘door’ as an element of an ODM will miss crucial attributes of walls or doors that will affect the detection capability of an AS. A wall or door made of translucent material would not be capable of detection by some technologies (e.g. LiDAR). In the ODM, the material of the walls and doors should also be specified.

Office environment

In the ODM for an autonomous shuttle bus operating in pedestrianised areas, the ODM must include people as part of the model. However, defining “people” as a feature of the ODM may miss crucial differences between children and adults that affect the safety of the AS operation. “Children” and “adults” should therefore be defined as separate elements of that particular ODM. The appropriate elements to choose and their granularity would be expected to differ for each ODM depending upon the particular operational context of the system.

Defining people

Once created the ODM shall be validated to check that both the scope of the defined model and its level of detail are appropriate. The results of the validation activity shall be explicitly documented ([C]).

The process of defining and validating the ODM is an iterative process, and any initial ODM specification may need to be expanded or reduced based on the outputs of later process activities. This may include new information that is obtained about the design, operation, capabilities, limitations and/or potential failure conditions of the AS.

Define and validate the Operational Domain Model (ODM)

An operating scenario of an AS represents a particular set of actions and events that the AS may undertake in a defined environment situation. We adopt the characterisation used in [10] as a description of the AS, its activities and/or goals, its static environment, and its dynamic environment.

The following distinction is often made between “scenes” and “scenarios” [46]:
- A scene describes a snapshot of the environment including the scenery and dynamic elements, as well as all actors’ and observers’ self‐representations, and the relationships among those entities. Scene descriptions will inevitably be incomplete and vary from one or several observers’ points of view.
- A scenario describes the temporal development between several scenes in a sequence. Every scenario starts with an initial scene, and the temporal development is characterised by a set of actions and events.

Scenes and scenarios

An example of a scene could be a typical office environment, with office furniture, artificial lighting, and an autonomous robot in a corridor that is also used by human staff members. A scenario could develop where an autonomous robot and a human are approaching a right‐angle in a corridor from opposite directions.

Figures 8 and 9, taken from [10] provide an example of a scenario description for an autonomous vehicle.

Autonomous vehicle scenario

thumbnail_scehmatic overview of the scenario.png

scehmatic overview of the scenario.png

medium_a qualitative description of the same scenario.png

small_a qualitative description of the same scenario.png

thumbnail_a qualitative description of the same scenario.png

a qualitative description of the same scenario.png

It is crucial for safety assurance of the AS that the operating scenarios that are relevant to the AS within its defined ODM are identified as completely and correctly as possible. If we refer to [Figure 6](/sace/activities/define-and-validate-the-operational-domain-model-odm#page), we must identify the operating scenarios present in the area inside the orange boundary defining the scope of the ODM. The green area in the diagram represents all of the identified scenarios within this area. It is not possible for a complex, open environment, such as an urban street, to provide an exhaustive detailed specification of the operating scenarios, since the set of scenarios is simply too large.

Where operating scenarios exist within the scope of the ODM, but are not correctly identified, these could pose a safety risk to the AS since the scenario could be hazardous, but would not be assessed as part of the safety assurance process and therefore no mitigations put in place. This is represented by the red area in Figure 6. It is very important therefore for safety assurance to have confidence that this area is as small as possible by ensuring the operating scenarios are sufficiently well defined.

It is partly as a result of the possible existence of the red region in Figure 6 that resilience is such an important requirement when designing ASs; these unanticipated operating scenarios require that the system is able to adapt in a safe manner to unplanned events. This is discussed in more detail at Stage 6.

Resilience

The operating scenario descriptions shall be validated to provide evidence that they are sufficiently complete and correct. Evidence can be provided based upon review of the scenario specification. A rigorous specification to a defined format makes the scenarios more amenable to review and reduces ambiguity. It is important that review is carried out by a range of experienced stakeholders. Providing simulations of the defined scenarios may help to ensure the reviewers have a correct understanding of the scenarios. The defined scenarios can also be checked against collected field data from AS in operation to ensure that all encountered scenarios are captured in the operating scenario specification.
The results of the data validation activity shall be explicitly documented ([F]).

The definition of scenarios should be an iterative process, where the scenarios are refined based on increased understanding of the relevant and important aspects of the scenario space. This refinement may be informed by the outcome of the validation activities described above, particularly the results of simulation and field‐based validation tests.

Define and validate the operating scenarios

This activity requires as input the AS operating context assurance argument pattern ([G]), as well as the artefacts from Activity 1, Activity 2 and Activity 3 ([B], [C], [D], [E]and [F]). The activity uses these artefacts to create an instantiated AS operating context assurance argument ([H]) which demonstrates that the defined ODM is sufficient to support the safe operation of the AS.

AS operating context assurance argument pattern

Instantiate AS operating context assurance argument pattern

Correctly understanding the interactions that the AS has with elements of the operating environment is a crucial part of identifying the hazardous scenarios. This activity focuses on defining those interactions.

A complex operating environment can have a large impact on any hazardous behaviour of an AS, since it increases the probability of the system encountering unusual and unanticipated scenarios. Although complex environments are not unique to AS, traditional systems often rely on the involvement of a human as an operator of the system to deal with any resulting unanticipated or unusual scenarios. For autonomous operation, a human operator is not available to deal with these scenarios, so the AS itself must be able to deal safely with such situations so that they do not become hazardous (through inappropriate decision making). This requires therefore that analysis of hazardous scenarios for an AS incorporates consideration of the operating environment in a more systematic and explicit manner than is currently the case for human‐controlled systems.



medium_SUDA agent model of an AS.png

small_SUDA agent model of an AS.png

thumbnail_SUDA agent model of an AS.png

SUDA agent model of an AS.png

The ODM ([B]) is a key input used to identify the interactions that the AS may encounter during operation.

The consideration of interactions is not limited solely to the interactions between an AS and other agents explicitly required for the purpose of carrying out a task (sometimes referred to as ‘mission interactions’). Consideration must also be given to unexpected interactions with features such as the type of terrain encountered, as well as with unexpected agents that are not ordinarily involved in the task (‘non‐mission interactions’) [18].

Mission and non-mission interactions

- The ODM for an autonomous robot operating in an office environment may identify possible interactions with office workers, visitors, other robots etc.
- The ODM for an autonomous car may identify possible interactions with different types of road surfaces and weather conditions etc.
- The ODM for an autonomous insulin infusion pump may identify possible interactions with different healthcare professionals (e.g. doctors, nurses), as well as the patient, other medical devices etc.

ODM and identified interactions

Having identified the elements of the ODM with which the AS may potentially interact, the defined operating scenarios ([E]) can then be used to identify the particular interactions that may occur with those elements. This requires each scenario to be broken down into a set of steps that are undertaken by the AS as described below:

1. **Define start and end points of the scenario**: identify the logical start/end points (the former may often be the event that triggers the scenario, and the latter asserted as the point at which the scenario is successfully completed).
2. **Define the steps**: establish the steps involved in undertaking the scenario. Some of these steps may represent an ‘understanding point’ in the scenario where the AS requires information about a particular feature of the operating environment. These understanding points should be captured as questions with a binary (yes/no) output, such as “is an obstacle present in the planned path of the robot”. Other steps may represent a ‘decision point’ where the AS must make a choice, such as to reduce speed.
3. **Identify the interactions**: Determine which of the defined steps may involve interaction with elements of the operating environment defined in the ODM.

Define interactions between the AS and the environment

Having modelled the steps and interactions of the operating scenario, each of the identified decision points can then be analysed in order to determine the nature of any hazardous scenarios that may arise.

Each of the decision points identified in Activity 3 should be selected. For each, the possible environmental states that may arise within or emanate from the operating environment are identified before enumerating the decisions that the AS could select at that decision point.

A decision point for an autonomous robot moving in a building may correspond to the detection of an object in the robot’s path. The options available to the AS at this point are:
1. Continue on the current path at the current speed, OR
2. Continue on the current path at a reduced speed, OR
3. Change path to avoid the object, OR
4. Stop and wait.

Decision point for an autonomous robot 

The different possible scenarios and environmental states that relate to the decision can then be identified by considering the real world state and the belief state of the AS at the point at which the decision is made, along with each of the possible options. The real world state represents the actual state of the operating environment, whilst the belief state represents the understanding that the AS has of the state of the operating environment.

An example of some enumerated situations for an autonomous robot encountering an object in its path is shown in Table 1. The real world and belief states are represented as Boolean states where ‘True’ represents the presence of the object and ‘False’ represents that the object is not present. So situation 5 in Table 1 represents the situation where there is an object in the path of the robot, but the robot is unaware of the object and continues on its current path at its current speed.

Enumerated situations for an autonomous robot

For each of the identified scenarios, the outcome can be defined in order to identify those that may be potentially hazardous.

small_table2.png

thumbnail_table2.png

table2.png

For an autonomous car turning right at a roundabout, one decision point is the car entering the roundabout. The options for the car at this decision point are:
1. Enter the roundabout
2. Stop and wait.

A possible interaction at this point is with another road user, in this example a cyclist. Relevant scenarios can be identified by considering the real-world presence of the cyclist in combination with the car’s belief that a cyclist is present, and each of the options identified above. For example, one situation is that a cyclist is not present, but the car believes that there is a cyclist and decides to stop and wait. This scenario could lead to a hazardous outcome as an unnecessary and unexpected stop could potentially cause a rear‐end collision.

Autonomous car turning at a roundabout

For an autonomous insulin pump that is monitoring a patient’s blood sugar level, one decision point is whether to alter the infusion rate to the patient. The options for the pump at this decision point are:

- Increase the insulin infusion
- Decrease the insulin infusion
- Maintain the current rate of insulin infusion
- Stop the insulin pump.

Relevant scenarios can be identified by considering the real-world change in the patient’s blood sugar level in combination with the pump’s belief in the current sugar level, and each of the options identified above. For example, one scenario is that the patient’s blood sugar level rises, the pump also has a belief state that the blood sugar level has risen and yet decides to maintain the current rate of insulin infusion. This scenario could lead to a hazardous outcome as the patient's blood sugar level could increase to unsafe levels.

Autonomous insulin pump

Considering the possibly hazardous scenarios for an autonomous robot encountering an object in its path in Table 1, it is possible to also assign a severity of outcome, and consider whether minor or serious injuries (or even fatalities) are likely.

It is also possible to further consider how this severity could be impacted by factors pertaining to the
operating enviornment.

An autonomous robot moving in a building that fails to detect an object in its path, and therefore continues on the current path at the current speed, could impact a static object. If the static object is an adult human, the severity of the impact could be minor (bruise or laceration). Should the static object be a small child, then the severity of the impact could be major (broken bones). The robot may have detected the static object, but decided to maintain its current path, but reduce speed. Wet floor surfaces may reduce the effectiveness in braking (through wheel slippage), and this higher then expected collision speed could impact the severity of the collision.

Autonomous robot 

For an autonomous car turning right at a roundabout, the autonomous car may fail to detect a dynamic object that would intersect it should it enter the roundabout. If this dynamic object is a car, and was collided into by the autonomous vehicle, the occupants may suffer only minor injuries (if any at all). Should the dynamic object be a cyclist, then the collision may result in a fatality. Should the road surface be icy, the force of impact may displace an impacted car onto a fixed object in the environment, thereby increasing the severity of the collision for the car’s occupants.

Autonomous car 

For an autonomous insulin pump that is monitoring a patient’s blood sugar level, the autonomous infusion pump could decrease the insulin infusion without medical need, should the patient be experiencing a spike in blood sugar levels at the time the infusion was reduced, then the patient may become hyperglycaemic. Should the patient have additional health complications, or the reduction in insulin infusion go unobserved by a clinician, then the severity could be impacted and result in a fatality.

The analysis undertaken in this activity shall be documented ([WW]).

Analyse AS decisions

Based upon the analysis performed in Activity 6 it is possible to define the hazardous scenarios associated with the operation of the AS in the defined operating context. These may be specified using the general form presented earlier:

<AS operating scenario> <relevant environment state(s)> AND <decision>

- An AS hazardous scenario for an autonomous robot is: <The autonomous robot is following a planned path>< with a static object present in the path>  **AND**  < the robot maintains its current speed and direction>

- An AS hazardous scenario for an autonomous car is: <the car is approaching a roundabout><with a cyclist on the roundabout to the vehicle’s right>  **AND**  <the car enters the roundabout>

- An AS hazardous scenario for an autonomous insulin infusion pump is: <the insulin pump is monitoring the patient’s blood sugar level >< when the sugar level rises> **AND** < the pump maintains the current insulin infusion rate>

Hazardous scenarios

The system environmental belief state is NOT an element of what constitutes a hazardous scenario, as belief states are merely hypothesised causes of a failure.

System environmental belief state 

These AS hazardous scenarios shall be explicitly documented ([XX])

Identify AS hazardous scenarios

This activity requires as input the AS hazardous scenarios assurance argument pattern ([I]), as well as the artefacts from Activities 5 to 8 ([WW], [XX], [YY]). The activity uses these artefacts to create an instantiated AS hazardous scenarios assurance argument ([J]) which demonstrates that the hazardous scenarios have been correctly identified.

AS hazardous scenarios assurance case pattern

Instantiate AS hazardous scenarios assurance case pattern

The safe operating concept (SOC) specifies sufficiently safe operation of the AS within the operating context defined by [B], [D] and [E] taking account of the identified hazardous scenarios from [XX] at Stage 2. The SOC shall include a definition of system-level safety requirements that specify how the AS must behave in order to provide sufficient mitigation for the hazardous events. The system safety requirements may be specified for the AS using either natural language or more formalised representations. Whatever representation is chosen, of most importance is that the requirements are clear and unambiguous.

There is existing good practice for requirements specification that can be used to guide the specification of system safety requirements as part of the SOC, such as [32] which provides a generic requirements syntax:

**\<optional preconditions> \<optional trigger> the \<system name> shall \<system response>**

In addition to five templates for different requirement types:

**Ubiquitous requirement**
These are requirements that always hold and therefore have no preconditions or trigger (sometimes referred to as invariants). These requirements should have the form:
**The \<system name> shall \<system response>**

**Event‐driven requirement**
These are requirements that are initiated when, and only when, a triggering event (such as a change in the operating environment) is detected at the system boundary. These requirements should have the form:
**When \<optional preconditions> \<trigger> the \<system name> shall \<system response>**

**Requirements to handle unwanted behaviour**
These requirements are a variant of the event‐driven requirements that specifically deal with unwanted events (such as failures). These requirements should have the form:
**If \<optional preconditions> \<trigger>, then the \<system name> shall \<system response>**

**State‐driven requirement**
These are requirements that are valid while the system is in a defined state. These requirements should have the form:
**While \<in a specific state> the \<system name> shall \<system response>**

**Optional feature requirement** These are requirements that are valid only in systems that include a given feature or capability. These requirements should have the form:
**Where \<feature is included> the \<system name> shall \<system response>**

**Requirements with complex conditions**
Combinations of keywords When, While and Where can be use to build complex expressions to specify more complex As behaviours. For example:
**While the robot is moving, when a person is present, the robot shall issue an audible warning**

A structured syntax approach such as that described above provides a method for mitigating against syntactic ambiguity.

Mitigating against semantic ambiguity requires ways to ensure that the meaning of words and phrases is clearly defined within the relevant context and that only that meaning is used. This can be achieved through the use of domain or project specific dictionaries or through developing context‐specific ontologies. These should include definitions of concepts and elements included in the ODM.

Requirements syntax

The SOC for an autonomous car includes the following safety requirement: “The AV shall maintain sufficient distance between itself and any vehicle in front in order to provide enough time to react if the car in front suddenly brakes.” This requirement has been specified in order to mitigate the hazardous event of the autonomous car colliding into the rear of another vehicle. For such a requirement, the distance can be calculated for different driving speeds based upon a number of assumed properties such as the minimum and maximum braking performances of the cars and the frictional coefficient of the road surface (see [42] for details of such calculations). These assumptions must be validated and explicitly documented as part of the assurance case.

Autonomous car braking requirement

The SOC for an autonomous excavator operating on a construction site includes the following safety requirement: “The excavator shall ensure maximum tilting angle is never exceeded.” This requirement has been specified in order to mitigate the hazardous event of the excavator toppling whilst digging [41]. This could occur due to an incorrect configuration of the excavator whilst digging on uneven or unstable ground or as a result of resistance to the bucket.

SOC for autonomous car on construction site

The SOC for an autonomous insulin infusion pump used in a busy intensive care unit treating multiple patients concurrently may contain the following safety requirement: “The alarms and warnings function of the autonomous insulin infusion pump shall not unnecessarily distract or disturb ICU nurses from their other tasks”.

Autonomous insulin infusion pump

The safety requirements defined as part of the SOC should be defined with reference to the system‐level behaviour of the AS, that is behaviour that is visible on the boundary of the AS, rather than referring to any particular sub‐system or component. For instance, the safety requirement relating to rear collision in Example 17 should not be written as a requirement on the braking system, but, as is shown, on required behaviour of the vehicle as a whole. This requirement and its assumptions may then lead to safety requirements being derived upon the braking system through a consideration of the system design later in the lifecycle (see Stage 5).

System-level safety requirements

In addition to specifying system safety requirements, as part of the SOC it may also be necessary to define a reduced operating domain (ROD). As shown in Figure 15, a ROD is a definition of additional constraints on the ODM in order to reduce the operational scope of an autonomous capability under certain conditions. By operating in a ROD, the intention is that the AS is not exposed to scenarios that may lead to hazardous events under those defined conditions.

There are a number of reasons why a ROD may need to be defined. Often a ROD will be defined with respect to a particular system state where it would be unsafe to operate within the entire scope of the ODM. This may often be in response to a particular system or component failure mode.

small_defining a reduced operating domain 2.png

thumbnail_defining a reduced operating domain 2.png

defining a reduced operating domain 2.png

If there is a failure of the long range object detection sensors on an autonomous passenger shuttle, the vehicle enters a reduced operating domain that requires a low maximum operating speed for the vehicle. This enables the vehicle to transport the passengers to a safe location, but to do so without increasing the risk of collision whilst relying only on short range sensing capability.

Autonomous passenger shuttle

If there is a data communication failure between an autonomous insulin infusion pump, and the centralised electronic patient healthcare records system, the autonomous pump may be prevented from increasing or decreasing the rates of insulin infusion until either communications are restored, or action is taken by a healthcare professional.

Although RODs may be identified early in the lifecycle, it is likely that RODs will often be formulated or revised later in the lifecycle in response to understanding gained through later activities such as hazardous failures identified in [BB]at Stage 6.

In order to define an SOC that provides sufficient mitigation for the hazardous scenarios under the specified conditions (including failures modes), there is generally a choice as to the safety strategy that is adopted:

1. The AS continues to provide all of the Autonomous Capability (AC) under those specified conditions but a ROD is defined that limits the scope of operation under which that AC is provided.
2. The AS continues to operate under the same ODM but the AC is limited.
3. A combination of 1) and 2) is adopted.

Sufficient mitigation for hazardous scenarios

An autonomous car driving on a highway enters a zone with emergency roadworks in place. There are different safety strategies that could be adopted under these conditions. The car could continue to operate fully autonomously, but the maximum allowable speed could be reduced (Strategy 1). The car could continue to operate up to the normal speed but only provide lane-keeping capability, handing other control to the driver (Strategy 2). Or the car could have both its speed constrained as well as restricting its capability to lane keeping only (Strategy 3).

Autonomous car in emergency roadworks

Definition of sufficiently safe

Define safe operating concept of AS

The SOC defined in [L]shall be validated in order to check that:

- The AS safety requirements defined as part of the SOC specify sufficient mitigation for all of the identified hazardous scenarios ([XX]).
- The AS safety requirements are clear and unambiguous.
- The strategies defined by the RODs and reduced ACs provide a sufficient mitigation for the identified hazardous scenarios under the specified conditions.

Validation of the SOC will often require the involvement of multiple stakeholders, particularly the system developers and operators who have the necessary domain knowledge and understanding of the system operation. The stakeholders should provide an independent view on whether each of the points above is satisfied by the SOC definition.

One option for demonstrating the SOC for the purposes of validation is to use simulation. Simulation allows for earlier validation of the system, as well as more more rapid exploration of the operational space, than would be possible with the real system. Since simulating the entire AS can be prohibitive, a hardware in the loop approach may be used where pre‐captured real‐world sensor data is input to the simulation which simulates the AS response.

Simulation

Validate safe operating concept for AS

This activity requires as input the SOC assurance argument pattern ([N]), as well as the artefacts from Activities 10 and 11 ([L]and [M]). The activity uses these artefacts to create an instantiated AS operating context assurance argument ([O]) which demonstrates that the defined SOC sufficiently mitigates the hazardous scenarios identified for the AS operation.

SOC argument pattern

Instantiate SOC assurance argument pattern

This activity considers how the safety requirements are defined at each tier of decomposition in the system development process such that they adequately capture the intent of the safety requirements that were established at the previous tier of decomposition ([P]). This requires consideration of the design that is proposed for this tier of decomposition ([W]) such that safety requirements may be adequately decomposed and allocated to the relevant system components. The safety requirements must also be correctly interpreted for the allocated component to reflect the component design.

We can identify the different tiers for a typical AS by considering the example in Figure 18 below, taken from [35], which shows a typical logical architecture for the controller of an autonomous vehicle. At the highest level, we could consider the AS itself as a tier of decomposition. At this level, the interactions between the AS and other entities would be considered. The next tier in Figure 18 would decompose this to the three main subsystems (Sensing, Control Block and Actuation) and their interactions. Each of these subsystems can then be seen to decompose at the next tier to components such as camera, LIDAR and GPS for the sensing subsystem, or Perception, Decision and Planning for the Control Block. Whereas the components may be considered the lowest level tier for the Sensing subsystem (since these are provided as developed units to the AS developer), further tiers of decomposition may be required for other components such as the Perception component, which is broken down to Localisation, Detection and Prediction. As illustrated in this example, the number of tiers required as part of the development process may not be uniform across all elements of an AS.

Different tiers of an autonomous system

In theory, all of the information required to demonstrate the adequacy of the decomposed safety requirements would be captured in the initial high‐level requirement. In practice, however, some of the information will always remain implicit. For AS, this could particularly be due to the complexity of the operating environment, but also to the fact that design decisions will always be made later in the development lifecycle that require greater detail in the requirements. This detail cannot be properly known until the design decisions have been made. For this reason, it is important to ensure that the intent of the safety requirements is maintained between levels of abstraction, with consideration being given to information that may not be fully specified in the requirement.

Intent of the safety requirements

Figure 19, adapted from [21], illustrates how safety requirements are derived for a component of a single tier of decomposition in the development of an AS. This takes as input information from the previous tier of decomposition relating to the safety requirements placed on the components at this tier, as well as the design information. This information is used to create a design for the component (as discussed in Stage 5). This design will specify the sub‐components that are required and the relationships between those sub‐components.

small_a simple example of the logical architecture for the controller of an autonomous vehicle.png

thumbnail_a simple example of the logical architecture for the controller of an autonomous vehicle.png

a simple example of the logical architecture for the controller of an autonomous vehicle.png

At the highest level of abstraction, which is the overall AS system level (“tier 0”) the safety requirements will be those defined by the SOC (see Stage 4). These will then be decomposed to more detailed safety requirements at the next tier of design decomposition (“tier 1”). Each time the safety requirements are further decomposed to more detailed tiers it is expected that additional safety requirements will be needed. Thus the total number of safety requirements specified for the AS will increase as more detail is added to the design of the system. It may also be the case when safety requirements are derived for a component, that additional safety requirements relevant to a previous tier are also identified. These safety requirements should be fed back up, in order to be correctly decomposed to the relevant system components.

Decomposing safety requirements

Some of the sub‐components defined for a tier may be ready for implementation. For these components no further design activity is required; these components are at the stage where they can be implemented according to the requirements that are in place. These components are ready for verification and integration (see Stage 8). For other sub‐components further design may be required before they are ready for implementation. For these components a further tier of requirements decomposition will be required (along with further design activity). As shown in Figure 20, adapted from [21], different components may be ready to be integrated to the target AS platform at different tiers. Irrespective of the number of tiers in the development lifecycle of the system, the safety assurance considerations for the requirements at each tier remain the same: to derive safety requirements that adequately capture the intent of the the safety requirements specified at the previous tier, and to justify the sufficiency of those safety requirements.

medium_defining safety requirements for components of a tier of design decomposition.png

small_defining safety requirements for components of a tier of design decomposition.png

thumbnail_defining safety requirements for components of a tier of design decomposition.png

defining safety requirements for components of a tier of design decomposition.png

The definition of safety requirements is intimately linked to the design process of the AS. It is important to note therefore that safety requirements may need to be re‐defined throughout the lifecycle based upon the architectural and design decisions that are taken. For example, many AS make extensive use of third‐party components to implement parts of the design (such as sensors). If a third‐party component is selected to be used, it may be found that the safety requirements that were originally derived for that component cannot be fully met by that third‐party component (the same could be true for legacy or re‐used components). This would therefore necessitate changes to the system design to mitigate those limitations (such as introducing additional components). This 
would then require a re‐definition of the safety requirements allocated to the components to reflect the changes to the design. It may then be possible to demonstrate that the third-party component can meet these redefined safety requirements.

Essentially it may be necessary to revisit the safety requirements at any tier multiple times as changes are made to the proposed design solution. What is crucial is that throughout this process, as changes are made to the design and the safety requirements, traceability is maintained to the safety requirements of the higher tier, and ultimately to the SOC for the AS operation.

Link between safety requirements and the design process

Where safety requirements are allocated to a component that uses machine learning, the safety assurance of that component may be undertaken in accordance with the AMLAS guidance [20].

Machine learning

Define safety requirements

The safety requirements documented for each tier in [Q] shall be validated in order to check that they adequately capture the intent of the more abstract safety requirements defined at the previous tier.

small_the decomposition of AS safety requirements.png

thumbnail_the decomposition of AS safety requirements.png

the decomposition of AS safety requirements.png

This will require that it is checked that each of the higher level requirements can be satisfied if the safety requirements for this tier are correctly implemented. It is therefore important that the validation activities focus on the semantic equivalence of the safety requirements at different tiers.

Demonstrating that the intent of the safety requirements is captured requires more than simply stating that a relationship exists between safety requirements at different tiers. Some explanation and justification for the sufficiency of that relationship must be provided. Concepts such as “Rich Traceability” [12] could help in this regard.

Ensuring the intent of the safety requirements is maintained throughout decomposition may often be more challenging for AS than it is for traditional systems due to the sometimes large ”semantic gaps“ that can exist. The nature of these gaps and the challenges of addressing them for AS is discussed in more detail in [7]. 

Relationships between safety requirements

The output from the safety requirements validation activity shall be documented in the safety requirement justification report ([R]).

Validate safety requirements

This activity requires as input the safety requirements argument pattern ([S]), as well as the artefacts from the previous activities of this stage ([P], [W], [Q]and [R]). The activity uses these artefacts to create an instantiated safety requirements assurance argument for the AS ([T]) which demonstrates that the defined safety requirements sufficiently capture the intent of the SOC for the AS.

Safety requirements argument pattern

Instantiate safety requirements argument pattern

Given a set of safety requirements defined at a tier of decomposition ([Q]) of the AS design, a design for that tier shall be produced that ensures those safety requirements can be met. This will involve making design decisions that are appropriate given the overall context of the safety requirements, the operating context and the known failure modes. In particular decisions relating to the system architecture are of particular importance when considering the satisfaction of safety requirements for an AS.

In the context of AS, we use the following definition of system architecture, adapted from [1]: “An architecture comprises the rules and constraints for the design of a system, including its structure and behavior, that would meet all of its established requirements and restrictions under the anticipated environmental conditions and component failure scenarios.”

Definition of system architecture

A system architecture may capture either the logical or physical structure of the system. Decomposition of an AS design may often involve a transition from logical architecture to a physical one. During such a transition there is often an increased likelihood that errors may be introduced to the design.

Logical and physical structure of the system

The design of the system architecture at each tier shall consider the need for robustness, fault tolerance and runtime monitoring in order to satisfy the safety requirements defined for that tier.

***Robustness*** can be defined as the delivery of a correct service in implicitly‐defined adverse situations arising due to an uncertain system environment [31]. Robustness is therefore a mechanism that is particularly important for AS since it enables the AS to mitigate hazardous system failures associated with hard to predict, and thus unexpected, changes in a complex operational 
environment. This may for example include objects in the environment that were not included as part of the ODM which the system fails to detect, or unexpected effects of particular lighting conditions that lead to “phantom objects” being detected. Since these events have not been anticipated, specific hazardous failures will not have been identified as part of the assurance process. However, it is expected that hazardous failures of this type would be identified, leading to a requirement for the AS to be designed such that it is robust to those types of failures.

Robustness in AS systems is typically achieved through redundancy in the architecture. The purpose of the redundancy is to provide compensation in the design of the system for potential limitations in system components that could result in unexpected system failures. Such redundancy may be required for both hardware and software elements of the architecture, and may be used at multiple architectural levels.

Redundancy in the architecture alone will not necessarily provide the robustness for the AS that is required. Identical components will contain the same limitations and failure modes. This would mean that the components, when exposed to the same set of inputs, would suffer the same failures. To be effective, there must therefore be diversity in the redundant architectural elements. This requires some level of independence between the components, for example by using components developed by different teams of people in an attempt to ensure the same mistakes are not replicated in each component, or by providing conceptual diversity through providing different specifications for each of the redundant components.

Ensuring diversity for software components particularly challenging [29], [4] and therefore requires particular attention for AS (with their high reliance on software). The use of artificial intelligence techniques such as machine learning (ML) for developing components also provides unique considerations for diversity.

Diversity in architecture redundancy 

***Fault tolerance*** can be defined as the delivery of a correct service despite faults arising from the AS itself. The focus of fault tolerance is therefore on the detection and recovery from anticipated failures, which should be identified as part of the hazard assessment of the AS (see Stage 6). Since the  hazardous failures are known during AS development, checks can be included in the system design in order to detect the faults. This could include, for example detecting inconsistencies between the system state as characterised by the sensors and the system state predicted by a model [31]. Once a fault is detected, fault recovery strategies enable the continued provision of the service. Standard software fault tolerance strategies can be applied effectively to AS [39].

There are three basic fault tolerance approaches that can be used: recovery blocks, N‐version programming, and N‐self‐checking software [25].

A recovery block approach provides fault tolerance for a component by using alternate variants of the component plus an adjudicator. The adjudicator tests the outputs from the components. If a component fails then the next alternate component is used and so on. This mechanism can be used to ensure that the behaviour of the component continues to be provided to the AS once a failure occurs in the component. The tests performed by the adjudicator must be sufficient to identify the hazardous failure based on consideration of the safety analysis results [BB] obtained from Stage 6.

In contrast, with an N‐version programming approach, all variants provide outputs to the adjudicator which then decides by considering all the outputs together which result to use. All variants of the component must be functionally‐equivalent, but diversely‐designed. An advantage of this approach is that it does not require tests to be defined for the adjudicator, so can be used for situations where it may be challenging to specify specific test criteria for a hazardous failure.

N‐self‐checking software is a hybrid approach that involves the use of self‐checking software components (SCSC). Each SCSC could use a recovery block or an N‐version program approach. At least two SCSC are required, whose outputs are then checked. Such an approach provides fault tolerance for components or subsystems which are themselves fault tolerant.

Fault tolerance approaches

***Runtime monitoring***

The provision of a runtime monitor as part of the AS design enables the behaviour of the AS during operation to be checked against defined constraints or behavioural predictions. The runtime monitor should be independent from the components it is monitoring, but may take the same inputs. One of the biggest challenges with using runtime monitors is being able to correctly define the constraints or bounds on behaviour that the monitor will check. This definition should make reference to the SOC ([L]) and the hazardous scenarios ([G]) identified from previous stages.

Runtime monitoring may, in addition to monitoring for unsafe behaviour of the AS, also monitor behavioural trends of the AS over periods of time. This would enable the identification of, for example, the deterioration in performance of the AS which may act as a leading indicator of future unsafe behaviour. In this case it is important to determine which information to monitor and how that can be interpreted as representing a threat to the safety of the AS.

The key design decisions that are taken at each tier shall be documented in the AS development log ([V]).

Design process for tier n

Create design at tier n

Justification shall be provided for how each of the key design decisions that have been made at tier n ([V]) help to ensure that the safety requirements can be met by the AS. In particular this should consider how the design decisions relate to the robustness and fault tolerance of the AS, and the way in which this supports the safety of the system. It is also important that the design decisions are reviewed to check that no inappropriate decisions are taken that mean that the safety requirements cannot be satisfied by the proposed design. The justification for the design at each tier shall be documented in the AS design justification report ([Y]).

Even if appropriate design decisions have been taken, the design should also be reviewed to check that errors have not been made in the design. It is particularly important for AS, where substantial use is made of software components, that this includes review of the proposed software design. In particular, the review should check for potentially hazardous errors. The review of the design shall 
 be undertaken by a suitable person who is independent from the design activity itself. The results of the design review at each tier shall be documented in the AS design review report ([Z]).

The sufficiency of the design process used (as documented in [X]) shall also be reviewed to check it is sufficiently rigorous. It should also be checked that the defined process has been correctly followed when developing the design.

The level of independence that is required of the people responsible for the review may vary depending upon the level of risk associated with the AS. For high risk systems it may be required for the review to be undertaken by personnel from a different organisation to the design organisation. For lower risk systems it may be sufficient for the review to be undertaken by personnel from the same organisation who are capable of providing an independent view.

Independent reviewers

Review and justify design at tier n

This activity requires as input the design assurance argument pattern ([U]), as well as the artefacts from the previous activities of this stage ([V], [W], [X], [Y]and [Z]). The activity uses these artefacts to create an instantiated design assurance argument for the AS ([AA]) which demonstrates that all tiers of the AS design are sufficient to ensure that the defined safety requirements can be met.

AS design assurance argument pattern

Instantiate design assurance argument pattern

The proposed design of the AS at each tier shall be analysed to identify the potential hazardous failures that could arise as a result of that design. Although an AS has been developed to satisfy the identified safety requirements, it still may be the case that the AS may be capable of doing something else, under certain conditions, that may be hazardous. Such hazardous failures can often not be identified until detail of the design solution, and in particular the characteristics of the system components is understood.

Potentially hazardous failures are identified by considering possible deviations from intended behaviour that may arise for the AS. Analysis shall be undertaken based upon the information 
regarding the design at the current tier under consideration ([VW]). The analysis shall consider information about the particular components proposed as part of that design. In particular the known limitations of the components, their known failure modes, and the conditions under which the components may fail, or under which their performance may deteriorate, shall be considered in the analysis.

The cameras chosen to be used on an autonomous passenger shuttle to determine its distance from the footpaths adjacent to the road are found to not function well in conditions that present sunlight that is both of high intensity, and at an acute angle to the road. This is a potentially hazardous failure since it may mean that the shuttle drives too close to the footpath.

Autonomous passenger shuttle 

Analysing the AS design for possible deviations must be undertaken predictively at higher levels of abstraction where specific component solutions have not been chosen. For this analysis, a technique such as HAZOP [17] that uses a set of defined guidewords applied to elements of the design in order to prompt the identification of possible deviations, can be applied. For more detailed levels of design, once the properties of the actual components used are known a more specific deviation analysis techique such as failure modes and effects analysis (FMEA) [6] can be used, perhaps in combination with Fault Tree Analysis [26] in order to understand the causal path. Safety analysis such as this forms part of traditional safety engineering processes. For AS it is important that deviations are specifically identified relating to the autonomy of the system (particularly understanding and decision making deviations).

Although a fairly traditional HAZOP analysis may be applicable to AS, particular attention must be paid to subtle deviations when interpreting the guidewords. For example, when analysing a proposed perception component, careful consideration must be given to how the following guidewords may be defined and interpreted:

- More (more than one object detected when only one is present)
- Less (less objects are detected than are actually present)
- As well as (an extra area is classified as navigable as well as the intended route)
- Part of (only part of an object is detected)
- Other than (an object is classified other than what it is)

It may also be the case that additional guidewords are required when considering AS such as:
- Intermittent (considers intermittent detection and/or classification)
- Erroneous but credible (elicits failure conditions relating to information that is incorrect
but ‘believable’)

HAZOP analysis

Having identified the possible deviations, the hazardous failures are determined by considering which of the deviations, if they occurred in the AS, could result in a hazardous outcome.

The results of the analyses, including the identified hazardous failures shall be documented in the safety analysis report ([BB]). The safety analysis report shall also provide a justification for the sufficiency of the analysis approach used, including the suitability of the approach for the particular tier under consideration and the appropriateness of any modifications made to existing 
techniques to consider autonomy related deviations.

Identify potential AS hazardous failures at tier n

Mitigations shall be defined for all the identified potential hazardous AS failures ([BB]). The mitigations could take various forms such as:

- defining required design changes
- limitations to the operating concept
- deriving additional safety requirements

Where changes are made to the AS design in order to mitigate identified hazardous failures then Activity 19 shall be repeated to ensure additional hazardous failures have not been introduced by those changes. The suitability of the design changes shall be justified as part of the design justification report ([Y]).

Limitations on the operating concept may include changes to the reduced operating domain (ROD) for the AS to provide additional constraints. The changes to the ROD shall be reflected in the safe operating concept (SOC) definition ([L]).

Any additional safety requirements that are derived shall be added to the existing safety requirements definition ([Q]) for implementation. For some of the identified potential hazardous failures it may be determined that the existing design is already sufficient to mitigate those failures (such as through redundancy in the architecture). Where this is the case, this justification shall be 
documented as part of the design justification report ([Y]).

For an autonomous robot operating in an office building, a potential hazardous failure identified from analysis of an object detection component is it may under certain conditions fail to detect walls made of translucent material. In mitigation to this, a design change is proposed to add an additional sensor of a different type.

Sensors for autonomous robot

Define mitigations for identified hazardous failures

This activity requires as input the hazardous failures argument pattern ([DD]), as well as the relevant artefacts from previous activities ([BB], [Q]and [Y]). The activity uses these artefacts to create an instantiated hazardous failures argument for the AS ([EE]) which demonstrates that the potentially hazardous failures identified at each tier are acceptably managed.

AS hazardous failures argument pattern

Instantiate hazardous failures argument pattern

In assessing the operation of an AS outside its ODM, it is important to understand the characteristics of the outside ODM environment relevant to the AS and its behaviours. Therefore this activity requires a description of the relevant key features that are anticipated in the environment outside of the ODM ([FF]). This is used to establish the scenarios that may arise due to excursions outside of the ODM.

A drone may be blown out to sea due to severe weather conditions and lose contact with its base station. It must be aware that it is now outside of its defined ODM, and that a landing on water may not be appropriate.

Drone in severe weather conditions

Those scenarios shall then be analysed to determine those which may be hazardous. There are a number of techniques that can be applied for this analysis such as Hazop [17], STAMP/STPA [28] or FRAM [13]. The analysis should involve personnel who understand the operating environments and can establish the most likely hazards for the AS if outside of the ODM. The identified hazardous scenarios due to excursions outside the ODM shall be documented in the Out of Context Analysis Report ([GG]) along with details of the analysis performed and any limitations on the analysis, e.g. assumptions regarding the environment outside the ODM.

How to use

How to use the SACE guidance