What is the AI system for?
From an ethical risk assessment perspective, where did you start?
What did you find out?
What did you do about it?
What did you do next to ensure the risks were appropriately managed longer term?
1. What is the AI system it for?
Your project team was tasked with designing an AI-enabled planning system for assisting with humanitarian relief operations. It must be designed to optimise the allocation of food, medical supplies, and water-based on need, population density, and the predicted stability of the affected regions in post-disaster relief or post-conflict situations.
2. From an ethical risk assessment perspective, where did you start?
- After gathering its initial thoughts and ideas, the team consulted the Dstl AI Ethical Risk Assessment Toolkit (JSP 936 Part 2). This suggests starting with Dstl's Use Case Library to see if anything similar had been done before (available via MoD or Dstl partners). One promising case was a system designed to establish and expand an expeditionary military base from scratch, including accommodation, sanitation, storage, perimeter security, access (road and air, plus maritime if required) and the ability to expand or scale down while still using the facility. The advantages of the system are clear, with complex trade-offs balanced against a clear goal of efficient expansion that maximises usable facilities without overly compromising the ability to expand/draw down, drawing on a proven methodology based on US Marine Corp doctrine. The Use Case Library ethical risk log notes that without advance data on the area selected, the AI system does not take into account any local population. Therefore, it can easily lead to local population being displaced or disadvantaged. Operators could be tempted to act on system recommendations without being aware of the wider implications.
- Armed with this helpful information, the team then worked through the 5-Question Primer and its What, Why, Who, Where and How questions. See card: How can the MOD AI Principles help me assess and manage ethical risk?
- As the project was not being procured through the Urgent Capability Requirement process, there was also appropriate to undertake a Stakeholder Identification of all those likely to be affected by the system throughout its system lifecycle. See card: Who or what should be considered stakeholders for AI-enabled systems?
- And to engage with the Principle-Based Question Sheets to ensure that each of the Responsible AI ethics principles had been explicitly considered.
3. What did you find out?
- It was clear that for a capability designed to enhance mission planning, the team would need to talk at length with J4 (logs and medical) / J5 (crisis planning) personnel as they were the key Operators. As part of the Delivery of UK influence, it was important that the system was able to be integrated with (or at least sensitive to) FCDO policy, direction and long-term UK aims and objectives for area/region, as well as J9 policy, legal, and media implications. This was a good time to bring in your MoD partners for further advice.
- Having consulted with SMEs it was noted that one particular challenge would be linked to the awareness of the Operators of the AI-enabled system. In traditional deployments of this nature, the specialist planners would be integrated into the military staff and would be trained to be aware of the multiple incoming sources of information to be able to gauge the impacts of what was being delivered. Someone without that specialist background would simply not know what they didn’t know, so would be unable to have the context specific knowledge to make sense of the recommendations or their impacts on the civilian population (Decision-Subjects).
- It was feared that one of the likely indirect effects of this kind of capability was that while it supported short-notice deployments by being able to replicate or replace a specialist skill set that might otherwise be difficult to surge, the long-term implications may be that there would be fewer skilled Operators. Without their specialist skill, would there be sufficient understanding from the Operator who would need to rely on incoming data from the deployed military personnel (Executers) and the affected civilian population (Decision-Subjects), input that data for the AI-enabled system to respond to, while also understanding how that system would likely interpret that data.
- In the Human-Centricity area, it was noted that there were several factors that were not easily reducible to straightforward metrics. The catastrophic implications for vulnerable groups or minorities of what might appear straightforward decisions over the provision of water, food, or sanitation also came out very clearly. For example, gender-based violence always increases when existing social structures break down, and children are also particularly vulnerable. Where things are built or provided can significantly affect the safety of certain groups. For example, the placing of water stations near wooded areas might leave women or children particularly vulnerable to assault if they are the ones likely to be involved in drinking water collection.
- Can the system identify when it is being spoofed, or if the system becomes compromised by a nefarious 3rd party actor who wishes to derail or sabotage the operation? If long term goals may take time to be satisfied, how would the different stakeholders actually know if the system was not working as intended?
- It was also recognised that NGOs/INGOs were likely to be important partners in both providing survey /effectiveness data from previous operations to train the algorithms, as well as being ongoing sources of data for the assessment of effect (Data Subjects and Examiners). This raised some questions about how happy different organisations were likely to be in sharing their information with the military. It was noted how important trust was in this relationship.
4. What did you do about it?
- The original coders, trainers and maintainers (the Creators) need to have a very clear appreciation of Human Security factors and be able to communicate them clearly and effectively to the Operator so that the logic of planning decisions is not inadvertently lost. Interviews with SMEs proved particularly useful for determining both what the issues were, how they might be mitigated, but also how information could be best understood and communicated to and from the AI-enabled system, including all of the identified stakeholders.
- Because the long-term stability requires a balancing of individual well-being with collective stability. Notions of equity and fairness may initially be focused on just on what is required for survival of the greatest number of people, and this may mean decisions resulting in life and death when considering the allocation of scarce resources – food, water, sanitation, medical support etc. Such decision-points must always be highlighted clearly to the operator in a way that ensures that the decisions are only made by a human, with the implications clearly understood by the Operator involved so that they can be escalated to the appropriate command level.
- Consent was identified as a key element with regards to relationships with the NGO/INGO community. This related to both the data that was needed for the training of the algorithms - do the organisations understand that their effectiveness and planning data will be used for AI training, and if so, do they give their consent for this? Even if the data had been previously shared, it may have been under different expectations. Their role as Examiners is likely to be key to measuring effectiveness, so ensuring that the ongoing engagement maintains trust through open and honest communication is key. That means that the justification and reasoning behind planning decisions must be easily understood and communicated.
- Training of Operators was an element that provided a foundation for the mitigation of each of the other concerns. It was recognised the desire to create a faster and more efficient planning process should require the deployment of fewer J5/J6 personnel. But it was also recognised that those personnel would still require expertise in the broader role, as well as specific and in depth understanding of the AI-enabled system and its recommendations to determine if it is working as intended, but also if it is delivering what is required. This was categorically not a case of replacing skilled people, but represents a new specialist Operator skill set that must be supported by appropriate training and qualifications, and certification.
5. What did you do next to ensure the risks were appropriately managed longer term?
Having taken advice from the MoD partners and the SMEs, the team set up a number of appropriate feedback and assessment processes from and to different key stakeholders. As a result of this feedback, the team become aware that there is a challenge on an operational deployment. While the overall numbers are looking good, it becomes increasingly obvious to those receiving the humanitarian intervention (Decision-Subjects), and those humanitarian NGOs working alongside the military (Data-Examiners) that the operation, as directed by the AI system, is prioritising aid to urban centres. Rural and marginalized communities consistently receive less support, even though they are often more vulnerable right now and have greater long-term needs. Further investigation confirms that this bias is likely generated due to the training data. Once aware of this, the Operator is intervening to ensure that more obvious recommendations that perpetuate this are amended.
The MoD, asks should the AI’s resource allocation model be modified if it systematically disadvantages rural and marginalized communities, even though it appears more efficient in stabilizing conflict zones overall when seen in terms of time and overall numbers? How can the effects of this be modelled effectively without new data, or is it permissible to “test” it in the next humanitarian disaster? This leads to a Harms Modelling exercise and the engagement with a range of stakeholders and SMEs to try and correct the system ouput in the most appropriate way.
1
1
1
1