Filed under:
Responsibility
Understanding

 

1. What is it for?
An AI system is sought to assist military commanders by recommending strategic and tactical decisions based on battlefield data. The system would free up multiple staff officer positions, and by being able to process all incoming data at speed, provide faster recommendations based on operational need and commander’s intent. In addition to increasing the speed and accuracy of decision-making, this will also significantly reduce the logistic and security footprint of a forward deployed HQ.
 

 

2. From an ethical risk assessment perspective, where did you start?
The Dstl AI Ethical Risk Assessment Toolkit (JSP 936 Part 2) suggests starting with Dstl's Use Case Library (available via MoD or Dstl partners) which offers examples of a variety of decision-assistance tools for logistical, human resource and planning decisions, and some common ethical challenges that appear across these. The 5-Question Primer and its What, Why, Who, Where and How questions expanded those considerations, demonstrating that the nature of many of the decisions that might be taken here were related to life-and-death, both of friendly and enemy forces, as well as the civilian population. Stakeholders likely to interact with, or be affected by, the system throughout its system lifecycle were identified. Finally, the team considered the Principle-Based Question Sheets to ensure that each of the Responsible AI ethics principles had been explicitly considered. See card: How can the MOD AI Principles help me assess and manage ethical risk? and, Who or what should be considered stakeholders for AI-enabled systems?.
 

 

3. What did you find out?
By developing and exploring both best- and worst-case scenarios, a number of significant ethical challenges were identified that might arise in certain cases. One such scenario would be a time-sensitive military operation or strike being launched by a commander on the system’s recommendation informed by flawed data or analysis, leading to excessive casualties combined with a failure to achieve the military objective. In such a situation, who holds responsibility for the civilian casualties— the commanding officer for approving the AI’s decision, or the AI developers for the system's miscalculation? How should accountability be divided when an AI is integrated into command decisions?
 

 

4. What did you do about it?
What safeguards could best ensure the system worked as required? In the worst-case scenario, the AI system's recommendation was based on flawed data about civilian movements, leading to a tragic outcome. This reveals a failure not only in the system's predictive capabilities but also in the design of its human-machine interaction. As a developer, responsibility extends beyond creating algorithms to ensuring the system actively supports human judgement, rather than replacing it. For example, the system should include mechanisms that highlight uncertainties, flag ethical concerns, and provide a clear rationale for its recommendations. If the AI had emphasised the limitations of its civilian movement data or required additional validation steps before recommending an airstrike, the commanding officer might have reconsidered or sought additional intelligence.
Clearly, the best case would be that the AI was sophisticated enough not to make such errors, so to prevent such situations, developers must implement robust testing processes, and work from diverse and genuinely representative training datasets. Not to do so could potentially leave them ethically and perhaps legally liable for preventable mistakes. Additionally, it is crucial to design systems that enhance human decision-making rather than rush it. The intense time pressure described in this scenario highlights the need for systems that encourage commanders to pause and deliberate where possible, even in urgent situations. For example, the AI could be programmed to present alternative courses of action, explicitly quantify risks, and provide a "human-in-the-loop" safeguard for all decisions involving high civilian risk. See card: What does having a person “in the loop” actually mean?.
Accountability in this scenario is a shared responsibility. The developer bears responsibility for the accuracy and ethical integrity of the AI system, but the military organisation deploying it must also ensure proper training for operators, establish guidelines for its use, and foster a culture that prioritises critical evaluation of AI recommendations. Commanders, as decision-makers, must remain ultimately accountable for their actions, but they should be equipped with tools that enable informed and ethical decision-making.
 

 

5. What did you do next to ensure the risks were appropriately managed longer term?
Battlefield AI must be rigorously validated to ensure it performs reliably under diverse and unpredictable conditions. This means building a long-term picture to demonstrate appropriate robustness. In partnership with key stakeholders, including independent assessor (Examiners) a robust monitoring and oversight process is implemented to ensure that the system operates as required. This must be combined with continuous auditing of the AI’s performance in realistic and ethically sensitive scenarios.