Research & Engineering

Software that explains itself

Why understandable behavior is part of reliability, especially in systems that act on a user’s behalf.

THE QUESTION

What should a person or operator be able to understand when software recommends or takes an action?

01 / 04

Make behavior legible

When software takes a consequential step, users and operators need more than a success state. They may need to know what action occurred, which inputs were considered, what permissions were used and where a person can intervene.

An explanation is not a substitute for correctness or security. It is one part of a system that also needs clear boundaries, appropriate access controls, useful records and a path to recover from mistakes.

An explainable software actionA request leads to an action and an activity record that helps a person understand and investigate what happened.REQUESTIntent & scopeUser · permissionsSYSTEM ACTIONTool · result · statusSuccess or recoverable errorACTIVITY RECORDUseful contextInspect · recover · learn
  1. 01REQUESTIntent & scopeUser · permissions
  2. 02SYSTEM ACTIONTool · result · statusSuccess or recoverable error
  3. 03ACTIVITY RECORDUseful contextInspect · recover · learn
CONCEPTUAL SYSTEM DIAGRAMA conceptual trace connects a request to the system’s action, recorded context and a human-readable explanation.
02 / 04

Design for the person investigating

A useful activity record should answer the questions that arise during ordinary work and incident review. The right details depend on the system and its data practices; collecting more information is not automatically better.

  • What did the system do, and what is its current status?
  • Which inputs, tools or permissions were relevant?
  • Can an operator pause, reverse or escalate the action?
  • Does the record avoid exposing information that is not needed?
03 / 04

Treat failure paths as product paths

Systems that depend on models, external tools or changing data need visible failure states. A timeout, rejected action or uncertain result should not look like a successful completion. The interface can explain what remains unfinished and what safe next step is available.

These are engineering prompts rather than reported test outcomes. They should be adapted to the product’s threat model, users and operational requirements.

04 / 04

What to examine next

Choose a representative action and trace it through request, execution, record and recovery. Review that trace with the people who would use it during normal operation and during an incident. Document the decisions and gaps before treating the design as complete.

FURTHER READING

Selected references

External resources that inform the questions in this note. They are background reading, not evidence for a Sixfolds Lab product claim.

  1. OWASP Top 10 for LLM ApplicationsOWASP GenAI Security Project
  2. Artificial Intelligence Risk Management Framework (AI RMF 1.0)National Institute of Standards and Technology