So we let AI systems plead the 5th? Hmmm. IMO we can’t just talk about a narrow explanation from an ML system, agreed that that will not clarify. But rather we should look at explanations from a systems based perspective, how else could we establish any (however imperfect) narrative of causality? But this also reminds me that I need to write more distinguishing retrospective vs prospective accountability, because prospective accountability doesn’t rely so heavily on explanation…more to come.
Sort of but not for the reasons a human is allowed to. An LLM can neither incriminate nor exonerate itself because nothing it says has any bearing on why it did what it did. Any response an LLM generates to "why did you conclude that" is probabilistically derived from the training data and the inputs and outputs in the current context It is not derived from a memory trace of internal thought process. This is inherent to how LLMs work as opposed to other models of AI.
Thanks for expanding on this. I agree we shouldn't rely on LLM-based explanations. The explanation that might support accountability needs to be produced by the people in the AI system making sense of how the system as a whole produced a particular effect.
Very nice broad summary of key issues. I do wonder though why we need to grapple with the question of how can AI systems explain and justify their behavior. Explanations for decisions and predictions are as suspect coming from an AI system based on machine learning models as they are when given by people. Such introspections are at best guesswork, even if we exclude prevarication. Someone's account of why they did something has ultimately had little value when constructing the causal chains of accountability. Why should it be different for AI?
So we let AI systems plead the 5th? Hmmm. IMO we can’t just talk about a narrow explanation from an ML system, agreed that that will not clarify. But rather we should look at explanations from a systems based perspective, how else could we establish any (however imperfect) narrative of causality? But this also reminds me that I need to write more distinguishing retrospective vs prospective accountability, because prospective accountability doesn’t rely so heavily on explanation…more to come.
Sort of but not for the reasons a human is allowed to. An LLM can neither incriminate nor exonerate itself because nothing it says has any bearing on why it did what it did. Any response an LLM generates to "why did you conclude that" is probabilistically derived from the training data and the inputs and outputs in the current context It is not derived from a memory trace of internal thought process. This is inherent to how LLMs work as opposed to other models of AI.
Thanks for expanding on this. I agree we shouldn't rely on LLM-based explanations. The explanation that might support accountability needs to be produced by the people in the AI system making sense of how the system as a whole produced a particular effect.
Very nice broad summary of key issues. I do wonder though why we need to grapple with the question of how can AI systems explain and justify their behavior. Explanations for decisions and predictions are as suspect coming from an AI system based on machine learning models as they are when given by people. Such introspections are at best guesswork, even if we exclude prevarication. Someone's account of why they did something has ultimately had little value when constructing the causal chains of accountability. Why should it be different for AI?