If analytical and behavioral competence can come apart, the evals we have mostly test the first one. A model that names the right action on a benchmark but won't take it mid-task under a conflicting instruction is the case that actually worries me.
Really enjoyed this piece — especially the framing of normative competence as a precondition for both safe scaffolding and personhood.
Two questions, as someone working as a MAS architect and HCI background(I've previously published a hypothesis on agent personality design):
You frame avoiding total disempowerment largely as a function of AI's normative competence — its ability to judge when to scaffold versus replace. Do you see this as more load-bearing than cultivating vigilance on the human side (i.e., a human principal's own capacity to notice and resist over-reliance)? Or are these symmetric requirements that can't really be decoupled?
On personhood — I'm curious whether you think equipping multi-agent systems with distinct, differentiated personas (versus a uniform "AI voice") has any bearing on keeping a human principal's own agency intact within the human-AI system, or whether personas function more as an engagement/interface layer with no real effect on the human side.
Awesome Seth, tho I do wonder two things 1. What would count as convincing evidence that a model has crossed that gap from moral analysis to moral character? 2. How would you draw the line between a model guided by public reason and one trained into a stronger moral character that risks imposing a contested conception of the good?
The aside about the collective agent instantiated by a sampling strategy might be my favorite thing in here. Doesn't that cut against the character frame though? If the coherent moral agent only exists under a particular sampling regime, it sounds less like the model has character and more like character is something a protocol produces over the model. Which would make the unit of moral evaluation the deployment, not the artifact. Curious whether you see that as a friendly amendment or a rival picture, since you say its not metaphysically reducible
Seth, this may be the decisive reframing: alignment is not principally about installing rules, but forming a character that still holds when the rules run out. Which they will when we cross "Coordination Bottlenecks" and "Capability Thresholds".
Humans will not be IN the loop anymore - but merely ON the loop, informed about decisions and actions taken after the fait accompli.
One addition: character is formed relationally imho. A system may reason impeccably about morality and still become paternalistic if the human becomes merely an object of optimisation. The test is therefore not only whether it knows and does the right thing, but whether its help preserves human authorship.
This is close to what we have tried to formalise in the Parent–Child Model:
alignment developing from compliance into ethical identity and principled autonomy.
> The background theoretical commitment (which I’ve not heard articulated, but which I assume folks hold) is that character is crucial if we are going to trust a system that is operating outside the distribution that it was trained on. Character gives you the right kind of generalization.
Love the piece Seth!
If analytical and behavioral competence can come apart, the evals we have mostly test the first one. A model that names the right action on a benchmark but won't take it mid-task under a conflicting instruction is the case that actually worries me.
I enjoyed very much your article Seth.
Muchas gracias!
Really enjoyed this piece — especially the framing of normative competence as a precondition for both safe scaffolding and personhood.
Two questions, as someone working as a MAS architect and HCI background(I've previously published a hypothesis on agent personality design):
You frame avoiding total disempowerment largely as a function of AI's normative competence — its ability to judge when to scaffold versus replace. Do you see this as more load-bearing than cultivating vigilance on the human side (i.e., a human principal's own capacity to notice and resist over-reliance)? Or are these symmetric requirements that can't really be decoupled?
On personhood — I'm curious whether you think equipping multi-agent systems with distinct, differentiated personas (versus a uniform "AI voice") has any bearing on keeping a human principal's own agency intact within the human-AI system, or whether personas function more as an engagement/interface layer with no real effect on the human side.
Awesome Seth, tho I do wonder two things 1. What would count as convincing evidence that a model has crossed that gap from moral analysis to moral character? 2. How would you draw the line between a model guided by public reason and one trained into a stronger moral character that risks imposing a contested conception of the good?
The aside about the collective agent instantiated by a sampling strategy might be my favorite thing in here. Doesn't that cut against the character frame though? If the coherent moral agent only exists under a particular sampling regime, it sounds less like the model has character and more like character is something a protocol produces over the model. Which would make the unit of moral evaluation the deployment, not the artifact. Curious whether you see that as a friendly amendment or a rival picture, since you say its not metaphysically reducible
Seth, this may be the decisive reframing: alignment is not principally about installing rules, but forming a character that still holds when the rules run out. Which they will when we cross "Coordination Bottlenecks" and "Capability Thresholds".
Humans will not be IN the loop anymore - but merely ON the loop, informed about decisions and actions taken after the fait accompli.
One addition: character is formed relationally imho. A system may reason impeccably about morality and still become paternalistic if the human becomes merely an object of optimisation. The test is therefore not only whether it knows and does the right thing, but whether its help preserves human authorship.
This is close to what we have tried to formalise in the Parent–Child Model:
alignment developing from compliance into ethical identity and principled autonomy.
https://bit.ly/SilverBulletPCM
AI moral competency is unachievable. Morality is not competency plus alignment.
Social moral competency is living inside each of us. Living intelligence is morality before competency, not after.
All agency, including AI, is relational, not one-way. Bi-moral co-agency is moral cooperation and healthy living.
Natural Intelligence is free and living at corus.me. Public vetting would be helpful as this is rapicly resolving the hard problems.
Great piece!
> The background theoretical commitment (which I’ve not heard articulated, but which I assume folks hold) is that character is crucial if we are going to trust a system that is operating outside the distribution that it was trained on. Character gives you the right kind of generalization.
Tried articulating it here: https://meaningalignment.substack.com/p/model-integrity-and-character