How a model knows word order at all

Attention alone can't tell "dog bites man" from "man bites dog" — positional encoding is the patch that fixes this, and it's stranger and more elegant than you'd expect.