Our preprint: Orthogonal Self-Attention is out now.