Channel Attention

Exploring attention design: why simpler cross-channel methods outperform SE

The Efficient Channel Attention paper made a compelling case that cross-channel interaction drives attention gains, but the evidence tells a more complicated story. ECA skips SE's dimensionality reduction and applies a 1D convolution directly to channel means. It works, clearly beating SE on chess tablebases. The catch: a kernel size of one, which removes any cross-channel interaction, performs just as well. That undermines the paper's central claim. The authors tuned k exhaustively yet never tested the degenerate case that would have challenged their hypothesis.