Good feedback. The intent is to prevent hot swapping it in combat, so if I used it again I’d probably write something explicit like the benefits require attunement and wearing the hat, and the penalties persist while attuned.