He's worried that he can't celebrate at the pub and no one will see his hat