The Rambam wrote about a talmid chacham wrapping a turban and making a bracha upon wrapping a turban. Rabbis don’t wear turbans anymore; perhaps the black fedora is seen as a post-Godfather version of a turban?
We live in a world that sadly emphasizes the chitzonius over pnimiyus. People look at someone in a kippah serugah, tzitzit tucked inside the trousers, and a polo shirt and khakis, versus someone in a black Borsalino, white shirt, and black jacket and pants, and they automatically assume that the guy in khakis and a kippah serugah “isn’t really frum,” despite the fact that he could possibly be more religious than the guy in the levush.