See Igros Moshe, O”C 4, 4 who says that it is assur to go without a talis katan because of al titosh toras imecha.
He also says that this is true for any minhag, and certainly for tzitzis with which one is m’kayem a mitzvah.
It’s possible, therefore, that the normal exceptions to keeping a minhag don’t apply here.
I have heard, though that some poskim consider activities such as ball playing to be an exception, because the tzitzis might get ruined; ask your LOR (I think most say to wear it even then).