The minhag of the Gaon was not to wear tefillin chol hamoed. This has become the minhag of all of Eretz Yisroel. Even so, if you feel you must, do it betzin’a and don’t talk about it. No matter what you do, yesh al mi lismoch, see above posts. After all, it is difficult to abandon the Rama.
