Great Tzaddikim would be very careful not to speak when wearing even one Tefilla. Whether it’s truly assur or not, speak to your L.O.R. It seems from your question that you are taking the Tefillin off at the end of Shachris and having just taken off the Shel Rosh, you need to say something, your Tefilla Shel Yad is still on. If so, my eitza (check with your Rav!) would be to tuck your shirt underneath the Bayit (Shel Yad) so that now there is a Chatzitza between the Bayit and the arm, which technically means you are not wearing it…
