As one of the above posters mentioned I think there needs to be a definition of what a “friend” is, and what being friends with a person entails.
Each situation must be handled differently and each person should have a Rav or Rebitzen whom he/she can go to in order to seek out advice.
However, if I may be so bold as to state my opinion. Unless it is going to be clearly detrimental to a person there is no reason to break off an interpersonal relationship. I feel that Baalei Tshuva and FFB alike need to strengthen themselves in emunah to the point that we are not afraid to be friends with people because of their “influence”. We have the light and therefore should be able to push away any darkness that they bring.
One of the things the world today lacks is people talking to people. Human interaction has been reduced to texts and tweets and anonymous blog posts like this. We have lost the ability to connect properly. To break one of the last remnants because of hashkafic differences is insane. Now of course I am not saying to go to a party, or be mchalel shabbos h”v to maintain a friendship. Each person has the ability to control an aspect of the relationship.
OP if this is what your rebbeim have told you than I am not one to argue. And especially if you are single it is even more difficult. But please think about the benefits for yourself, for your friends, and for all of klal yisroel.