The Mishmeres Shalom Siman 12:4 explains that it is only for those who learned from that Tzaddik and had a connection to him. He derives it from the Rash”i In Yevamos 122B who uses the word “Regel” to describe a yartzeit;
???? ????? – ???? ?????? ????? ?????? ????? ???? ????? ??? ???? ????? ????? ???. ??????? ??????? ????? ?? ??? ????? ??????? ????? ??? ??? ?? ??? ???? ?????? ???? ?????? ???? ??? ???? ?????? ???? ??? ??????? ?????? ????? ??? ?????? ????? ?? ???? ?? ??? ??? ?????? ????? ??
The Ketzos Hashulchan 1:24:5 says that not saying Tachanun is all based on Minhag. In the Badei Hashulchan 24:19 he mentions a minhag in Egypt to not say tachanun if there is a Bar Mitzvah, and in Poland if there is Yartzeit of a Tzadik.