It’s not just the milk that’s an issue, but many other products that contain milk. The cholov yisroel or pareve versions of these products are not widely available and are much more expensive, especially because they are rarely available on sale.
My daughter has a food intolerance which severely limits her diet. Many of the products she can eat are dairy cholov stam. I would not restrict her diet futher by permitting her to eat only cholov yisroel. And given that food allergies and intolerances are increasingly common, I’m sure I’m not the only parent with this particular concern.