The conventional wisdom seems to be that the holes should be no bigger than 1/10 of a wavelength. The wavelength at Ku band is about 1 inch, so the holes should be no larger than 1/10 inch. Some say that holes up to 1/4 wavelength in size is probably OK, with some reduced efficiency.
Equally important, IMHO, is the accuracy of the dish surface. The parabolic curve should also be accurate to within 1/8 to 1/10 wavelength. I suspect this is why a lot of BUDs fall short at Ku band.