Audio issues, syncing, not getting the top quality available, etc are the main issues. Up-conversion is also been beat by the other 2 devices, so they produce a better image. Codec support and Plex for people streaming their library. Plus the gaming that a smaller subset of people even use. A lot of the stuff would not be considered consumer usage though.
If Apple comes out with a new version, I'm sure they will take the top again (unless Nvidia also releases an update but that's on hold until after the Switch release if ever), but right now they aren't worth it unless you just want to stay with the Apple UI. As purely a consumer device it is way over priced for a UI that you like.
Yeah, it isn't perfect, but it is definitely the most performant hardware platform in my experience. The whole UI just operates very quickly compared to everything else out there. The only audio issue I see is the way it prevents you from just passing the original audio through, and that is a big one for a lot of people, so I get it. It does it for video, as well as any others, but why not audio? I never see any audio sync problems like I used to on the FireStick though. As for video upscaling, I found my Sony TV from 2020 did a better job than the Shield's "AI" upscaling, especially in dark, noisy scenes, so I just passed through the native video and let the TV take care of it, and I do the same on the AppleTV. My other TVs are old or small enough not to matter.
I abandoned Plex because of their crappy software, but if you want an AppleTV to work correctly with different video codecs, Infuse solves that very well. Jellyfin is my media server of choice these days FWIW.
I actually switched from NVIDIA Shields to ATV4K because of a number of buggy behaviors, mostly due to audio over HDMI. I do miss the flexibility of the platform, but I am not the only user in the house, and the AppleTV "just works" for them in a way that Google, Amazon, and Roku products did not.
I have my parents using Roku, and that is what I'd recommend to anyone who came to me asking what they should get, unless I knew they were big Apple or Google fans.