I like the GH guitar better, it feels more reliable to me, but Rock Band is such a better experience, that we are skipping this iteration of Guitar Hero.
Rock Band has had weekly downloads that for the most part have been worth it, and are less expensive and provide more options than the GH packs (you can download single songs if you want instead of being forced to buy an entire 3-pack of songs for the 1 you want).
Biggest issue with the Activision feud is until they start indiivdual instrument sales (Feb I believe is the current ETA) you won't be able to play 4 players unless a friend brings another RB guitar over, or you buy a 2nd RB pack.
Not that I'd advocate buying a 2nd RB pack, but it wouldn't be a total waste, you can play vs a 2nd drum set on the same TV for example, but I believe the most you can play together coop on 1 song is Mic + Guitar + Bass + Drums.
Rock Band is the clear choice this round. For guitar, GH3 for the most part is just faster, with the same difficulty stepping, but a wider window to hit a note so it actually got easier for some people. With Rock Band, the fret speeds are slower, and a little easier, making the move to harder difficulties easier (to my friends anyways).
My experiences are all based on the 360 btw, but so far the experiences are identical from what I've read.