It is a fair question because people are judging who is the best simply on offense. If offense is that important, then it would not matter as much which player you take to play the outfield. Keep in mind that Cabrera did play the outfield for a couple of years.
I would pose the same question for the infield. Which would you take to play 1B, Cabrera, Kemp, or Trout? I am sure all 3 can play a half way decent 1B. You would be wasting both Trout's and Kemp's arm and speed by playing them at 1B. Even with that said, I would still take Kemp and Trout at 1B over Cabrera because they are both still better all around players