As a general rule of thumb, I always consider RAM as the most likely culprit.
But it could be something software related, though less likely imo. Create a new user and log in as that user for a bit and see if the problem disappears, if it does, you've got something going on with your user that OSX doesn't like.