I'm really having trouble understanding what you're saying. In the first post, you said the receiver was recognizing a DVD player as digital, but not the DVR. Then your latest post said you tried two DVD players "and still nothing". So I'm quite confused. Is a DVD player successfully detected as Dolby Digital or not?
I will try to make some general suggestions, because I'm too confused to know exactly what's going wrong for you. Please don't take my specific step-by-steps below as a condescending type of thing. I'm really trying to help out here, but I think you might be confused on what you have (or I'm confused with what you're trying to describe). I looked up your Kenwood's manual on the manufacturer website, so I have some clue about what your receiver supports. I'm assuming your 510 DVR is similar to my 522 DVR.
First, scrap those RCA cables. Totally unplug them from both the DVR and the the receiver. I think you may just be confusing yourself with these. Next connect an S-Video cable from the DVR to "VIDEO 3" on your receiver. Not Video 1 or Video 2. These other inputs will NOT work for your setup (because Video 1 on your Kenwood does NOT have any digital audio input, and Video 2 only has a coax digital audio input, which your 510 does not support). So you MUST use Video 3. That is the ONLY input you have that supports both OPTICAL digital audio and S-Video. OK - truth be told - your Kenwood's DVD input also supports both S-Video and optical digital audio, but I assume you want to save that for your DVD player. But IF YOU WANT, you can steal the DVD inputs and hook up your 510 to them. Just remember that when you want to watch satellite you must switch your receiver to "DVD" if you do this.
So now you have two and only two cables going between your DVR and receiver. One optical digital audio, and one S-video. That's all you need, or want. They are both hooked up into the receiver's Video 3 inputs.
Now set up your receiver to "auto detect" digital audio signals. This should be in the manual somewhere. It is surely the default, and if you just turn off the receiver and then turn it back on again I'm willing to bet that it will come up in auto-detect mode on it's own accord. I am not sure if you have to do anything on your 510 DVR to enable digital audio out. On most other components I've run into, digital audio out is enabled for Dolby Digital by default. However, it is NOT enabled by default for DTS, since sending a DTS signal to a receiver that can't handle it will result in a hellish squeal that may fry your speakers. But we're not concerned about DTS here, your 510 does not have it AFAIK. I mention it only in case you wonder about DTS not working from a DVD player for example. You probably have to manually enable DTS on the DVD player. That's fine for your Kenwood receiver. It can handle DTS. So no hellish squeals for you! Now back to the topic at hand (DD)...
Tune your receiver to Video 3 where you plugged in your 510. Now find something on satellite that's being broadcast in DD. That may not be an easy task, since DD is rare on Dish from my experience. I assume there's some front panel display on your Kenwood that would indicate that it's receiving and decoding DD. Maybe is says "Dolby Digital" or maybe it has some little icon that looks like a group of 5 or six speakers. Maybe just an LED indicator light.
If you can't find anything on satellite being broadcast in DD, complete your testing with a DVD player instead. Most any recent DVD release you can rent will have DD. Unplug the two cables from the back of your 510 (leaving the other ends plugged in to the Video 3 on your Kenwood). Plug those cables into the back of a DVD player and fire up a movie. Note that most DVD's do not use DD on the startup menus. You have to actually get into the meat of the movie before you'll see DD decoding indicated on your Kenwood. Some DVDs require you to turn on DD. Usually in a Setup or Languages menu. But many (most) have DD enabled by default. Note that enabling DD on a particular DVD (disk) is seperate and distinct from enabling DD on your DVD player itself, which I touched on above. I would recommend playing "Saving Private Ryan" on the DVD player. When you watch that first beach assault scene, you'll KNOW you're hearing digital audio ... because you'll be ducking to avoid the bullets seemingly flying around you!
If all this still does not work for you, I'd start tending toward agreeing that your receiver might be hosed. But not until you've positively yanked those RCA cables out of the picture. You will NEVER get any digital signal over those analog RCA outputs. The best you can hope for with an analog connection is ProLogic (Left, Right, Center, Rear). You can only get 5.1 (LeftFront, RightFront, Center, LeftRear, RightRear, and Subwoofer) from a digial connection (optical or coax). Your receiver may take ProLogic and simulate some fancy sounding "surround sound" thing, but it's incorrect to call this "5.1" ... it is not 5.1 Unplug those RCA cables so you don't confuse yourself about this simulation.
Good luck!