OK, first with your old receiver. The only thing it will handle is basic Dolby Digital 5.1, or analog 5.1. Your BD player doesn't have 5.1 analog outputs, so the only thing you can work with is Dolby Digital.

The re-encode option on your player encodes to DTS, which is why you won't hear anything. When you play a DTS-HD track, what you would get over coaxial or optical outputs is the "core" 1.5mbit DTS track, which as mentioned your receiver doesn't support. Your only option to hear anything would be to change to PCM output, which should produce stereo sound (but not 5.1).

When you play Dolby TrueHD tracks, you get the core 640kbit Dolby Digital 5.1 track over coaxial/optical, which your receiver does support.

Now looking at the STR-DG520. That receiver has HDMI inputs, but they are switching only - you can't get any sound from them. You WOULD be able to hear DTS and Dolby Digital 5.1 tracks over the coaxial/optical inputs, but you WOULD NOT be hearing the actual uncompressed DTS-HD MA or TrueHD tracks, but rather still DVD quality sound. So I don't think I'd really spend the money to upgrade. It isn't a good match for BD.

The STR-DG720 is slightly better. It can decode 7.1 LPCM over HDMI, but the problem is your BD player will not internally decode DTS-HD MA to LPCM. So it isn't a good match for your player either, but you would hear uncompressed Dolby TrueHD or from discs mastered in LPCM originally.

Really, the first Sony receiver to do what you need is the STR-DG1200. You can also look at the Pioneer VSX-1018AH-K, Denon AVR-789 or Yamaha HTR-6160 or RX-V663, but they're all about the same price (though you should be able to get them all for less than list price).