Paksen:
You are correct, S-video gives a higher quality picture than composite (RCA) video, and the digital connections are for your audio signals, not the video.
Other ideas: I had a similar situation. I don't know if yours is the same, but here's what happened to me and how I fixed it. Hope this helps.
Problem one: Although the Monitor Out section of my AV amp has both S-video and composite RCA sockets, the output signal format to the TV matches the input signal format it got from the source. In other words, my DVD signal comes into the S-video socket on the amp's "DVD input" section, and the amp sends the DVD signal on to the TV through the S-video "Monitor out" socket. So far, so good.
But when my VCR composite RCA signal goes into the RCA socket on the amp's "VCR input" section, the amp sends THAT signal on to the TV through the RCA "Monitor out" socket. It will NOT use the S-video cable that was already hooked up. RCA in = RCA out! And never the twain shall meet...
Problem two: Although each video input port on my (Pioneer) TV has both S-video and composite RCA sockets, the TV will only look at one of them at a time! I have to go into the TV setup menus and tell it (for example) that Video1 is S-Video, and any RCA signal I sent to that port afterwards is either ignored or looks like garbage. So although I can physically attach both an S-video and an RCA cable to the same port, I cannot USE both of them unless I want to go into setup and tell the port to "use the other socket" everytime I change the source.
So, my first suggestion is to leave the S-video plugged into Vid1 (you did say the DSS and DVD worked, right?), but plug the RCA cable into Vid2. Check your TV's documentation: If the ports -- as on my TV -- have to be programmed, set Vid1 to S-video (it's probably there already if your DSS/DVD look good) and set Vid2 to composite (RCA) video. If this works, it doesn't cost anything, but you will have to switch back and forth between Vid1 and Vid2 depending on your source.
My final solution cost about $30, but is simpler to operate. For around this price you should be able to pick up a small dongle that will take RCA composite input and reformat it to work with an S-video cable. So my VCR now goes from its RCA socket into the dongle using a standard RCA cable, but then out the dongle's S-video socket into the amp over an S-video cable. Since all video devices now come into the amp through S-video sockets, I only have to use a single S-video cable out from the amp to the TV, and I only have to program the one Video1 TV port to accept an S-video signal.
Note: The dongle DOES NOT actually increase the quality of the composite video signal, so it won't look any "better" than if you can get a regular composite connection to work right. It's designed for convenience only, just for the way I'm using it: to "fool" my amp into thinking that it's also an S-video feed, so that I only need one S-Video output cable to the monitor. I bought my dongle at
http://www.keene.co.uk for about 13 British Pounds plus shipping.
Good Luck! -Bruce