Happy, do this:
Disconnect any speaker wires from the sub, either in or out. Connect your main speakers to the "A" terminals, period. Run them with the speaker selector on "A" output, period.
It sounds like you had the mains on "A" and the sub on "B", and the sub's input crossover was causing oscillations. There should be nothing connected to the "B" terminals at all.
The fact that the sub has RCA inputs means it it a powered sub, so plug it in. If it has an input level control, set it at a low level, about 9 o'clock (assuming the usual 7-to-5 range).
Get an RCA "Y" cable, one that has two female jacks to one male plug. If you have a stereo cable to reach from the receiver to the sub, plug the "Y" cable into the receiver 'sub out', and run the cable from the "Y" to the sub inputs. If you have a mono cable, use the "Y" at the sub end to connect the cable to the sub's inputs.
Start with the volume low, including the sub output level if there is one. Use the DD decoding. Play a DVD with a decent sound track. If you don't have a good start at this point, let us know.
Larry
www.fineelectricco.com