You just want to make sure you dont short the output of any amplifier - either the headphone output or the amp output.
You never feed the output of a audio power amp into a pot unless it's an audio attenuator rated for that particular purpose.
Make sure pot wiper goes to input of amp, not the h/p socket. Otherwise when you turn it down you will effectively short h/p socket.
I am not sure of the wiring on the 4500 but it should work ok as long as the load you put on the h/p output isnt too high (low resistance). If you stick with something above 1k you should be ok. Would also pay to check what the amp input specs are. You may have to put a resistor in series to limit input. Some also require some form of input coupling ie capacitors.