Why is it wrong to harm animals?

Several philosophers and religious figures believe/d that it is wrong to harm/abuse animals because this type of behaviour can lead to these people abusing humans. They deny that harming animals is an intrinsic harm to animals.

This is a strange argument because firstly, surely any pain or suffering is a negative experience and intrinsically evil in itself, especially when inflicted unnecessarily and cruelly?

Secondly, if animals were inanimate objects who didn't really have feelings or whose feelings didn't matter, why would harming them automatically lead to the abuse of humans? After all to believe this, we must believe that the human inflicting the abuse must be a cruel person and behaving cruelly. However, one cannot behave cruelly without a subject who can experience that cruelty in a negative way, as a bad experience. Otherwise the abusive person can not really have any ill-intent.

To illustrate, if I find it a great stress release to smash a tennis ball against a wall, or crack rocks with a pick-axe, noone would say I am being cruel and that I would necessarily transfer my frustration onto people. That's because it isn't cruel to hit, smack, smash, splinter an inanimate object. Cruelty can only be associated with abuse inflicted on a being that can suffer. Suffering in itself is a cruel and horrible experience. Surely, any moral system will have as a fundamental premise that unnessary suffering is a bad thing and should be prevented, alleviated or reduced as much as possible?