In my experience with the prisoners, they don't attack you on purpose. They only attack what's in front of them if they can't proceed forward to where they currently want to go. This includes urns, coffins, walls, other prisoners, etc. When there is a crowd of them trying to worship you, not all can get close enough, which will lead to them running in place and attacking what's in front of them. And unfortunately, they can't smack the prisoner they intend to, instead the prisoners in the way are pushed a little aside and the attack hits you. If they want to for some reason get around you and they can't fit in between you and the wall, they'll hit what's in front of them, usually you included.
I have had a moral dilemma myself, deciding whether or not to kill them (I like to take games seriously). They prove to be very inhibitory when you're trying to kill the Octopus-headed guards.
There was one scenario where they helped me, though: I jogged past an open cell full of them to get a good hiding spot to back stab the Octopus-headed guard, I ran to the guard, back stabbing it, not killing it in one hit (new Royal using her beginning rapier). When it got up, it shot the shock orb that holds you in place and hit me with it. But the worshipers blocked it from getting to me. Pretty cool moment :D
Just thought I'd post in Discussion section first to be courteous, if I'm wrong somehow :P