I don't think you can't be warden because no one can hear you. I think you can't be warden because you didn't give any orders before opening the cells. If that only happened once, I wouldn't have any problem, but when it happens 3 times already, that's when it becomes a problem.
For goodness'...