What should I do? Report it or talk to his father about it?

I would talk to your sons father first alone.Let him know that you have thought about filing a report. Then wait, if things start to get worst contact the police. Have your son call you at any time he gets scared or anything and keep your phone close by you while he is at his fathers and call him to check up on him every once in a while. Don't stress yourself out, but do be prepared for anything.