Remember, wrasling with pigs only gets you muddy, and the pig enjoys it.
This is a time, generally, for the high road. Oh, there will be people who will wonder, a little. Generally they are the type that inherently hold women to a lower level. And a few of the pigs friends will believe him enthusiastically.
But, overall, by staying calm and being the above the board adult will have you coming out on top after the wind blows.
The best thing to do is just ignore him. Eventually he will realize that he's having no effect and will move on to something else. Don't worry about what other people think, it's pretty rare that someone can have a big effect on that.
Now, if he steps over the line from just running his mouth to accosting you, hangup phone calls, damaging property etc then it's time to document it and then get the po po involved. Don't try to handle it yourself, that will only make things worse.