Do i go...or do i not...?

I would first talk to him about how you feel. He may not realize what he is doing. Even though he is a pastor he is still human. And tell him if he thinks your mom will have a problem them ask him to meet your mom. He might just feel a little hurt, pastors want to be there for thier "flock" and it might hurt them not to come through