It is a difficult situation you find yourself in and none of us are experts on what to do but by brainstorming you are getting the benefit of views that span decades having faced similar situations. My hat dips for you by listening and considering.
So bearing in mind what I've previously said-
"my dad asked when he could see my husband..." Your dad can ring/write/text your husband. They are adults. So if others ask when they can talk to your husband- "you can contact him". You aren't his doorkeeper.
"My mum knows he doesn't want to see her". Then why did she drop in? There is no excuse for his conditions not to be met. He would feel that there is no escaping your family of which, at the moment and foreseeable years, is what he wants. Your family must respect that.
If your children have an upset and you believe they would benefit from seeing their grandparents - take them to their home. Just because your husband doesnt want a relationship with them shouldnt mean anyone else should be distanced from love ones.
"I just couldn't help but feel anxious when i saw my mum and then the conversation just made it worse for me. It's a shame that I have to feel like this." As your mum shouldnt have attended your house, you could have supported him in that you could have said "she knew she wasnt permitted to attend here, I'll make it clear again as its your wish".
Chance meeting might also occur- shopping centres and the like. It isnt your responsibility to be the united nations and keep separated these people. Again, they are adults and they have the responsibility to avoid each other. That's where a lot of your anxiety is originating from.
What is your responsibility? Well, supporting your husband for his quirkiness (compared to your family) is important as he is an individual with not much in common with your family. Respecting his wishes and advertising those wishes when to opportunity arises with those he doesnt not want contact with. Maintaining a healthy loving relationship with your own family and continuing to promote their relationships with your children bearing in mind you feeling of your husband when it comes to decision made. eg always check with him with dates of events and who minds the kids.
Praise- he might need it! when you return from a visit to your family you can say "its really good of you to not interfere with my relationship with my parents and we can all remain happy".