If a critic wants to come to your conference, take his registration money and welcome him. Declining his registration just makes you look bad. Better your critics come to your events openly than have them recruit supporters to attend your events.
If there are many enthusiastic blog posts about your event, why would you fear a handful of critics? You do have a blogger outreach program don't you? If not, maybe you should get one.