AALS and Boycotts

Tom's post suggests that a boycott  of a hotel owned by someone who donated funds to oppose same-sex marriage by groups who are part of the AALS is legitimate, but wonders at some point a market power counterargument kicks in.  I'm still sorting out what I think about this but what I wonder is whether this isn't more of a speech issue than a market one.  That is, it is one thing to organize a boycott against the hotel if there is something about the business itself that is objectionable (e.g., there is evidence of discrimination against homosexuals in the actual operations of the hotel).  I assume no one would find such a boycott objectionable.  But isn't it another thing for a legal academic group (which presumably thinks freedom of speech and expression is a good thing) to organize a boycott based on the owner's speech (in this case evidenced by a donation), even if we disagree with that speech?  I'd be interested in hearing the thoughts of others on this.