At what point does promoting fascist ideas become akin to shouting “fire” in a crowded theater?

As a UC Berkeley alum and, now, faculty, I’m keenly aware of the history of the free speech movement in the Bay Area and beyond. I recall sitting in an American History class in Wheeler Hall, learning about the events that led to the massive protests and arrests (of students) in the 1960s. NPR has a concise but helpful account of those events here.
So, it’s no surprise that the university would continue to stand for free speech, even on controversial topics.
Knowing this does not make me feel any better about Milo Yiannnopoulos coming to speak on campus tonight. He was invited by the Berkeley College Republicans (yes, they exist), and the campus has decided to let the event stand.
Which… I get. From a free speech perspective, you really can’t advocate banning a speaker based on their ideology (no matter how messed up and bigoted it is).
But at the same time, my understanding is that free speech is limited in some contexts based on its potential consequences. The famous example is that you can’t claim free speech rights when shouting “FIRE!” in a crowded theater – people will be trampled, resulting in however many injuries and possibly even deaths.
At what point do we examine the speech acts of fascists and decide that they are doing the equivalent of shouting “FIRE!” in a crowded theater? At what point does hate speech contain the inevitable threat of its fulfillment? If the alt-right is targeting people who feel disaffected and promising them power if only they agree to persecute certain groups…when does that rhetorical violence shift into real-world violence, with empirical effects on bodies and selves?
I wish I had a better way to delineate where rhetorical violence becomes real violence. But I suspect that even getting people to agree on what constitutes hate speech is problematic, let alone which groups are more or less deserving of value, access to resources, rights, and more.
I also wish that I had the emotional energy to protest right now, since I agree with some of my colleagues that Yiannapoulos’s presence on a college campus makes it an unsafe work environment and is likely a Title IX violation. And I’m not just concerned about him as a speaker; I’m also concerned about his followers (one of whom shot a protester at the University of Washington).
So I’ll probably be doing self-care through dance instead of protesting. I’m not so sure I’d be good for in-your-face conflict right now anyway, and knowing how you handle conflict is an important facet of activism.
I think this is a conversation worth having, though, regardless of whether it happens in digital or in-person spaces. What do you think? Is giving fascist speakers a platform akin to promoting violence?