Allam, who took over in 2010, oversaw the club's rise to the Barclays Premier League last season but has infuriated fans with his determination to rebrand the club Hull Tigers.
A number of supporters protested during the game against Crystal Palace last week by unfurling a banner which read "We are Hull City".
A campaign group called City Till We Die has also been set up and it has called on Allam to launch a consultation with all the fans regarding the proposed change.
Allam, however, gives short shrift to any opposition and he told the Independent on Sunday: "I don't mind 'City till we die'.
"They can die as soon as they want, as long as they leave the club for the majority who just want to watch good football."
He added: "How can they call themselves fans, these hooligans, this militant minority, when they disturb and distract the players while taking away the rights of others to watch the football, and of companies who have paid good money for advertising?"
Source: PA
Source: PA