Although Charlton have been given the all clear to speak to Taylor, they have not done so yet. However, Taylor says that "out of respect" he will meet with the London club.
He said: "I've still got a house in Southend, my daughter still lives there, and it's been hard at times for my wife to be away from our friends and family.
"We go back on a regular basis at weekends and Adam Pearson has been brilliant in that respect.
"But, at the same time, we really enjoy living in Ferriby and, as things stand, I'm the Hull City manager for the foreseeable future."
Taylor has been in charge of City since October 2002 and has taken the side to two promotions in that time.