The restaurant says it's seen fewer customers since construction started in August, casting doubt about the future.