‘Affordable housing: No matter what they say in public, they're terrified of it’