Neighbours building infill isn’t ‘any of your business’: expert