A well-known pigeon racer has been refused permission to keep 250 birds in a 15-metre back-garden loft after neighbours objected.