It's hard to prove a negative, but neither the Laws, nor the Instructions to Technical Officials, nor the BWF umpire manuals mention any right of the umpire to refuse a challenge.
The only limitation on challenges follows from §5.8.5 ITTO:
(...) The player must clearly say ‘Challenge’ to the Umpire and/or
make a clear signal by raising the hand. Any such ...