As an umpire and a player, I've never encountered this scenario, but I can see where it might occur.
The play should have been called a let and replayed:
2.9.1.2 if the service is delivered when the receiving player or pair is not ready,
provided that neither the receiver nor his or her partner attempts to strike
the ball;
If you served and the player made no attempt to return the serve and claimed that you took too long, it could be argued that it should be played as a let.
The rule was actually meant more for when a player serves when the receiver is clearly not ready to receive, but if you make an attempt to return the serve, you cannot call a let. This has happened to me personally many times because I am legally blind (no peripheral vision) and I am focused upon getting ready to get in my stance/position but the server sometimes will not look at me to confirm, with eye contact, that I'm ready and serve before I am set.
That doesn't appear to be what happened in this case, but the idea is the same. If your opponent attempted to return your serve, but claimed you took too long, too bad on his/her part, it's your point. If they did not attempt to return the serve, it's a let.