I don't think that it's entirely accurate to say that Casey built the club to the current success that it is, because quite frankly, I don't understand how he's been the coach for as long as he has. Historically, his player rotations have been somewhat strange, as has his justification. And as @jcmcclorey mentioned, he didn't advance in the playoffs. The simple reason is that you have to win in the playoffs, and Casey has a 21-30 record (19-22 as the higher seed) in the playoffs with Toronto. I'll give a possible explanation as to why that is the case.
Specifically, I would refer to the 2015 series against Washington (after which, I think he should have been let go), and then talk about this most recent playoff run. In this article, you can see that Casey was making some very strange comments in light of how he had managed that series and also season. In particular, he calls James Johnson the most talented player on their roster, while only playing him 12 minutes over the entire series, where they struggled defensively (and also offensively). When we look to this past playoffs, we again see some personel decisions that are bizarre, generally as it pertains to Valanciunas. Some of the reason for that is his general tendency to plays the game on the opponent's terms, as opposed to forcing them to play on Toronto's terms. With Valanciunas, it typically leads to benching him, where he provides Toronto with an offensive advantage. Jumping around a bit, we also see this trend last year with respect to Casey's willingness to play Valanciunas vs Ibaka or PJ Tucker. One issue with the team over his tenure (that, to his credit did get fixed this year) was the lack of ball movement in the offense, and this can be seen in the falling assist rate over the course of his time in Toronto prior to this year. This is notable, because assist rates are fairly indicative of playoff success; whereas, Toronto has been very dependent upon free throw shooting over the last few years, which is actually not indicative of playoff success (see here). Now, as I had mentioned, they did fix some of these problems throughout the regular season (also, you'd note that in the last article, they were 30th in DREB% last year, but only 12th this year), but seemed to revert to more of their isolation-focused basketball from years past.
Edit: One other issue with Casey has been his tendency to not hold the star players accountable. This article comes from 2 years ago where they went to the ECF in 2016. The article talks about DeRozan's struggles in the first 2 games of their opening round series against Indiana, where Indiana focused on taking DeRozan out of the offence and forcing the other Raptors to beat them. What's notable is the contrasting suggestions from Casey and Lowry on what DeRozan should do in Game 3. Casey said he should be a facilitator, while Lowry said he should keep shooting. The author also notes the advantage they had with Valanciunas against Ian Manhinmi. Clearly, DeRozan listened to the "the little devil yammering in his other ear", going 7-19 in 36+ minutes in Game 3. Just as importantly, this shows a severe disconnect between the coach and the star players. I do also remember there being some whispers about a locker room schism in the 2014-15 season (the one that ended with the sweep to Washington) where you had a DeRozan-Lowry-Lou Williams faction against a Valanciunas-Amir Johnson-James Johnson group, with the latter being very dissatisfied that the former could do whatever they liked without repercussion, but Valanciunas would constantly be benched. Now, I can't find any sources for this, but I do recall seeing some articles about this in 2015.
