First lets make sure that most reading this get the general "standard" football formation on offense and defense, set up across from each other here (from womensfootballguide.files.wordpress.com):

With some insight from this Dummies.com article, each positions is quite adapted to play certain roles:
- Generally safeties have player molds emphasizing flexibility, with moderate speed, good ability to change direction, and fair strength. This allows them to help in coverage on a wide range of pass routes, adjust to follow plays as they shift around, and yet also be able to track up inside to slow/stop running backs that break their runs through the first tier of the defense. So, it's important to note, to fit this dynamicism, you'd generally want them to line up towards the center of the field with a large cushion to allow them to manage so many things.
- Generally linebackers have player molds that are a bit more bruising, with pretty decent strength and size, but also some speed. These guys are regularly important in stopping runs or blitzing the quarterback. They don't usually have the speed to cover the fastest guys for longer routes without help, so you'd also tend to want them to stay towards the middle of the formation where there is more protection.
- Generally cornerbacks are have player molds emphasizing speed, span, and are ideally fairly tall as well. All this is to match up with the best receivers the other team can throw at them. They need to be able to stick with the receivers making quick moves when playing man-to-man coverage, or roam a lot of space when playing zones. Given that safeties and linebackers are both patterned to play the middle, and that the speediest wide receivers often play the edges since that gives them the greatest danger of breaking the other 9 guys, cornerbacks are predominantly found at the edges as well.
So, what is the best way to respond if the offense elects to go to a 3 wide receiver set (replacing the fullback with a third receiver, usually lined up in the slot [somewhere inward from one of the other two wide receivers])?
Pulling one of the cornerbacks in to cover the slot receiver leaves a big question mark of how to cover the receiver he is leaving. It is unlikely anyone else on the team can match up with that receiver one-on-one.
Pulling a safety in leaves less protection and more room for the primary receivers or running backs to bust huge plays.
Linemen are usually too big and slow to be much help on a receiver.
So as I understand it, that often leaves the linebackers as often the best option.
Usually the matchup doesn't work out to be too bad. Because unlike the two standard receiver positions... the weak-side receiver (which generally has little focus on strength, since most runs are to the other side [where the tight end is] means this receiver rarely has to get involved in blocking)... and the flanking receiver (who usually has a lot of reliance put on size, working to give him dominance even in the tightest coverage by his single defender), the slot receiver is usually built to be pretty tough and physical. This is so that he can take the punishment he'll often get, since he generally attempts to catch balls in the intermediate region between the line and secondary... meaning many guys will often be nearby to quickly unload big hits as soon as he catches the ball. If somebody too light and athletic plays the slot, the general thinking is be quite prone to injury. And also may struggle to keep possession after the catch without the strength to face the hits and hold onto the ball.
So usually a linebacker will at least line up fairly with a slot receiver. Whereas diverting a cornerback (or safety) to cover him usually risks big play touchdowns.
Now, all this isn't to say the matchup is always great. Indeed, just because it can lead to more injuries doesn't mean a team won't play such a formation!
And, as noted, linebackers really tend to be built a bit more towards dealing with the bigger guys inside and making lots of tackles. Just a bit of a speed edge might be enough for the slot guy to reap great rewards. Plus, the linebackers also must often help cover the tight end and running backs coming out on passing routes as well, plus contain mobile quarterbacks.
This high quality Denver Post explanation and this equally high quality post on FieldGulls.com go into a lot more detail breaking down some of the different positions and matchups within the receivers and secondary.
The troubles often really are more indicative of the overall team skill as well. If you have cornerbacks that can be relied upon to hold single coverage with the main wide receivers, then your safeties can play up more. Or if your defensive linemen can really occupy the offensive line, you can drop more linebackers into coverage, giving less room for the extra receivers. But if those guys are struggling, more attention needs to be paid in those other areas.
Now, as you hint at, the best response often actually ends up being switching out one (nickle) or two (dime) of the linebackers for more of those faster secondary players. If you have the preparedness and players to make those switches.
But this may expose other problems as well.
I'm certainly no football mastermind... but maybe the offense was running the ball well with the RBs or QB, preventing the defense from making a LB-to-DB switch, which would further reduce their effectiveness against the run (or that making the switch would fairly obviously set matchups that would unlock a previously contained run game). Or perhaps the slot receiver was just a bit too physical for a safety or cornerback, and a bit too fast for the linebacker, given the amount of time and room he was getting to operate (in other words, the play calling was giving the QB and slot WR too much time to operate). Maybe they just thought that the linebacker they'd take off would cost them more overall than the DB they'd put on.
Who knows! But the defense was most likely making changes they thought would help contain the slot receiver, they just weren't working. But indeed, mismatches often do seem like one of those things commentators like to obsessively harp on... even though they often really don't know the complexity behind them! These coaches certainly weren't born yesterday, they've probably been facing slot receivers for decades! It just didn't work out that the coaches choices or the personnel they used were as sharp as they have been on other days, and they came up against a coach/team/player that really showed it up!