What happens to the team when an NFL draftee doesn't sign with a team? Do they get some sort of compensation for the lost pick?
Broadly, no: there is no specific compensation for failing to sign a rookie. It doesn't really happen, due to the rookie contracts being strictly defined; Bo Jackson is the only actual unsigned drafted player of note in recent memory (he refused to sign with Tampa Bay due to them allegedly sabotaging his NCAA eligibility, and played the year in MLB for the Kansas City Royals, before being drafted the following year by the L.A. Raiders).
From the 2020 NFL Collective Bargaining Agreement, the following is from Article 6 (covering the draft):
If a Drafted Rookie has not signed a Player Contract during the period from the date of such Draft to the thirtieth day prior to the first game of the regular season: (i) the Club that drafted the player may not thereafter trade to another Club either its exclusive negotiating rights to such player or any Player Contract that it signs with such player for the player’s initial League Year; and (ii) the Club that drafted the player is the only Club with which the player may sign a Player Contract until the day of the Draft in the subsequent League Year, at which time such player is eligible to be drafted in the subsequent League Year’s Draft by any Club except the Club that drafted him in the initial Draft.
There is additional language in there, but nothing about compensatory picks or similar; Appendix V covers compensatory picks in general, but none are awarded by rookies failing to sign.