In answer to your first question, let's consider how it should be called.
From Law 14 - The Penalty Kick, Section 1 - Procedure:
The defending goalkeeper must remain on the goal line, facing the kicker, between the goalposts until the ball has been kicked.
The players other than the kicker and goalkeeper must be:
- at least 9.15 m (10 yds) from the penalty mark
- behind the penalty mark
- inside the field of play
- outside the penalty area
After the players have taken positions in accordance with this Law, the referee signals for the penalty kick to be taken.
Now, if encroachment as defined above occurs, Law 14 - The Penalty Kick, Section 2 - Offences and Sanctions states that:
Once the referee has signalled for a penalty kick to be taken, the kick must be taken. If, before the ball is in play, one of the following occurs:
- the player taking the penalty kick or a team-mate offends:
- if the ball enters the goal, the kick is retaken
- if the ball does not enter the goal, the referee stops play and restarts with an indirect free kick
- the goalkeeper or a team-mate offends:
- if the ball enters the goal, a goal is awarded
- if the ball does not enter the goal, the kick is retaken; the goalkeeper is cautioned if responsible for the offence
- a player of both teams offends the Laws of the Game, the kick is retaken unless a player commits a more serious offence (e.g. illegal feinting); if both the goalkeeper and kicker commit an offence at the same time:
- if the kick is missed or saved, the kick is retaken and both players cautioned
- if the kick is scored, the goal is disallowed, the kicker is cautioned and play restarts with an indirect free kick to the defending team
A summary of the outcomes for encroachment (and other penalty kick infringements) is also included in Law 14 - The Penalty Kick, Section 3 - Summary Table.
As for your second question, with the 2016 rewrite of the Laws of the Game, referees were explicitly given broad discretion. In Law 5 - The Referee, Section 2 - Decisions of the Referee, it states:
Decisions will be made to the best of the referee`s ability according to the Laws of the Game and the ‘spirit of the game’ and will be based on the opinion of the referee who has the discretion to take appropriate action within the framework of the Laws of the Game.
The other big change in 2016 was the requirement to caution the goalkeeper for unsporting behaviour if a retake is required due to them leaving their goal line before the kick is taken.
Whether it was intended or not by IFAB, the combination of these two changes have led to referees only taking action when there is blatant encroachment. The alternative, i.e. strict enforcement, would dictate that the goalkeeper would be sent-off for being millimetres off their line for a saved / missed penalty kick twice in a match. This is clearly a disproportionate punishment.
Finally, here's an example of a referee having a penalty kick retaken multiple times for encroachment by both the goalkeeper and outfield players. In my opinion, at the time when this occurred in 2004, he would have been correct. However, by 2017 standards, none of the kicks required referee intervention.
Also, here's a better quality video of a different incident, with replays showing the encroachment. At the time, in 2005, the first two of these calls were probably correct, but to be consistent, the third kick should have been retaken, as a Brazilian defender (in yellow) entered the penalty area before the kick was taken and it was missed.
However, applying 2017 standards, I believe that only the blatant goalkeeper infringement on the second kick should have been retaken. In 2017, this would also require the goalkeeper to be cautioned, but this seems proportionate, given that he was one to one-and-a-half metres off his line before the kick was taken.