Sometimes tournament organizers show the team colors near the scoresheet. E.g, during Champions League games this information is shown in the upper left corner of the screen, but it can also depend on the broadcaster.
Here e.g. the corresponding indicator in the upper left corner shows that Real Madrid players wear white, while Liverpool players are playing in red. The photo is from 2018 Champions League final game.
Source: si.com
But I guess your question comes from the context of the 2018 FIFA World Cup, in which tournament organizers decided to show the flags of the corresponding national teams instead of the team colors.* Which also makes sense, as national team jersey colors usually represent the colors of their national flag.
Here, e.g. it's pretty much obvious that Saudi players wear green (symbolizing their national flag), while Russians are in red-white kits (representing 2 of the 3 colors of their flag).
But this is not always the case as sometimes teams need to use their secondary kits when facing an opponent with a similar kit (e.g. Portugal and Morocco). Then you should probably listen to the commentary, as commentators announce the team colors several times during a game.
Source: FIFA
In club competitions there are no flags, so the above method doesn't work. In this case if you know which teams are playing but don't know the team colors and the colors are not depicted on the scoresheet, then keep an eye and ear on fans' reactions of what's happening on the field. If they are cheering when team A controls the ball and booing when team B does it, then the team whose name is on the left hand side of the scoresheet is team A, because home teams are always on the left hand side of the scoresheet and away teams on the right.
*Note: apparently the main broadcasting graphic was showing the team colors as well.