Yes. The protection provided by such equipment (in karting and other sports) is the ability of the material to flex, thereby dispensing kinetic energy into elastic energy, and to disperse forces across a wider surface area, thereby lowering the actual force experienced by any given location.
When a vest is cracked (or potentially for other materials, warped or bent) it loses a significant portion of its ability to flex. This means more of the energy is left to be absorbed by the body, which equates to a higher likelihood and severity of injury. To a lesser extent, if a cracked component is hit, it will be less able to disperse forces across an area - because the individual fragment must take up all of the force - and if the crack itself is hit, this effect is almost negligible.
Given the importance of safety from impact in sports generally and karting particularly, you are better to abandon any noticeably damaged safety equipment and replace it (or at least, the damaged components). Even the smallest of visible cracks can indicate a structural weakness in the component that, when placed under stress, will lead to failure and a lack of adequate protection.