Your divisor here isn't "frames", but the number of times you threw at a full rack. So with three full racks in the tenth with your sample game, you're correct that you should divide by 12.
If you don't close out the tenth, then you could be dividing by only 10.
Pseudocode for Excel. I'll indicate the ball number after the frame:
if (10.1 == "x" and 10....