Concept

Entropy: It is a measure of model's uncertainty about the future. It is low if the model expect a perticular symbol to occur with a high probability. It increases when the estimated probability distribution approaches normal.