Diagram: Entropy of a binary Information Source

The diagram below shall demonstrate the characteristics of a binary information source providing two different messages (A and B):

The probability of the message A is always

P(A) = 1 - P(B),

because the sum of both probabilities must be 1. Other messages than A or B did not occur.

The maximum entropy (H = 1) will be reached if both messages have the same probability (each 0.5). In this case the information source must be coded with an average code length of 1 bit. Only if the probabilities differ, i.e. one of the messages appears more frequent, there would be an option to get a better code efficiency.

Values:

 <   ^   >