wire / wire-format / shannon-entropy

Shannon entropy

H(X) = -Σ p(x) log2 p(x). Measures average information content in bits. Minimum bits to encode a random variable. Source coding theorem: can compress to H(X) bits per symbol (lossless) or H(X|Y) with side info.

wire-format kind wire-format status standard verification verified tier B wire-format@1

aka: Shannon entropy

wire format

spec: Shannon information theory (1948)

H(X) = -Σ p(x) log2 p(x). Measures average information content in bits. Minimum bits to encode a random variable. Source coding theorem: can compress to H(X) bits per symbol (lossless) or H(X|Y) with side info.

fieldsizemeaning
Entropy H(X)bitsH(X) = -Σ p(x) log2 p(x). Uniform distribution maximizes entropy: H = log2(n) for n symbols.
Joint entropy H(X,Y)bitsH(X,Y) = -Σ p(x,y) log2 p(x,y). H(X,Y) ≤ H(X)+H(Y).
Conditional entropy H(X|Y)bitsH(X|Y) = H(X,Y) - H(Y). Information X provides given Y.
Mutual information I(X;Y)bitsI(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X). Symmetric measure of dependence.
Channel capacitybits/useC = max I(X;Y) over input distributions. Shannon's channel coding theorem: can achieve any rate R < C with arbitrarily low error.

example:

00 00

Shannon entropy (placeholder)


provenance

agent: curl -H 'accept: application/json' wire.phall.io/wire-format/shannon-entropy or /wire-format/shannon-entropy.json