Information theory

Information theory is a branch of mathematics that extends into biology, engineering, psychology, medical science, communications, and sociology. The theory has also been applied to linguistics, cryptography, phonetics, and communication engineering.

Claude E. Shannon founded modern information theory in 1948. The theory involves the communication of information and the behavior of information as it is transmitted, stored, and retrieved. Numerous elements come into play with information theory, including entropy, a source of information, a channel, a receiving device, a destination, and noise. Furthermore, the theory defines a basic unit for measuring information. Information theory also describes the elements and the quantities of communication systems and explains the concept of uncertainty as it relates to information.

Overview

Information theory was founded in 1948 when a mathematician named Claude Shannon, who was working at Bell Telephone Laboratories, published a paper titled “A Mathematical Theory of Communication.” The paper focused on the communication of information and the most efficient way of transmitting it. Mathematicians and scientists throughout the world showed interest in Shannon’s paper, and the information theory discipline was soon born.

Information theory involves several elements. Perhaps the most basic of these elements is that information can be measured as a mathematical quantity. One of the most significant elements of the theory is entropy. Shannon explained that entropy is a measure of uncertainty in a message source. This means the message contains a degree of uncertainty or randomness. Shannon explained what happens in a noisy conversation. He proved that signals can be transmitted with an arbitrarily small frequency of errors when suitable coding is used and the rate does not exceed the channel’s capacity.

Information theory also focuses on the fundamental unit of information. This unit, which is called a bit, is a yes/no or on/off situation. It can be expressed by the binary digits 1 and 0, which represent Boolean two-value binary algebra. A 1 means the power is on, while a 0 means the power is off. Combinations of bits allow for more complicated information. For example, digital systems can represent text, images, and sound as sequences of bits. These combinations form more complex digital representations.

Some of information theory’s other elements involve communication systems. It states that these systems include a source of information that produces a message, which a transmitting device turns into a form that can be transmitted by a certain means. Furthermore, these systems include the channel that transmits the message. Communication systems also have a receiving device that decodes the message, turning it back into a form that is close to the original. The systems include the destination of the message or the intended recipient. Lastly, communication systems have a source of noise, which is distortion or interference. This noise changes the message during transmission.

The theory further explains the quantities that communication systems yield. The theory suggests that information is a degree of order, or nonrandomness. This nonrandomness can be measured mathematically, and so a mathematical characterization of communication systems yields certain quantities. These include the rate that information is generated at the source; the channel’s capacity for handling information; and the average amount of information in the message.

The theory also describes uncertainty as it relates to information. Uncertainty is the process of selecting objects from a set of objects. For example, if a device produces three symbols (A, B, or C), an individual is uncertain which symbol will be produced next. Uncertainty decreases when the symbol appears; the individual has received information. In this sense, information is a decrease in uncertainty. Furthermore, uncertainty may be measured. In this example, the device has an entropy of log2(3) bits if the three symbols are equally likely. However, this kind of measurement does not work if a second device is combined with the first one. For instance, if a second device produces two symbols (1 and 2), it would have an entropy of log2(2) bits if the symbols are equally likely. But if the second device were combined with the first device, the combined device would then have six possibilities—A1, A2, B1, B2, C1, and C2. The device would now have an entropy of log2(6) bits if the six outcomes are equally likely. People do not typically view information in this way. For example, if an individual receives one book and then receives another book, they will add the amount of information that has been received. In other words, once a person receives the second book, the individual would state that they received twice as much information, because one book is added to another.

Many of the techniques used with information theory are derived from the science of probability. For example, probability is used when given estimates of an information transmission’s accuracy under noise interference. Furthermore, probability is used with encoding and decoding approaches to reduce uncertainty or error.

One important use of information theory in cryptography is in post-quantum cryptography, including US standards approved by NIST in 2024. Other disciplines have developed from information theory, including coding theory. Coding theory, which is also called algebraic coding theory, involves the design of codes that correct errors and allow for reliable information transmission over noisy channels. Coding theory utilizes several algebraic techniques that deal with group theory, finite fields, and polynomial algebra. These ideas are used in wireless communication systems such as 5G New Radio (NR), which uses low-density parity-check and polar codes. Information theory also contributes to quantum information science, which includes quantum communication, quantum computing, and quantum cryptography. Information theory remains an active field with major applications in communication, coding, and data processing.

Bibliography

“About the National Quantum Initiative.” National Quantum Initiative, www.quantum.gov/about/. Accessed 16 Apr. 2026.

“Announcing Approval of Three Federal Information Processing Standards (FIPS) for Post-Quantum Cryptography.” NIST Computer Security Resource Center, 13 Aug. 2024, csrc.nist.gov/news/2024/postquantum-cryptography-fips-approved. Accessed 16 Apr. 2026.

“5G; NR; Multiplexing and Channel Coding (3GPP TS 38.212, Version 17.12.0, Release 17).” European Telecommunications Standards Institute, 2025, www.etsi.org/deliver/etsi_ts/138200_138299/138212/17.12.00_60/ts_138212v171200p.pdf. Accessed 16 Apr. 2026.

Markowsky, George. “Information Theory.” Britannica, 6 Apr. 2026, www.britannica.com/science/information-theory. Accessed 16 Apr. 2026.

“National Quantum Strategy.” Quantum Gov., www.quantum.gov/strategy/. Accessed 16 Apr. 2026.

Shannon, Claude E. “A Mathematical Theory of Communication.” Bell System Technical Journal, vol. 27, no. 3, 1948, pp. 379–423; vol. 27, no. 4, Oct. 1948, pp. 623–56. www.nokia.com/bell-labs/claude-shannon/assets/images/discoveries/1948-04-21-a-mathematical-theory-of-communication-parts-I-and%E2%80%93carousel-01.pdf. Accessed 16 Apr. 2026.

Soni, Jimmy, and Rob Goodman. A Mind at Play: How Claude Shannon Invented the Information Age. Simon & Schuster, 18 July 2017.

Tucci, Linda. “Information Age.” TechTarget, 8 May 2023, searchnetworking.techtarget.com/definition/information-theory. Accessed 16 Apr. 2026.

Weisstein, Eric W. “Coding Theory.” Wolfram MathWorld, mathworld.wolfram.com/CodingTheory.html. Accessed 16 Apr. 2026.