For four decades, information theory has been viewed almost exclusively as a theory based upon the Shannon measure of uncertainty and information, usually referred to as Shannon entropy. Since the publication of Shannon's seminal paper in 1948, the theory has grown extremely rapidly and has been applied with varied success in almost all areas of human endeavor. At this time, the Shannon information theory is a well established and developed body of knowledge. Among its most significant recent contributions have been the use of the complementary principles of minimum and maximum entropy in dealing with a variety of fundamental systems problems such as predic­ tive systems modelling, pattern recognition, image reconstruction, and the like. Since its inception in 1948, the Shannon theory has been viewed as a restricted information theory. It has often been argued that the theory is capable of dealing only with syntactic aspects of information, but not with its semantic and pragmatic aspects. This restriction was considered a v~rtue by some experts and a vice by others. More recently, however, various arguments have been made that the theory can be appropriately modified to account for semantic aspects of in­ formation as well. Some of the most convincing arguments in this regard are in­ cluded in Fred Dretske's Know/edge & Flow of Information (The M.LT. Press, Cambridge, Mass., 1981) and in this book by Guy lumarie.

Inhalt

1. Relative Information - What For?.- 1.1 Information Theory, What Is It?.- 1.1.1 Summary of the Story.- 1.1.2 Communication and Information.- 1.2 Information and Natural Language.- 1.2.1 Syntax, Semantics, Lexeme.- 1.2.2 Information, Learning, Dislearning.- 1.3 Prerequisites for a Theory of Information.- 1.3.1 Relativity of Information.- 1.3.2 Negative Information.- 1.3.3 Entropy of Form and Pattern.- 1.3.4 Information and Thermodynamics.- 1.3.5 Information and Subjectivity.- 1.4 Information and Systems.- 1.4.1 A Model of General Systems.- 1.4.2 A Model of Relative Information.- 1.4.3 A Few Comments.- 1.5 How We Shall Proceed.- 1.5.1 Aim of the Book.- 1.5.2 Subjective Information and Relative Information.- 1.5.3 Minkowskian Observation of Events.- 1.5.4 A Unified Approach to Discrete Entropy and Continuous Entropy.- 1.5.5 A Word of Caution to the Reader.- 2. Information Theory - The State of the Art.- 2.1 Introduction.- 2.2 Shannon Measure of Uncertainty.- 2.2.1 The Probabilistic Framework.- 2.2.2 Shannon Informational Entropy.- 2.2.3 Entropy of Random Variables.- 2.3 An Intuitive Approach to Entropy.- 2.3.1 Uniform Random Experiments.- 2.3.2 Non Uniform Random Experiments.- 2.4 Conditional Entropy.- 2.4.1 Framework of Random Experiments.- 2.4.2 Application to Random Variables.- 2.5 A Few Properties of Discrete Entropy.- 2.6 Prior Characterization of Discrete Entropy.- 2.6.1 Properties of Uncertainty.- 2.6.2 Some Consequences of These Properties.- 2.7 The Concept of Information.- 2.7.1 Shannon Information.- 2.7.2 Some Properties of Transinformation.- 2.7.3 Transinformation of Random Variables.- 2.7.4 Remarks on the Notation.- 2.8 Conditional Transinformation.- 2.8.1 Main Definition.- 2.8.2 Some Properties of Conditional Transinformation.- 2.8.3 Conditional Transinformation of Random Variables.- 2.9 Renyi Entropy.- 2.9.1 Definition of Renyi Entropy.- 2.9.2 Meaning of the Renyi Entropy.- 2.9.3 Some Properties of the Renyi Entropy.- 2.10 Cross-Entropy or Relative Entropy.- 2.10.1 The Main Definition.- 2.10.2 A Few Comments.- 2.11 Further Measures of Uncertainty.- 2.11.1 Entropy of Degree c.- 2.11.2 Quadratic Entropy.- 2.11.3 R norm Entropy.- 2.11.4 Effective Entropy.- 2.12 Entropies of Continuous Variables.- 2.12.1 Continuous Shannon Entropy.- 2.12.2 Some Properties of Continuous Entropy.- 2.12.3 Continuous Transinformation.- 2.12.4 Further Extensions.- 2.13 Hatori's Derivation of Continuous Entropy.- 2.14 Information Without Probability.- 2.14.1 A Functional Approach.- 2.14.2 Relative Information.- 2.15 Information and Possibility.- 2.15.1 A Few Prerequisites.- 2.15.2 A Measure of Uncertainty Without Probability.- 2.16 Conclusions.- 3. A Critical Review of Shannon Information Theory.- 3.1 Introduction.- 3.2 On the Invariance of Measures of Information.- 3.3 On the Modelling of Negative Transinformation.- 3.3.1 Classification of Terms.- 3.3.2 The Problem of Modelling "True" and "False".- 3.3.3 Error-Detecting Codes.- 3.4 On the Symmetry of Transinformation.- 3.4.1 A Diverting Example.- 3.4.2 Application of Information Theory.- 3.4.3 On the Symmetry of Transinformation.- 3.4.4 On a Possible Application of Renyi Entropy.- 3.5 Entropy and the Central Limit Theorem.- 3.5.1 The Central Limit Theorem.- 3.5.2 An Information Theoretical Approach to the Central Limit Theorem.- 3.5.3 Relation with Thermodynamics.- 3.5.4 Continuous Entropy Versus Discrete Entropy.- 3.6 On the Entropy of Continuous Variables.- 3.6.1 The Sign of the Continuous Entropy.- 3.6.2 A Nice Property of Continuous Entropy.- 3.7 Arguments to Support Continuous Entropy.- 3.7.1 On the Negativeness of Continuous Entropy.- 3.7.2 On the Non-invariance of Continuous Entropy.- 3.7.3 Channel Capacity in the Presence of Noise.- 3.8 The Maximum Entropy Principle.- 3.8.1 Statement of the Principle.- 3.8.2 Some Examples.- 3.9 Arguments to Support the Maximum Entropy Principle.- 3.9.1 Information Theoretical Considerations.- 3.9.2 Thermodynamic Considerations.- 3.9.3 Axiomatic Derivation.- 3.9.4 A Few Comments.- 3.10 Information, Syntax, Semantics.- 3.10.1 On the Absolute Nature of Information.- 3.10.2 Information and Natural Language.- 3.11 Information and Thermodynamics.- 3.11.1 Informational and Thermodynamic Entropy.- 3.11.2 Thermodynamic Entropy of Open Systems.- 3.12 Conclusions.- 4. A Theory of Relative Information.- 4.1 Introduction.- 4.2 Observation, Aggregation, Invariance.- 4.2.1 Principle of Aggregation.- 4.2.2 Principle of Invariance.- 4.2.3 A Few Comments.- 4.3 Observation with Informational Invariance.- 4.4 Euclidean Invariance.- 4.4.1 Orthogonal Transformation.- 4.4.2 Application to the Observation of Probabilities.- 4.4.3 Application to the Observation of Classes.- 4.5 Minkowskian Invariance.- 4.5.1 Lorentz Transformation.- 4.5.2 Application to the Observation of Probabilities.- 4.5.3 Application to the Observation of Classes.- 4.6 Euclidean or Minkowskian Observation?.- 4.6.1 Selection of the Observation Mode.- 4.6.2 Application to the [Uncertainty, Information] Pair.- 4.7 Information Processes and Natural Languages.- 4.7.1 On the Absoluteness of Information.- 4.7.2 The Information Process.- 4.7.3 Natural Language.- 4.7.4 Information and Natural Language.- 4.8 Relative Informational Entropy.- 4.8.1 Introduction to the Relative Observation.- 4.8.2 Informational Invariance of the Observation.- 4.8.3 Relative Entropy.- 4.8.4 Comments and Remarks.- 4.9 Conditional Relative Entropy.- 4.9.1 Relative Entropy of Product of Messages.- 4.9.2 Composition Law for Cascaded Observers.- 4.9.3 Relative Entropy Conditional to a Given Experiment.- 4.9.4 Applications to Determinacy.- 4.9.5 Comparison of H(?/?) with Hr(?/?).- 4.10 On the Meaning and the Estimation of the Observation Parameter.- 4.10.1 Estimation of the Observation Parameter.- 4.10.2 Practical Meaning of the Observation Parameter.- 4.10.3 On the Value of the Observation Parameter u(R).- 4.11 Relative Transinformation.- 4.11.1 Derivation of Relative Transinformation.- 4.11.2 Some Properties of the Relative Transinformation.- 4.11.3 Relative Entropy and Information Balance.- 4.11.4 Application to the Encoding Problem.- 4.12 Minkowskian Relative Transinformation.- 4.12.1 Definition of Minkowskian Relative Transinformation.- 4.12.2 Some Properties of Minkowskian Relative Transinformation.- 4.12.3 Identification via Information Balance.- 4.13 Effect of Scaling Factor in an Observation with Informational Invariance.- 4.14 Comparison with Renyi Entropy.- 4.14.1 Renyi Entropy and Relative Entropy.- 4.14.2 Transinfo…