Information Dynamics

l Introduction.- 2 Dynamical Systems: An Overview 7.- 2.1 Deterministic Dynamical Systems.- 2.1.1 Fundamental Concepts.- 2.1.2 Attractors.- 2.1.3 Strange Attractors: Chaotic Dynamics.- 2.1.4 Quantitative Description of Chaos.- 2.1.5 Chaotic Dynamical Systems.- 2.2 Stochastic Dynamical Systems.- 2.2.1 Gaussian White Noise.- 2.2.2 Markov Processes.- 2.2.3 Linear and Nonlinear Stochastic Dynamics.- 2.3 Statistical Time-Series Analysis.- 2.3.1 Nonstationarity: Slicing Windows.- 2.3.2 Linear Statistical Inference: Correlations and Power Spectrum.- 2.3.3 Linear Filter.- 3 Statistical Structure Extraction in Dynamical Systems: Parametric Formulation.- 3.1 Basic Concepts of Information Theory.- 3.2 Parametric Estimation : Maximum-Likelihood Principle.- 3.2.1 Bayesian Estimation.- 3.2.2 Maximum Likelihood.- 3.2.3 Maximum-Entropy Principle.- 3.2.4 Minimum Kullback-Leibler Entropy.- 3.3 Linear Models.- 3.4 Nonlinear Models.- 3.4.1 Feedforward Neural Networks.- 3.4.2 Recurrent Neural Networks.- 3.5 Density Estimation.- 3.6 Information-Theoretic Approach to Time-Series Modeling: Redundancy Extraction.- 3.6.1 Generalities.- 3.6.2 Unsupervised Learning : Independent Component Analysis for Univariate Time Series.- 3.6.3 Unsupervised Learning: Independent Component Analysis for Multivariate Time Series.- 3.6.4 Supervised Learning : Maximum-Likelihood.- 4 Applications: Parametric Characterization of Time Series.- 4.1 Feedforward Learning : Chaotic Dynamics.- 4.2 Recurrent Learning : Chaotic Dynamics.- 4.3 Dynamical Overtraining and Lyapunov Penalty Term.- 4.4 Feedforward and Recurrent Learning of Biomedical Data.- 4.5 Unsupervised Redundancy-Extraction-Based Modeling: Chaotic Dynamics.- 4.5.1 Univariate Time Series : Mackey-Glass.- 4.5.2 Multivariate Time Series : Taylor-Couette.- 4.6 Unsupervised Redundancy Extraction Modeling: Biomedical Data.- 5 Statistical Structure Extraction in Dynamical Systems: Nonparametric Formulation.- 5.1 Nonparametric Detection ofStatistical Dependencies in Time Series.- 5.1.1 Introduction and Historical Perspective.- 5.1.2 Statistical Independence Measure.- 5.1.3 Statistical Test: The Surrogates Method.- 5.1.4 Nonstationarity.- 5.1.5 A Qualitative Test of Nonlinearity.- 5.2 Nonparametric Characterization of Dynamics: The Information Flow Concept.- 5.2.1 Introduction and Historical Perspective.- 5.2.2 Information Flow for Finite Partitions.- 5.2.3 Information Flow for Infinitesimal Partition.- 5.3 Information Flow and Coarse Graining.- 5.3.1 Generalized Correlation Functions.- 5.3.2 Distinguishing Different Dynamics.- 6 Applications: Nonparametric Characterization of Time Series.- 6.1 Detecting Nonlinear Correlations in Time Series.- 6.1.1 Test ofNonlinearity.- 6.1.2 Testing Predictability: Artificial Time Series.- 6.1.3 Testing Predictability: Real-World Time Series.- 6.1.4 Data Selection.- 6.1.5 Sensitivity Analysis.- 6.2 Nonparametric Analysis of Time Series : Optimal Delay Selection.- 6.2.1 Nonchaotic Deterministic.- 6.2.2 Linear Stochastic.- 6.2.3 Chaotic Deterministic.- 6.3 Determining the Information Flow ofDynamical Systems from Continuous Probability Distributions.- 6.4 Dynamical Characterization ofTime Signals: The Integrated Information Flow.- 6.5 Information Flow and Coarse Graining: Numerical Experiments.- 6.5.1 The Logistic Map.- 6.5.2 White and Colored Noise.- 6.5.3 EEG Signals.- 7 Statistical Structure Extraction in Dynamical Systems: Semiparametric Formulation.- 7.1 Markovian Characterization of Univariate Time Series.- 7.1.1 Measures ofIndependence.- 7.1.2 Markovian Dynamics and Information Flow.- 7.2 Markovian Characterization of Multivariate Time Series.- 7.2.1 Multidimensional Cumulant-Based Measure of Information Flow.- 7.2.2 Nonlinear N-dimensional Markov Models as Approximations ofthe Original Time Series.- 8 Applications: Semiparametric Characterization of Time Series.- 8.1 Univariate Time Series : Artificial Data.- 8.1.1 Autoregressive Models : Linear Correlations.- 8.1.2 Nonlinear Dependencies: Non-Chaos, Chaos, and Noisy Chaos.- 8.2 Univariate Time Series: Real-World Data.- 8.2.1 Monthly Sunspot Numbers.- 8.2.2 The Hidden Dynamics of the Heart Rate Variability.- 8.3 Multivariate Time Series: Artificial Data.- 8.3.1 Autoregressive Time Series.- 8.3.2 Nonlinear Time Series.- 8.3.3 Chaotic Time Series : The Henon Map.- 8.4 Multivariate Time Series : Tumor Detection in EEG Time Series.- 9 Information Processing and Coding in Spatiotemporal Dynamical Systems: Spiking Networks.- 9.1 Spiking Neurons.- 9.1.1 Theoretical Models.- 9.1.2 Rate Coding versus Temporal Coding.- 9.2 Information Processing and Coding in Single Spiking Neurons.- 9.3 Information Processing and Coding in Networks of Spiking Neurons.- 9.4 The Processing and Coding ofDynamical Systems.- 10 Applications: Information Processing and Coding in Spatiotemporal Dynamical Systems.- 10.1 The Binding Problem.- 10.2 Discrimination of Stimulus by Spiking Neural Networks.- 10.2.1 The Task: Visual Stimulus Discrimination.- 10.2.2 The Neural Network: Cortical Architecture.- 10.3 Numerical Experiments.- Epilogue.- Appendix A Chain Rules, Inequalities and Other Useful Theorems in Information Theory.- A.1 Chain Rules.- A.2 Fundamental Inequalities ofInformation Theory.- Appendix B Univariate and Multivariate Cumulants.- Appendix C Information Flow of Chaotic Systems: Thermodynamical Formulation.- Appendix D Generalized Discriminability by the Spike Response Model ofa Single Spiking Neuron: Analytical Results.- References.