Description:

ITE can estimate Shannon-, Rényi-, Tsallis entropy; generalized variance, kernel canonical correlation analysis, kernel generalized variance, Hilbert-Schmidt independence criterion, Shannon-, L2-, Rényi-, Tsallis mutual information, copula-based kernel dependency, multivariate version of Hoeffding's Phi; complex variants of entropy and mutual information; L2-, Rényi-, Tsallis-, Kullback-Leibler divergence; Hellinger-, Bhattacharyya distance, maximum mean discrepancy, and J-distance.

ITE offers solution methods for

• Independent Subspace Analysis (ISA) and
• its extensions to different linear-, controlled-, post nonlinear-, complex valued-, partially observed models, as well as to systems with nonparametric source dynamics.

ITE is

• written in Matlab/Octave,
• multi-platform (tested extensively on Windows and Linux),
• free and open source (released under the GNU GPLv3(>=) license).

Changes to previous version:

• Two Shannon entropy estimators based on the distance (KL divergence) from the uniform/Gaussian distributions: added.

• Shannon entropy estimator based on Voronoi regions: added.

BibTeX Entry: Download

Corresponding Paper BibTeX Entry: Download

URL: Project Homepage

Supported Operating Systems: Linux, Windows

Data Formats: Matlab, Octave

Tags: Entropy, Mutual Information, Divergence, Independent Subspace Analysis, Separation Principles, Independent Process Analysis

Archive: download here

Other available revisons

Version	Changelog	Date
0.63	Conditional Shannon entropy estimation: added. Conditional Shannon mutual information estimation: included.	June 9, 2016, 23:42:14
0.62	Von Mises expansion based estimators: included for 7 unconditional quantities (Shannon entropy, Shannon mutual information, Kullback-Leibler divergence, Rényi divergence, Tsallis divergence, Pearson Chi^2 divergence, Hellinger distance. Analytical value (for Gaussian random variables) and quick test: added for the Hellinger distance.	April 17, 2016, 17:19:00
0.61	Explicit additive constant computation in generalized kNN based Renyi entropy estimators: enhancement suggestion has been added. Analytical value computation of the exponentiated Jensen-Renyi kernel-2: simplified.	February 8, 2015, 14:04:27
0.60	Quick test on the Tsallis divergence: introduced. Pearson chi square divergence estimation in the exponential family (MLE + analytical formula): added.	June 3, 2014, 00:17:33
0.59	Adaptive partitioning based Shannon mutual information estimators: added. Quick tests: updated to handle the new estimators.	May 16, 2014, 22:13:48
0.58	3-way interaction indices based on the embedding of the (i) Lancaster interaction and (ii) 'joint - product of the marginals' signed measures to a RKHS: added. Quick tests: updated to cover the new estimators.	April 29, 2014, 22:08:50
0.57	Kullback-Leibler divergence estimation based on maximum likelihood estimation + analytical formula in the chosen exponential family: added. A new sampling based entropy estimator with KDE correction on the left/right sides: added. Quick tests: updated with the new estimators.	April 10, 2014, 18:35:22
0.56	Distribution regression (supervised entropy learning, aerosol optical depth prediction based on satellite images): added. MMD distance computation based on U-statistics, expected kernel: upgraded to cover new kernels (exponential, Cauchy, Matern, polynomial, rational quadratic, inverse multiquadratic).	March 27, 2014, 22:06:54
0.55	Shannon entropy and cross-entropy estimation based on maximum likelihood estimation + analytical formula in the chosen exponential family: added. Quick tests: updated with the new estimators.	March 8, 2014, 00:18:00
0.54	Renyi and Tsallis entropy estimation based on maximum likelihood estimation + analytical formula in the exponential family: added. Quick tests: updated according to the new estimators.	February 24, 2014, 18:28:32
0.53	f-divergence estimation based on second-order Taylor expansion + Pearson chi square divergence: added. Shannon mutual information estimation based on KL divergence: added. Quick tests: updated with the new estimators.	February 2, 2014, 14:04:19
0.52	Sharma-Mittal divergence estimation: added using (i) maximum likelihood estimation + analytical formula in the exponential family, (ii) k-nearest neighbors. Quick test for (i) Sharma-Mittal divergence, (ii) Shannon mutual information: added. Normal variables: added to the Pearson chi square divergence quick test.	January 10, 2014, 00:02:47
0.51	ITE has been accepted for publication in JMLR; citing information: added. Block-MMD (maximum mean discrepancy) estimator: added. 'Extreme large' k in kNN based estimators: an overflow issue discovered + corrected. +Some refactorization, documentation upgrade.	December 29, 2013, 17:36:00
0.50	Entropy and Kullback-Leibler divergence estimation based on power spectral density representation and Szegő's theorem: added. Different noisy examples have been added to the image registration quick test.	December 18, 2013, 20:53:28
0.49	MMD (maximum mean discrepancy) estimation based on U- and V-statistics: incomplete Cholesky decomposition based accelerations added. Refactorization; improved navigation in the documentation.	December 1, 2013, 16:44:15
0.48	Sharma-Mittal entropy estimation based on k-nearest neighbors (S={k}): added. maximum likehood estimation + analytical value in the exponential family: added.	November 11, 2013, 19:26:39
0.47	Chi-square mutual information estimation based on Pearson chi-square divergence: added. Shannon entropy estimation based on an alternative linearly corrected spacing method: added.	November 1, 2013, 19:29:12
0.46	Phi-entropy (f-entropy) estimation based on the spacing method: added. Pearson chi square divergence (chi square distance) estimation based on k-nearest neighbors: added.	October 21, 2013, 19:31:52
0.45	Exponentiated Jensen-Tsallis kernel-1 estimation based on Tsallis entropy: added. Exponentiated Jensen-Tsallis kernel-2 estimation based on Jensen-Tsallis divergence: added.	October 9, 2013, 21:44:21
0.44	Exponentiated Jensen-Renyi kernel-1 estimation based on Renyi entropy: added. Exponentiated Jensen-Renyi kernel-2 estimation based on Jensen-Renyi divergence: added.	October 1, 2013, 18:16:38
0.43	Exponentiated Jensen-Shannon kernel estimation: added. Jensen-Tsallis kernel estimation: added.	September 20, 2013, 19:56:45
0.42	High-level information theoretical estimators: 'eval' changed to 'function handles' -- speeds up computations. Cost object initialization: now allows setting field values (alpha, number of kNN-s, ...) through its argument. => possibility to override default values, automatic inheritence in meta estimators. Quick tests introduced: consistency of the estimators (analytical vs. estimated value), positive semi-definiteness of Gram matrices determined by distribution kernels, image registration. Refactorization; documentation: improved.	September 7, 2013, 16:46:42
0.41	Probability product kernel estimation based on k-nearest neighbors: added, Jensen-Shannon kernel estimation: added.	July 12, 2013, 21:33:42
0.40	Bhattacharyya kernel estimation based on k-nearest neighbors: added, Expected kernel estimation: added, Kernel on distributions (K) object type: added.	June 23, 2013, 13:13:27
0.39	Symmetric Bregman distance estimation based on nonsymmetric Bregman distance: added, Symmetric Bregman distance estimation using the k-nearest neighbor method: added.	June 12, 2013, 13:12:52
0.38	Jensen-Tsallis divergence estimation: added, Bregman distance estimation: added.	June 1, 2013, 10:20:00
0.37	K divergence estimation: added, L divergence estimation: added, kNN squared distance computation: refined.	May 12, 2013, 15:35:39
0.36	Jensen-Rényi divergence estimation: added, Jensen-Shannon divergence estimation: added.	April 26, 2013, 18:45:27
0.35	An alternative Jacobi optimization based ICA solution with general entropy/mutual information estimators: added; The method extends the RADICAL ICA scheme to general objectives.	April 2, 2013, 10:37:51
0.34	Jacobi optimization based ICA solution with general entropy/mutual information estimators: added. The method extends the SWICA scheme to general objectives.	March 22, 2013, 11:41:05
0.33	Two one-dimensional Shannon entropy estimators based on the maximum entropy method: added.	March 6, 2013, 09:44:47
0.32	ICA and ISA structures: introduced for unified treatment of the estimators. It will also enable embedding of general ICA optimization algorithms such as the Jacobi method. 'stepwiseLS' mAR estimator: deleted. 'kdpee.c': MSVC does not provide log2. A more elegant solution: added.	February 25, 2013, 12:42:40
0.31	EASI (equivariant adaptive separation via independence) real/complex ICA method: added. Adaptive (k-d) partitioning based Shannon entropy estimation: added.	February 9, 2013, 11:31:39
0.30	Upper tail dependence via conditional Spearman's rho: added. Multivariate conditional version of Spearman's rho weighting the lower tail: added.	January 25, 2013, 15:11:25
0.29	Lower tail dependence via conditional Spearman's rho: added. Multivariate conditional version of Spearman's rho weighting the lower tail: added.	January 13, 2013, 11:38:17
0.28	Multivariate extension of Blomqvist's beta (medial correlation coefficient): added. Average pairwise Spearman's rho: added.	January 2, 2013, 22:51:22
0.27	Approximate correntropy independence measure estimator: added. Correntropy induced metric, centered correntropy induced metric estimators: added. Correntropy, centered correntropy, correntropy coefficient estimators: added. Handling of identically constant random variables in distance correlation computation: included.	December 28, 2012, 14:41:57
0.26	Distance covariance estimation via HSIC (Hilbert-Schmidt independence criterion): added. Energy distance estimation via MMD (maximum mean discrepancy): added. Energy distance estimation: added. We computed the square of distance correlation: sqrt included.	December 22, 2012, 14:12:33
0.25	Distance covariance, distance correlation estimation: added.	December 15, 2012, 10:20:06
0.24	MMD (maximum mean discrepancy) estimation based on U- and V-statistics: added.	December 12, 2012, 12:16:05
0.23	Three multivariate extensions of Spearman's rho (Spearman's rank correlation coefficient): added. Association (A) cost object type: added.	December 7, 2012, 18:33:26
0.22	Cauchy-Schwartz and Euclidean distance based divergence estimators: added. Cauchy-Schwartz and Euclidean distance based quadratic mutual information estimators: added.	December 1, 2012, 13:33:31
0.21	Kullback-Leibler divergence estimator based on cross-entropy and entropy: added. Cross-entropy estimation based on k-nearest neighbors: added. Cross cost object type: added.	November 25, 2012, 20:56:32
0.20	Two Shannon entropy estimators based on the distance (KL divergence) from the uniform/Gaussian distributions: added. Shannon entropy estimator based on Voronoi regions: added.	November 21, 2012, 13:55:09
0.19	Two k-nearest neighbor based Kullback-Leibler divergence estimators: added. compute_CDSS.cpp: 'sqrt(T)' -> 'sqrt(double(T))', to increase compatibility with compilers.	November 21, 2012, 13:47:05
0.18	8 sample spacing based 1d Shannon/Rényi entropy estimators: added.	November 10, 2012, 12:53:37
0.17	Edgeworth expansion based Shannon entropy estimator: accelerated (C++ alternative). 'Tsallis entropy <- Renyi entropy' meta estimator: added.	November 6, 2012, 22:18:43
0.16	Edgeworth expansion based Shannon entropy estimator: added. Lookup table for the underlying H/I/D estimation formulas: added (documentation).	November 2, 2012, 16:03:25
0.15	The Hellinger and Bhattacharyya distances are now available in ITE. They can be estimated via k-nearest neighbor methods. A '/'->'*' typo: corrected in 'DL2_kNN_k_estimation.m'.	October 29, 2012, 17:27:28
0.14	Monte-Carlo simulation to compute the additive constants in Rényi entropy estimation: added. some accelerations/compatibility enhancements: performed.	October 29, 2012, 12:31:34
0.13	Tsallis entropy is now available in ITE. It can be estimated via k-nearest neighbors. A '/'->'*' typo: corrected in 'HRenyi_kNN_k_estimation.m'.	October 27, 2012, 22:14:18
0.12	Schweizer-Wolff's sigma and kappa: added. Hoeffding's Phi computation: scaled-up.	October 27, 2012, 21:21:27
0.11	multivariate version of Hoeffding's Phi: added.	October 20, 2012, 23:30:17
0.1	Initial Announcement on mloss.org.	October 11, 2012, 07:47:43

Comments

No one has posted any comments yet. Perhaps you'd like to be the first?

Leave a comment

You must be logged in to post comments.