An Algorithm to Generate Probabilities with Specified Entropy

Om Parkash, Priyanka Kakkar

Department of Mathematics, Guru Nanak Dev University, Amritsar, India.

DOI: 10.4236/am.2015.612174                

Abstract

The present communication offers a method to determine an unknown discrete probability distribution with specified Tsallis entropy close to uniform distribution. The relative error of the distribution obtained has been compared with the distribution obtained with the help of mathematica software. The applications of the proposed algorithm with respect to Tsallis source coding, Huffman coding and cross entropy optimization principles have been provided.

Keywords

Entropy, Source Coding, Kraft’s Inequality, Huffman Code, Mean Codeword Length, Uniquely Decipherable Code

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	Shannon, C.E. (1948) A Mathematical Theory of Communication. Bell System Technical Journal, 27, 379-423. http://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x
[2]	Renyi, A. (1961) On Measures of Entropy and Information. Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, 1, 547-561.
[3]	Tsallis, C. (1988) Possible Generalization of Boltzmann-Gibbs Statistics. Journal of Statistical Physics, 52, 479-487. http://dx.doi.org/10.1007/BF01016429
[4]	Tsallis, C. (2009) Introduction to Nonextensive Statistical Mechanics. Springer, New York.
[5]	Papa, A.R.R. (1998) On One-Parameter-Dependent Generalizations of Boltzmann Gibbs Statistical Mechanics. Journal of Physics A: Mathematical and General, 31, 5271-5276. http://dx.doi.org/10.1088/0305-4470/31/23/009
[6]	Abe, S. (1997) A Note on the q-Deformation-Theoretic Aspect of the Generalized Entropies in Nonextensive Physics. Physics Letters A, 224, 326-330. http://dx.doi.org/10.1016/S0375-9601(96)00832-8
[7]	Oikonomou, T. and Bagci, G.B. (2010) The Maximization of Tsallis Entropy with Complete Deformed Functions and the Problem of Constraints. Physics Letters A, 374, 2225-2229. http://dx.doi.org/10.1016/j.physleta.2010.03.038
[8]	Bercher, J.F. (2008) Tsallis Distribution as a Standard Maximum Entropy Solution with “Tail” Constraint. Physics Letters A, 372, 5657-5659. http://dx.doi.org/10.1016/j.physleta.2008.06.088
[9]	Bercher, J.F. (2009) Source Coding with Escort Distributions and Renyi Entropy Bounds. Physics Letters A, 373, 3235-3238. http://dx.doi.org/10.1016/j.physleta.2009.07.015
[10]	Parkash, O. and Kakkar, P. (2012) Development of Two New Mean Codeword Lengths. Information Sciences, 207, 90-97. http://dx.doi.org/10.1016/j.ins.2012.04.020
[11]	Huffman, D.A. (1952) A Method for the Construction of Minimum Redundancy Codes. Proceedings of the IRE, 40, 1098-1101. http://dx.doi.org/10.1109/JRPROC.1952.273898
[12]	Baer, M.B. (2011) Redundancy-Related Bounds for Generalized Huffman Codes. IEEE Transactions on Information Theory, 57, 2278-2290. http://dx.doi.org/10.1109/TIT.2011.2110670
[13]	Mohajer, S., Pakzad, P. and Kakhbod, A. (2012) Tight Bounds on the Redundancy of Huffman Codes. IEEE Transactions on Information Theory, 58, 6737-6746. http://dx.doi.org/10.1109/TIT.2012.2208580
[14]	Walder, J., Kratky, M., Baca, R., Platos, J. and Snasel, V. (2012) Fast Decoding Algorithms for Variable-Lengths Codes. Information Sciences, 183, 66-91. http://dx.doi.org/10.1016/j.ins.2011.06.019
[15]	Swaszek, P.F. and Wali, S. (2000) Generating Probabilities with Specified Entropy. Journal of the Franklin Institute, 337, 97-103. http://dx.doi.org/10.1016/S0016-0032(00)00010-7
[16]	Kraft, L.G. (1949) A Device for Quantizing Grouping and Coding Amplitude Modulated Pulses. M.S. Thesis, Electrical Engineering Department, MIT.
[17]	Campbell, L.L. (1965) A Coding Theorem and Renyi’s Entropy. Information and Control, 8, 423-429. http://dx.doi.org/10.1016/S0019-9958(65)90332-3
[18]	Kolmogorov, A. (1930) Sur la notion de la moyenne. Atti R. Accad. Naz. Lincei, 12, 388.
[19]	Nagumo, M. (1930) Uber eine Klasse der Mittelwerte. Japanese Journal of Mathematics, 7, 71-79.