Entropy - Encryption, Compression, etc.

Entropy in computer science can be as simple as counting how many times each character appears in a file. If the file is written in plain English sentences, there are predictable percentages for which certain letters will appear. For example the letter 'e' will appear way more often than the letter 'z' when viewing a written piece of work. Now what gets interesting is that you can use tools to calculate entropy and also be able to determine whether a file is for example compressed, or even encrypted. How? Let's see.

Download an entropy calculating tool such as binwalk

Run that tool against a file
   binwalk -E

Notice in the example below that the entropy is pretty flat, but has consistent deviations. It's a fare assumption to make that this file might be compressed because of that layout seen. Actually this file was a '.zip' compressed file.

Notice in the example below that the entropy is pretty flat, flatter than the previous. It's a fare assumption to make that this file might be encrypted because of that layout seen. Actually this file was a '.gpg' encrypted file.

Notice in the example below that the entropy is all over, no flatness at all. It's a fare assumption that this is not encrypted or compressed, but instead some more standard file or text. Actually this file was '.bmp' image file.

Finally notice in the example below that the entropy is also all over, no flatness at all. It's a fare assumption that this is not encrypted or compressed, but instead some more standard file or text. Actually this file was '.html' web page with the text from a story.

Why is this useful? In computer science, and especially in the information security realm, you run across unknown data all the time. It becomes very helpful then to learn whether that data is perhaps encrypted, or compressed, so that you can make attempts to reverse engineer it and get back to the original malicious content.

Copyright © 2015, this post cannot be reproduced or retransmitted in any form without reference to the original post.