As more and more home devices communicate across the Internet, security should be a concern. When there are hundreds of devices in a home, it may be extremely easy to correlate bits of information from multiple devices and rapidly make inferences about the home or the owners. No-one wants people to know when they are out of the home, or what devices they own, in case it attracts burglars for example.
This post addresses just one tiny aspect of the broader security topic, which is the communication between the device and the remote application which could be in the cloud. Huge, complex solutions (like the Death Star!) all have weaknesses, so the entire solution needs to be considered, not just HTTPS. However when developing with boards containing Ethernet or 802.11 (Wi-Fi) access, it is worth double-checking if HTTPS is supported, especially if your application may transfer status or sensor information that could be misused by others. Some very popular bits of hardware (should we name-and-shame?) do not support secure communication.
HTTPS provides one way of ensuring (to a degree) secure communication for solutions that need to communicate over the Internet. A secure ‘session’ is established and then the HTTP content is encrypted/decrypted at both sides. A session can be considered to be like a temporary pipe set up between two applications, and any information sent through the pipe will be encrypted. The way the session is established is based on a protocol called Transport Layer Security.
This post contains a summary of SSL/TLS which is the underlying protocol which allows HTTP to be carried securely. The information here is believed accurate, but is merely a short summary and there may be inaccuracies but also large chunks of missing relevant information. This is a ‘first stab’ and I hope others more knowledgeable can share their information so that we as engineers can create more secure solutions from the ground up. If you do spot mistakes or have contributions, please do reply or contribute.
It is hoped that this post does provide a reasonably approximate idea of how secure communication is possible using HTTPS.
The entire process to communicate securely using HTTPS involves a number of steps. The text below considers the client to be the user’s application (e.g. a web browser, or an embedded application with Internet connectivity). The server is considered to be a remote application such as a Google or Yahoo application.
The entire process involves these steps listed here as a reference (they are described later):
- Certificate authority's certificate (containing CA public key) is stored in client and server
- Server generates a certificate request
- CA sends a signed certificate to the server
- Signed certificate transfer from server to client
- Signed certificate verification by the client
- 'Pre-master secret' transfer from client to server
- 'Master secret' generation by both parties
- Bidirectional encrypted data transfer using the master secret
An entity that can be believed (e.g. belonging to a company like Verisign) is called the certificate authority (CA). It is used by other devices to obtain so-called ‘digital signatures’ which can prove the identity of the devices, for the benefit of other devices that also trust the certificate authority.
This whole signature concept revolves around the use of hidden and visible data, known as private and public keys. Keys are strings of bytes that are used for signing/verification , but can also be used for encryption/decryption. Both are described here.
For encryption/decryption, in essence a public key allows encryption and a private key allows decryption. Unless the private key is known, decryption is not possible. At conception, the certificate authority generates a pair (i.e. a public and a private key), and never reveals the private key.
For the purposes of signing/verification, the process is similar but subtly different. For signing, the private key is used to produce an output that can only be decrypted using the public key. Again, the authority never reveals the private key. Another concept called a hash is also used for signing/verification. The process is explained later but in summary a hash algorithm is a formula (the formula doesn’t rely on private or public keys, i.e. the formula is commonly known and fixed) that can convert a large block of data into a smaller block (known as the hash; it is basically a checksum). It is considered impossible to convert the output of a hash algorithm back into the original data.
Prior to establishing device-to-device communication, each device needs to obtain the public key of the certificate authority. This can be obtained from the CA by many methods, e.g. via e-mail. It is assumed that the devices trust the CA.
Any device that plans on later establishing its identity to other devices will need to generate its own private/public key pair too, and then send the public key to the CA in a format known as a certificate request. This file (or data stream, depending on how it is sent to the authority) contains the name of the device (and maybe other miscellaneous information) and the device’s public key. The authority uses the hash algorithm to generate the irreversible small block of data, and then proceeds to encrypt it using the private key (see diagram below). The data returned back to the requesting device is known as the signed certificate. This data contains the encrypted hash.
When inter-device TLS communication is to be established to this device (which will be known as the server), the client sends a ‘hello’ message to the server. The server responds with a hello too, and the signed certificate, which is the encrypted hash and the original data (the name of the device, the server’s public key and other miscellaneous information). The client also runs the same hash algorithm on this original data. It then proceeds to decrypt the encrypted hash using the public key of the certificate authority. If the two hash values match, then it proves that the ‘original data’ is unchanged compared to the data that the authority saw and signed (see diagram below). It still doesn't prove the server owns the certificate; that comes next.
For TLS secured encryption/decryption of arbitrary data (in both directions) such as HTTP content, the public/private key method is not used. Instead, a single ‘symmetric key’ method is used. A symmetric key can be used to encrypt/decrypt, and is only secure as long as a third party is not aware of the symmetric key. Therefore, the symmetric key is shared in an encrypted form between the two allowed parties using a public key. In actual fact, the item that is shared is a ‘pre-master secret’ and that is used to compute the actual symmetric key (known as the ‘master secret’).
In other words, the pre-master secret is sent to the server from the client, but encrypted using the server’s public key (which was obtained from the certificate earlier). It is also encrypted for another reason; if the server can successfully decrypt the secret using its private key, then in terms of the bigger picture, it means that the client is assured that the server can be trusted, since the certificate authority trusted the server in some manner (e.g. the CA may have manually verified the e-mail address of the server), in order to give it the signed certificate in the first place.
Both sides to the HTTPS session will convert the pre-master secret into the master secret. To do this, the pre-master secret is combined with some random numbers that were exchanged by both devices during the hello phase, and then fed into an algorithm which generates another random string of bytes. In other words, master secret = PRF(pre-master secret + randomClient + randomServer) where PRF is the algorithm (pseudo-random function).
This is done because the server may not trust the client sufficiently to allow it to generate a symmetric key on its own. For example, the client (e.g. a low-performance device) may implement a poor random number generator. With this PRF function, both sides of the TLS connection are assured of the quality of the symmetric key, and both sides contributed some random values which will prevent playback attacks.
Here is a summary of the popular algorithms use in TLS (other algorithms can be used too).
The private and public keys are used to encrypt/decrypt and sign/verify using the RSA algorithm. The hash method used as part of the signing/verification process is actually two algorithms combined; one is MD5 (popular with checksums) and the other is SHA-1 (also sometimes used for file checksums). The later TLS version 1.2 uses SHA-256 instead. Once the symmetric key has been established, it is used with an encryption/decryption algorithm (also known as a cipher) called AES in order to encrypt/decrypt the traffic (e.g. HTTP messages).
The device initiating the TLS call creates a TCP connection, and then begins the TLS handshake procedure, as shown here. Note that there are additional messages possible, but these are the common ones. The diagram here shows a ‘unilateral authentication’ procedure. Unilateral authentication means that the server has established to the client its identity beyond doubt. Mutual authentication means that the server can request the client to establish its identity too.
The ClientHello message contains just a random string of 28 bytes, and the list of ciphers that the client is happy to use once the TLS connection is complete. It also contains the TLS version number (e.g. version 1.2 is encoded in two bytes, as 03 03). Optionally, there may also be a compression method defined but it can be set to null.
The ServerHello message also contains a random string of bytes and the chosen cipher name. It is followed by the certificate, and ServerHelloDone messages. Although these are three messages, they may be combined into one or two IP packets.
The certificate is a copy of the server’s signed certificate. If you are curious this can be visually identified by checking the packet for the certificate’s serial number (8 bytes), which will not be encrypted. The certificate also contains the public key of the server (unencrypted of course). The Certificate is in ITU format X.509, which is an ASN.1 format not human-readable, but if you’re interested is viewable with software like Wireshark.
Once the certificate has been sent, the server sends a ServerHelloDone message. Note that in a mutual authentication scenario, the server would have sent a CertificateRequest message instead, at this point.
The ServerHelloDone message indicates that the server will now wait for a response from the client. The message contains no parameters.
The ClientKeyExchange message simply contains the pre-master secret in an encrypted form (i.e. encrypted using the server public key). The pre-master secret is 48 bytes in length, but after encryption it will be a larger size.
Both sides will use the decrypted form of this, plus the random values from the ClientHello/ServerHello messages to construct a master secret value.
This message has no content, and just signifies that subsequent messages will be encrypted using the cipher.
Finished (Encrypted Handshake) message
The message (displayed as ‘Encrypted Handshake message’ using wireshark) is actually not a message name. Since this content is now encrypted using the cipher, wireshark will not reveal it but actually it will be a ‘Finished’ message which contains some verification data composed of a hash of the master secret and some other values. Both sides will exchange the Finished message (in encrypted form) after they have both exchanged ChangeCipherSpec messages. If both sides receive what they expected, then the TLS connection can now be used for sending/receiving encrypted data.
This is the traffic (e.g. HTTP) encrypted by the cipher.
It is hoped that the information here provides enough detail for there to be an awareness of how HTTPS works, why you might want to use it. It is hoped that more people make a judgement and make use of HTTPS where needed, and vote with their wallet and adopt hardware that supports secure procedures.
RFC 2246, RFC 4346 TLS version 1.0, 1.1
RFC 3268 AES ciphersuites for TLS
‘Digital signature’ article on Wikipedia was the source for the diagrams