Merlin v0.8.0 Released

This post covers the release of Merlin v0.8.0 that includes several new features to increase Operations Security (OPSEC) and usability. One of the more notable features was the introduction of the augmented Password Authenticated Key Exchange (aPAKE) OPAQUE protocol. This protocol was discussed in detail in THIS post. Other key updates in this release include:

• Go’s Gob network traffic encoding
• JSON Web Tokens (JWT) for authorization
• JSON Web Encryption (JWE) for payload formatting
• HTTP/1.1 Support
• Proxy support
• Host header modification
• Go Modules

GOB Encoding

In order support a more efficient message encoding for network traffic, Merlin has switched from using JSON to Gob encoding. The Go language offers the gob message encoding as an efficient way to encode network traffic. This works well when both the client and server are written in Go but not when one of the components isn’t. All Merlin messages are now gob encoded into a byte array that is later sent across the network. Because of this, the HTTP traffic is now using the application/octet-stream content-type header. Additional information about the Gob encoding can be found here: https://blog.golang.org/gobs-of-data .

JWE

A Gob encoded Merlin message is then encrypted into a JSON Web Encryption (JWE) object using the symmetric key derived during OPAQUE authentication. The JWE compact serialization format is composed of five main parts: the header, the JWE encrypted key, the initialization vector, the cipher text (encrypted message), and the authentication tag.

The JWE header describes how the message is encrypted by defining the key management algorithm (alg) and content encryption type (enc). JWE message are encrypted into cipher text with a randomly generated per-message Content Encryption Key (CEK) using the specified content encryption type (enc). Merlin uses AES in Galois/Counter Mode (GCM) with a 256-bit key to encrypt messages.

The Merlin implementation uses the PBES2_HS512_A256KW key management algorithm (alg) to encrypt the CEK itself. This algorithm leverages Password Based Encryption Standard 2 (PBES2) with HMAC SHA-512 and AES 256 key wrapping. PBES2 uses the PBKDF2 hashing algorithm to generate an AES256 key encryption key. The password, in combination with a salt, is run through the PBKDF2 hashing algorithm for 500,000 iterations. When the PBES2 algorithm is used, the salt is passed in the JWE header in the p2s field and the number of PBKDF iterations are sent in the p2c field. For Merlin, the password is the symmetric key derived during OPAQUE authentication. The output from the PBKDF2 hashing algorithm is used to encrypt the Content Encryption Key (CEK), known as the JWE encryption key. The figure below illustrates an example of a JWE header.

With all of these pieces of the JWE determined, the Merlin message can be encrypted and then transformed into the compact serialization format as a URL safe string (see figure below). The string is again Gob encoded and is ready to be sent from the client to the server.

JWT

Previous version of Merlin did not have any type of message authentication or authorization. This allowed anyone to send a crafted Merlin message (a JSON structure) to the server and it would attempt to process it. The integration of the OPAQUE protocol provided authentication but did not provide message authorization. To combat this, JSON Web Tokens (JWT) were implemented in the HTTP Authorization header. The Merlin server will now return a 404 for all messages that do not contain a JWT.

When an agent is first executed, it will generate a JWT using the Pre-Shared Key (PSK). By default the PSK is merlin but it can be changed at run time with the -psk command line parameter. This JWT only allows the agent to send OPAQUE registration and authentication messages to the Merlin server. All other message types will receive a 404 from the server.

After the agent has completed OPAQUE authentication, the Merlin server will create a JWT using a randomly generate key created at run-time that is only used with JWTs. This generated key is not used for anything else. The JWT is created with a limited lifetime and returned to the agent in the last OPAQUE authentication message from the server to the agent. From this point forward, the Merlin agent will use this JWT signed by the server’s key and is now able to send any Merlin message type. It is important to note that the JWT itself is actually a JWE and that the agent is not capable of reading or decrypting it.

The JWTs created by the Merlin server have a very limited lifetime. The JWT’s lifetime is calculated by adding the agent’s sleep time to the maximum possible skew value. This value is then multiplied by the agent’s max retry value. For an agent using default values this works out to (30 seconds + 3 seconds) * 7 = 3 minutes 51 seconds. If for some reason the Merlin server receives a message from an authenticated agent with an expired JWT, the server will instruct the agent to re-authenticate.

HTTP/1.1

Merlin was first developed to exclusively use the newer HTTP/2 protocol and explicitly prevented the use of HTTP/1.1. This worked well to illustrate the offensive advantage of using a protocol that many tools were not equipped to decode or handle. However, it also introduced a couple of disadvantages.

Some web proxies will terminate a HTTP/2 connection and then re-establish a new HTTP/1.1 connection with the Merlin server. Because Merlin did not allow HTTP/1.1, the agent would be unable to communicate with the server. This behavior was also seen in Content Delivery Networks (CDNs). The CDN would allow a HTTP/2 connection to its internet facing endpoint but would only establish a HTTP/1.1 connection to the backend “origin” server. This precluded an operator’s ability to use a CDN.

This release allows the operator to force the use of a HTTP/1.1 connection when executing the agent with the -proto https command line flag. Additionally, the agent protocol can be hard coded at compile time using the Make file with PROTO=https argument (i.e. make agent-windows PROTO=https). There is not a HTTP/1.1 option for the server because the -proto h2 option includes both HTTP/1.1 and HTTP/2. This allows an operator to have one agent connecting over HTTP/2 and another over HTTP/1.1on the same listener.

With the ability to use a CDN, this release of Merlin includes the ability to set an HTTP Host header at run time with the -host www.google.comcommand line flag or the HOST=www.google.com compile time flag when using the provided Make file. This is a significant advantage for operators as it enables the ability to leverage the Domain Fronting technique.

Another problem is that HTTP/2 does not allow the ability to explicitly configure a connection to use a web proxy like HTTP/1.1. This meant operators using Merlin on a network that required all web traffic to go through a web proxy had no way to force the agent to use it. This turned out to be a significant problem given the number of organizations using a web proxy. This release provides an operator the ability to explicitly set a proxy using the -proxy https://192.168.1.1:8080 command line argument. Alternatively, it can be set at compile time with PROXY=https://192.168.1.1:8080 when using the Make file. You must specify the https (HTTP/1.1) protocol if you’re configuring the agent to use a web proxy. By default, Merlin will use the HTTP/2 protocol if you do not change it. However, Merlin does not automatically detect system proxy setting leaving a gap for operational use.

Go Modules

By default, Go will download supporting libraries directly from their source repository. This can be a problem for stability if a package maintainer changes code in a way that is not backwards compatible. To circumvent this, previous versions of Go used Vendoring by including a copy of the supporting libraries in the project’s root. Go would always search the vendor first before searching in other paths for the library. The downside to this is that you always have to ship libraries around with your repository and always requires you to manually update the library even when there are not any breaking changes.

Go version 1.11 includes opt-in support for using modules. Go modules are the new way to manage library dependencies and provide granular control. When using modules, a go.mod file is kept in the project’s root and contains a list of all supporting libraries. The unique part is that the package maintainer has full control over which version to use and even allows the use of multiple different version. If the supporting library doesn’t use semantic tagged versions, a commit hash can be used to specify the exact instance of the library. Additional details about Go modules can be found here: https://blog.golang.org/using-go-modules.

This is important to anyone that wants to build Merlin from source. Before building, you must set the GO111MODULE environment variable to on (i.e. export GO111MODULE=on) to signify the use of the opt-in feature. The Go compiler will throw multiple “cannot find package” errors if you forget to set the environment variable. If you’re using the Make file to generate Merlin binaries, it already sets the environment variable for you.

Wrap Up

This release of Merlin brings along several new features that aim to increase security and effeciency. JWE messages are used to transport encrypted messages across the network using Go’s gob encoding. The addition of the Authentication header and JWT implementation helps to only allow authorized messages to be processed. Supporting the HTTP/1.1 protocol allows an operator to set a Host header for domain fronting or to specify a network proxy.

Happy Hacking