Remote code execution on Sqreen: exploiting the microagent

Polygonal Background
19 November, 2020
  • Posted By Charles Fol
  • sqreen rce heap overflow rules js microagent
Full Article

Note from Jb Aviat, CTO @Sqreen

When Charles reached out to me to disclose this issue, we decided to react with one goal in mind: protecting our customers. Hence we built a disclosure schedule and reported the issue privately to our impacted users. After a month, we officially created the CVE and shared details about how we fixed this issue in this article. As a security nerd, I’m happy to invite Charles to disclose technical details to an elaborate, complex but elegant exploit he wrote before he disclosed the vulnerability. As a CTO, I’m learning the lessons with humility and improving what needs to be in our way to approach secure development. That’s also the right time to combine our proactive defensive measures with more offensive approaches, by launching a formal bug bounty program, that we will detail in an upcoming article.

I am a Sqreen customer - am I safe?

TL; DR: Yes.

Sqreen communicated with all their impacted users starting in July 2020 to help them upgrade their PHP agent, prior to releasing any public announcement on this. Most Sqreen customers updating their agent regularly were already protected.

The vulnerability only applies to Sqreen’s PHP agent for versions up to, and including 1.15.1 (released Dec 2019). The identified issue does not affect PHP agents 1.16.0 and later (released on or after April 2020) as this issue was corrected before they were aware of the problem, as part of a separate, non-related, engineering improvement.

Furthermore, we conducted a 20 days penetration test with Sqreen to dive deeper in their agents and platforms, which revealed an excellent level of security.

You can find more information on Sqreen blog here.

Introduction

While trying to exploit a simple SQL injection during a security assessment for Ambionics, we got repeatedly blocked by Sqreen, despite being as creative as possible with our payloads. This sparked our curiosity: why weren't we able to bypass this protection?

Contrary to Web Application Firewalls (WAF) that act as a front line, Sqreen is plugged deeper in your application: when a sensitive function is called, its arguments will be processed by Sqreen first, in order to determine if your intent is malicious or not. This means Sqreen will understand whether or not you're trying to attack, instead of using patterns to ban anything that looks like an attack. As such, changing the looks of our payload was no use.

Therefore, we tried to have a look at what Sqreen was doing behind the scenes, in the hope of bypassing its protection. However, as security researchers, we could not resist the urge to look for vulnerabilities in the product itself. We eventually spotted 3 bugs that, chained, lead to a heap buffer overflow which resulted in remote code execution.

We reported them to Jb Aviat, Sqreen CTO, who took the matter very seriously. Sqreen even proposed the idea of doing a blog post describing the issues and the exploitation. Every vulnerability that is described in this article has been fixed in a prompt manner by Sqreen, and their clients are now safe.

Agent, daemon, and backend

Note: this is an oversimplification of Sqreen's functioning. Furthermore, some parts are specific to Sqreen's PHP agent. For a better understanding of Sqreen, please refer to other blog posts, written by Sqreen employees, such as Building a dynamic instrumentation agent for PHP.

For an attacker, Sqreen can be divided into three blocks. First, the backend is the web interface through which clients monitor their applications. There, you can see which attacks are happening, pick the type of vulnerabilities you want to monitor, and how to handle attackers, all in real-time. For instance, you can choose to disable the protection against SQL injections, and enable the protection for Cross-site Scripting (XSS) vulnerabilities.

On your server runs the Sqreen daemon, which communicates with the backend using an HTTP API to receive its configuration as a set of rules (more on that later), and send back live data.

Finally, when your monitored application is started, a library, called the Sqreen agent, is loaded. This agent will ask the daemon what behavior it should monitor, then it will hook sensitive functions and methods. When a hooked function is called by your application, the agent extracts its arguments, along with input parameters (GET, POST, etc.), and sends them for processing to Sqreen's daemon. The daemon will then decide if it is safe to call the function, and choose an action: allow the request, block it, or even ban the user. The decision will be forwarded back to the agent, while logs are sent to the backend.

The agent and daemon communicate using MessagePack through a TCP socket spawned by the daemon. Here lies the first problem: the daemon listened on 0.0.0.0:7773 by default, and did not implement any kind of authentication. As such, one could connect to the daemon directly from the internet (or use an SSRF vulnerability if the daemon is not reachable directly) and mimic an agent.

When it connects to the daemon, the agent sends info to identify itself, such as its application ID, its hostname, version of PHP, running modules, etc. Along with this information, the agent can specify an alternative backend URL for the daemon to use. However, Sqreen checks the backend HTTPS certificate, to prevent a fake agent from specifying a fake backend URL. This could be bypassed by sending an HTTP URL instead of an HTTPS URL, and as such avoided the verification of the certificate.

This means that an attacker could force the daemon to use an arbitrary backend.

As a result, an attacker was in a position where they could make the daemon interact with a fake agent, and a fake backend server.

Playing around the rules

Let's dive a little more into the aforementioned rules. Rules are sent by the backend to the daemon. A rule will tell the daemon what function to monitor, and how to determine if its arguments are safe. There are different types of rules: a (very) simple example could be a rule that monitors calls to fopen(), and checks if the first argument contains ../.

Rules often need to handle complex input. If PDO::query() is called, for instance, the SQL query needs to be parsed and tokenized in order to find out if someone is trying to inject something. To handle those cases, the daemon is shipped with a v8 JS engine. Some rules, named JS rules, contain a piece of Javascript that will be evaluated in v8 by the daemon. Since this is equivalent to running code (albeit Javascript) on the daemon's machine, the rules are sent with a signature. This signature is computed using Sqreen's private key, stored offline.

In theory, if the signature for the rule is wrong, the rule should be ignored. However, the code responsible for checking the signature had a little, dramatic bug.

class RemoteCommand:
    def instrumentation_enable(self, runner, client, params=None):
        """ Enable instrumentation """
        ...
        for rule in params[1]:
            if verifier:
                LOGGER.debug('Check signature for rule: %s', rule.get('name'))
                verifier.verify_rule(rule, signature.REQUIRED_KEYS, signature.SIGNATURE_VERSION)
            ...


class RuleSignature(Signature):
    """ Helpers for signing and verifying rules signatures """
    __metaclass__ = ABCMeta

    def verify_rule(self, hash_rule, keys, version):
        """
        Return True if signature is correct and False otherwise.
        """
        ...

In RemoteCommand, instrumentation_enable() calls verify_rule(), expecting it to throw an exception if the signature is invalid. However, RuleSignature.verify_rule() will never raise an exception: it will return false.

This is an easy mistake to make, but it has terrible consequences. As an attacker, since we force the daemon to connect to our fake backend, we can send fake JS rules, which will be run in the v8 engine.

This is a good step towards code execution. However, we're now executing JS in a raw v8 engine, which is far from having a shell. Being unfamiliar with v8 exploitation, it seemed like a hard task. However, another bug came to the rescue.

PyMiniRacer: heap overflow

After the JS snippet is executed, its return value has to be converted to Python. PyMiniRacer is a python library developed by Sqreen that runs Javascript code and converts the result back to Python. To do so, it converts the JS values into a C structure named BinaryValue (abbr. BV), and then uses ctypes to map BinaryValues to Python objects. The code responsible for converting JS arrays into BVs was the following:

struct BinaryValue {
    union {
        BinaryValue **array_val;
        BinaryValue **hash_val;
        char *str_val;
        uint32_t int_val;
        double double_val;
    };
    enum BinaryTypes type = type_invalid;
    size_t len;
};

static BinaryValue *convert_v8_to_binary(ContextInfo *context_info,
                                         Handle<Value> &value)
{
    ...

    else if (value->IsArray()) {
        Local<Array> arr = Local<Array>::Cast(value);
        size_t len = arr->Length();
        BinaryValue **ary = xalloc(ary, sizeof(*ary) * len); // [1]

        res->type = type_array;
        res->array_val = ary;

        for(uint32_t i = 0; i < arr->Length(); i++) { // [2]
            Local<Value> element = arr->Get(i);
            BinaryValue *bin_value = convert_v8_to_binary(context_info, element); // [3]
            if (bin_value == NULL) {
                goto err;
            }
            ary[i] = bin_value;
            res->len++;
        }
    }
    else if (value->IsObject()) {
        res->type = type_hash;

        TryCatch trycatch(isolate);

        Local<Object> object = value->ToObject();
        MaybeLocal<Array> maybe_props = object->GetOwnPropertyNames(context);
        if (!maybe_props.IsEmpty()) {
            Local<Array> props = maybe_props.ToLocalChecked();
            uint32_t hash_len = props->Length();

            for (uint32_t i = 0; i < hash_len; i++) {
                Local<Value> pkey = props->Get(i);
                Local<Value> pvalue = object->Get(pkey);
                // store in local dict
                ...
            }
        } // else empty hash
    }

    ...
}

If the JS array is of size N, a C array of N BV pointers is allocated (ary) [1]. Then, the code converts each value of the array to a BV [3], and stores it in ary. Those familiar with JS, or even scripting-language exploitation in general, may have spotted the bug: a fixed size structure is allocated, but in the for loop, the stop condition compares i to arr->Length(), which is a dynamic value [2]. If the size of the array were to be incremented, the excess values would overflow, and overwrite the next heap chunk. Here's a POC:

var array = [
    {
        get first() {
            for(var i=0; i<100; i++) {
                array.push(0x41);
            }
        }
    }
];

// when accessed, the only element of the array will make the array grow by 100 items.
return array;

When trying to convert the only element of array to a BinaryValue, its properties will be read. Since its only property is in fact a getter, the Javascript code it contains will be run. We can use this to increase array's length.

We now have a heap overflow, but of pointers. Furthermore, the exploitation needs to take 3 unknowns into the equation: Python's version, v8’s version, and the libc's...

Pointer overflow

We can send several JS rules, which will all be evaluated in the same JS context.

Every time a JS rule is run, the following happens:

  • The JS code is evaluated
  • Its return value is converted to a BinaryValue (BV) using convert_v8_to_binary()
  • The BV is converted into a python variable using ctypes
  • The BV is freed.

Pros and cons

There are a few things to keep in mind when attempting to exploit this bug:

Pros:

  • Global Javascript variables are kept across rules execution
  • As shown in the POC, we can run code in between the creation of each BinaryValue

Cons:

  • We have no knowledge of the Python, v8, and libc version
  • This is a pointer overflow: we can't simply overwrite chunk data easily
  • JS objects are allocated and freed on another heap, so irrelevant
  • JS ArrayBuffers backing stores are allocated in the main thread, but freed from another
    • This will mess with the tcache
  • Lots of big chunks get allocated in between two JS executions, meaning malloc consolidation is frequent
    • This forces us to disregard fastbins

TL;DR

To exploit, we chose to make the BinaryValue array overflow into a JS ArrayBuffer. Using this ArrayBuffer, we were able to read and write BinaryValue pointers at will. We used it to create fake chunks, and make array elements point to them. This gave us an arbitrary free primitive. We then made the next pointer of a fastbin point to __malloc_hook's address and changed it to system.

General behavior

Let's take the following example:

return [
    3, {
        get trigger1() {
            // run js code
        }
    }, {
        get trigger2() {
            // run js code
        }
    }
]

This is an array of 3 elements, an int and three objects. When converting each object to a BinaryValue, its getter (triggerN) will be called.

This means we can run JS code before each BV pointer gets put in the array using the { get trigger() { <code> } } construct.

For exploitation, this is really useful: we can alternate between allocating BinaryValues and running JS code.

After the main BinaryValue has been converted to a Python value using ctypes, it is destroyed: each of its elements is destroyed recursively, and its allocated heap chunk is freed.

Exploitation

We refrained from manipulating Python objects as we had no experience doing so, and went with a classic libc exploitation.

We used three JS rules that we called one after the other.

Rule 1

The first one is very straightforward: it fills up heap holes and then creates 3 JS arrays of 0x1f elements. When those arrays get converted into a BV, 3 concurrent 0x100 chunks are created. After they are converted to Python values, they are freed, which results in those chunks ending up in the tcache.

Rule 2

We create an ArrayBuffer, which ends up in the first 0x100 chunk. It will stay allocated for the rest of the exploitation, and it will remain unused, as its sole role is to protect the third 0x100 chunk from malloc_consolidate()'s scrutiny.

We then create a JS array, overflowAry, of 0x1f elements, and return it. This fills the second 0x100 chunk with the array of BV pointers.

We make the last element of the array create another ArrayBuffer, overflowAB, which will be allocated in the last 0x100 chunk. We can then increase overflowAry size, so that the next values inserted in the BV pointers array (ary) overflow into overflowAB.

We want to create a huge ArrayBuffer chunksAB, fill it with fake heap chunks containing fake BVs, and make overflowAB's BV pointers point to them.

When the rule exits, those fake BVs will be freed. With another JS rule, we can control chunksAB's content and manipulate our (now freed) fake heap chunks. This gives us an easy arbitrary write using standard tcache exploitation techniques.

A problem remains: we need to know the address of chunksAB in order to point to its fake chunks. By reading overflowAB, we can learn the addresses of the BVs, not those of JS objects. To circumvent this, every BV that we create in the overflown part of the array (from 0x20 to 0x3f) has the same structure, which is detailed below. By calculating the offsets between each pointer in overflowAB, we can find two pointers that are exactly 0x4f0 apart; this means that in between them lies a chunkAB.

Rule 3

In this rule, we iterate through each chunksAB to find out which one had its fake chunks freed. Once it is found, we can read the contents of these chunks to find the address of the libc, and one chunk next pointer point to &__malloc_hook. We'll then change malloc_hook to system, which we can finally call by creating an ArrayBuffer whose size is the address of our command.

Demo

Here are the steps of the exploitation:

  • Fake agent: Connect to Sqreen's daemon, and make it connect to our fake backend
  • Fake backend: Send JS exploit rules to the daemon
  • Fake agent: pretend 3 sensitive functions were called to trigger the evaluation of the rules that the backend sent
  • Memory corruption triggered, reverse shell spawned.

Conclusion

A chain of vulnerabilities allowed us to get remote code execution on Sqreen-protected servers. Sqreen patched them as soon as possible, and made sure their clients were safe. We've made the JS exploit scripts available on Ambionics' GitHub repository. The CVE numbers are the following: