Understanding the Function Call Stack

There’s more than meets the eye under the function call hood

This post is based on a September 2021 Twitter thread that I wrote to describe the same concept regarding function calls and their hidden hierarchy. That thread was inspired by a series of tweets by inversecos who shared how malware authors will often use Native APIs instead of Win32 APIs as a mechanism to evade naive detections that assume every application will use the Win32 API function. I wanted to explain WHY that approach worked, and now I think it is appropriate to solidify my thoughts in the blog form.

Introduction

A common feature of Operating Systems is the need to provide a mechanism for developers to interact with the operating system. This feature typically manifests through something called an Application Programming Interface (API) which is composed of a series of “functions” that can be used by developers to perform common tasks. There are a few value propositions inherent in an API. First, the API function serves as an abstraction layer that simplifies complex tasks. It isn’t necessary for all developers to understand the complicated components of the file system (such as the Master File Table), how to determine where free space exists on the hard drive, and how to write data to the physical hard drive. This is all managed on behalf of the developer by the relevant functions. Second, the API acts to constrain the variability in how developers interact with the Operating System. The OS is a relatively sensitive system and the wrong changes to certain system settings can cause serious instability. For instance, many functions like Windows service creation require interacting with and changes to the Windows Registry. It can be dangerous to make direct changes to the registry because the OS has very specific expectations for how data is structured there and certain changes can cause a fatal system error resulting in a Blue Screen of Death (BSOD). To avoid the need for all developers to interact with and understand the structure of the Windows Registry, API functions provide an abstraction layer that manages that complicated problem for us.

How Functions Nest

In the context of Microsoft, there are actually multiple layers of functions that tend to nest or call each other. The most superficial layer or the layer that is intended for third-party developers to interact with is called the Win32 API. The cool thing about this API is that it is well documented by Microsoft meaning that it is easy to look up instructions on what a function does and how it should be called. Another guarantee that Microsoft gives is that documented functions will continue to work as expected in perpetuity (it might not literally be for perpetuity, but you can expect documented functions to work similarly for a long time). So generally speaking, the recommendation and desire from Microsoft is that third-party developers use these public or documented functions that are generally referred to as the Win32 API. However, these documented functions are not the only functions that exist. In fact, many documented functions actually call other documented functions or even undocumented functions behind the scenes.

Case Study — CreateFileW

In order to better understand the relationship between function calls or the nesting of function calls we should look into a case study of one commonly encountered Win32 API function, specifically CreateFileW.

Finding the Implementation via a DLL

The first step to digging into CreateFileW is to determine what Dynamic Link Library (DLL) implements this function. DLLs are binary files (files that implement code) that create shared functionality. The idea is that it doesn’t make sense for every programmer to reinvent the wheel when the task they are performing is common and predictable. Instead, that functionality can and should be integrated into DLLs which can then be referenced by applications. The Win32 API is implemented in a set of default or built-in DLLs, and functions that are meant for application consumption are marked as “exported functions”. To determine which DLL implements or exports CreateFileW we can reference the function’s public documentation. In the requirements section, we see that CreateFileW is implemented by kernel32.dll.

Once we determine which DLL implements the function we can investigate how the function works under the hood. One way to do this is to load the DLL into a disassembler like IDA or Ghidra. Once kernel32.dll is loaded into IDA and public symbols are applied we can click on the exports tab which has a list of all of the functions that are exported by the DLL. This list includes the friendly name of the function, the relative address of the function’s implementation, and the function’s ordinal number. We can search for CreateFileW and once we find it we can double-click on it in the list which will take us to the function’s implementation.

CreateFileW’s Code Implementation

When we arrive at CreateFileW’s implementation it immediately appears fairly underwhelming. This is the beginning of the function nesting that we alluded to earlier. Strangely, it appears that CreateFileW is calling itself, but it’s not quite that simple. Notice the notation __imp_CreateFileW, the __imp_ part indicates that this version of CreateFileW does not exist in the current DLL (kernel32.dll). Instead, this version of the function is actually being “imported” from a foreign DLL. Just as we discussed that DLLs export functions that are meant for other applications to use, the way a binary can use an exported function is through “importing” it as we see here.

The Import Table

We can view kernel32.dll’s Import Table to find the version of CreateFileW that is being imported. Notice in this case that the library is not kernel32, but instead, it is api-ms-win-core-file-l1–1–0 which is a different library. There’s a method to refer to functions with the same name, but that are implemented by different DLLs which follows this convention DLLName!FunctionName. For example, the version of CreateFileW that is implemented by kernel32.dll is referred to as kernel32!CreateFileW while the version that is implemented by api-ms-win-core-file-l1–1–0 is api-ms-win-core-file-l1–1–0!CreateFileW. Okay, so to continue the journey toward understanding the function call hierarchy, we must understand what api-ms-win-core-file-l1–1–0 represents.

Application Programming Interface (API) Sets

In the previous section we found that kernel32!CreateFileW was importing api-ms-win-core-file-l1–1–0!CreateFileW. The api-ms-win-core-file-l1–1–0 component refers to a technology that is present in newer versions of windows called API Sets. API Sets are meant to be transparent to users and developers, but it is important for us to understand how to navigate them to continue digging into the CreateFileW implementation. Geoff Chappell does an excellent job documenting this technology on his blog. Notice that there are many file-related APIs that are all redirected to the same library name. This library serves as a redirector to a third implementation of CreateFileW, but it isn’t obvious where this version of the implementation resides. In order to figure out where CreateFileW’s implementation is located we must “resolve” the API Set to determine where it directs execution. This can be done using James Forshaw’s wonderful PowerShell Module NtObjectManager which has a super helpful cmdlet called Get-NtApiSet. This cmdlet resolves API Sets to their redirection DLL.

Undocumented Function

By using Get-NtApiSet we find that api-ms-win-core-file-l1–1–0 resolves to kernelbase.dll. This means that the next function in the chain is kernelbase!CreateFileW is an undocumented function. An undocumented function is a function that, while externally available to third-party applications, is not documented anywhere publicly. This means that it can sometimes be difficult to use because it might take slightly different parameters than the documented version, and Microsoft generally reserves the right to change how undocumented functions actually work. This means that legitimate developers should generally avoid using undocumented functions, but malware developers might use these functions to present an unexpected look to defenders. We can now load kernelbase.dll into our disassembler, load symbols, and navigate the export table to find the CreateFileW implementation.

We can double-click on CreateFileInternal, the internal function called by kernelbase!CreateFileW, to view its content which reveals a call to an imported function called NtCreateFile.

Let’s check the import table to see what DLL implements this function. In the screenshot below, we see that NtCreateFile resides within ntdll.dll which means we can refer to this specific function as ntdll!NtCreateFile. Functions implemented by Ntdll belong to a special class of functions called Native functions. Generally speaking, Microsoft discourages the direct use of Native functions for similar reasons that it discourages the use of undocumented functions. The Win32 API often helps the developer with complicated tasks that might not be as easy to implement at the Native API layer.

We can load ntdll.dll into our disassembler, load symbols, and navigate to the implementation of NtCreateFile. Native APIs generally have a simple but very special important role in the hierarchy of execution. Their responsibility is to make the appropriate system call (syscall). A syscall is a special type of function call which is responsible for transferring execution from user mode to kernel mode via the SYSCALL instruction on 64-bit systems. Notice that the basic block on the bottom right performs a “syscall” operation. Also, the highlighted value,55h, represents the value of NtCreateFile’s counterpart in the kernel which we can call ZwCreateFile. A complicating factor is that the number associated with a particular syscall may change from version to version of the operating system, so in order to operate at this level, developers must be extremely conscious of the OS version that they are operating on. Making direct syscalls has become a relatively common visibility bypass recently especially because it’s a great way to bypass any user mode signal generation. To see how this is being leveraged check out the windows-syscalls project by j00ru.

While we could continue our analysis into the kernel I think we’ve explored enough to understand the layered nature of function calls. The important point is to realize that any “exported” function is an entry point for an application. While the vast majority of applications will follow the “prescribed” path of calling the relevant Win32 API function (kernel32!CreateFileW in this case), however, it is sometimes advantageous for attackers to perform actions in unorthodox ways. If defenders ASSUME that all file creation will be performed via the approved or prescribed path and as a result only pay attention to the use of Win32 APIs, then attackers can take advantage of the naivety of that approach to avoid visibility.

I’ve generated a graph representation of the function calls that we observed during this blog post so we can see the relationship between them. I’ve also highlighted exported functions in red to indicate that those functions are valid entry points into the graph (even if most developers would only choose to use the documented Win32 API function).

The major takeaway is that in the context of cybersecurity and detection we shouldn’t assume that attackers will choose the path most traveled or the prescribed path to perform a particular behavior. Therefore it is worthwhile for us to understand the options and evaluate opportunities to observe behavior at each functional level. Also, it is worth identifying that this graph represents only one path of functions that can be used to create a file. In my experience, there are often MANY possible paths that can be used. A future blog post will demonstrate how we can evaluate numerous paths to certain behavior in order to produce a complex graph and how to use that complex graph to evaluate detection rule coverage and predict ways in which adversary procedures can evolve in the future. The same principles apply. If an attacker can accomplish the behavior via an unexpected path then they will have the upper hand relative to stealth. Simultaneously, by mapping all possible paths or at least all known paths defenders can predict hypothetical implementations, tools, or paths even if they haven’t explicitly observed a tool that leverages that approach.

Understanding the Function Call Stack was originally published in Posts By SpecterOps Team Members on Medium, where people are continuing the conversation by highlighting and responding to this story.