Introducing Ghostwriter: Part 2

Part 1 introduced Ghostwriter. This article expands upon the making of Ghostwriter, the technology stack, and the thought process that went into the development. If you have not read Part 1, check out Ghostwriter here:

Stack Overview

Ghostwriter is a web application written in Python with the Django web framework. It is a collection of Python 3.7, HTML, JavaScript, CSS, Jinja, and Django code compartmentalized into multiple Django applications. This compartmentalization helps keep the code organized and easy to peruse during customization or development efforts.

The application uses a PostgreSQL backend that Django natively supports. However, should users desire to switch to a different type of backend, the Django settings.py can be updated to use SQLite, Oracle, or MySQL without any additional libraries. Django makes it easy to modify the Ghostwriter database models as well. Migrations are usually smooth and trouble-free, especially if you are customizing the models prior to using Ghostwriter in production.

Finally, Ghostwriter uses Django Q and Redis for automated queue management processing (AQMP). Ghostwriter automates a number of things like updating domain categorization data and DNS records. These tasks are handed off to Redis for background processing.

Setup is relatively simple. Ghostwriter can be run locally (for an individual consultant) or on a low-power server. The host only needs Redis and Python 3.7. Ghostwriter uses a relatively small number of non-stock Python and Django libraries. Ghostwriter is packaged with Docker, so everything can be deployed using Docker Compose and one command. The full details are available in the GitHub wiki.

Some details concerning why these specific technologies were selected is covered in this post:

Database Models

Even though Ghostwriter is comprised of multiple applications, they all talk to each other and share a single database.

As seen below, the database models are all linked to each other to tie everything together. The documentation in the wiki provides more detail and individual UML diagrams for each application.

A table exists for each type of object and we have included Django “fixtures” (predefined datasets) for many of these models. These fixtures are loaded into the Ghostwriter database to pre-populate the necessary models to get a new deployment up and running in minutes. Ghostwriter supports CSV file imports for other models, like domain names and servers.

The Making Of Ghostwriter

While evaluating past experiences, current workflows, and currently available solutions, we quickly identified things that caused us irritation or negatively impacted efficiency and productivity. The key here was investing time into this evaluation. We took a week in the SpecterOps offices to look inward and discuss what we liked, did not like, and wanted in a new tool.

When a team implements a new process there will (almost) always be some suboptimal workarounds, hacks, or peculiar decisions paired with good intentions to fix them later. Those good intentions rarely transform into corrective actions. If these issues aren’t smoothed out they have a tendency to just become the accepted norm and fester.

Think of this as an effect of the “Principle of Least Resistance.” Cal Newport describes this principle in his book Deep Work:

In a business setting, without clear feedback on the impact of various behaviors to the bottom line, we will tend toward behaviors that are easiest in the moment.

As previously stated, project management and process workflows are not glamorous, so it is understandable that when a team has time to optimize something they aren’t going to gravitate towards reporting. The dark side of this is the entire team continues to be collectively irritated by a dozen little things, all self-inflicted. Eventually these grievances are accepted as necessary speed bumps, bogging everyone down until someone takes corrective action or the eventual heat-death of the universe: whichever comes first.

So, the SpecterOps team set out with an ambitious plan: build our dream tool. That was a lofty goal, but a necessary one. By shooting for that distant star we hoped to at least hit the moon 😉. We called this project the SpecterOps Automated Reporting platform, or SOAR, and dedicated ourselves to seeing it through.

With the increased popularity of “Security Orchestration and Response” as a term, it’s a good thing we weren’t too attached to that name; however, SOAR would have been a great name for this platform. C’est la vie.

We named it Ghostwriter instead.

We laid out several important tenets for the project:

Flexibility

One of the first failures of custom management tools is a lack of flexibility. The tool is created to address the most immediate concerns with little regard for what will happen if the process ever changes in the future or needs to be enhanced. We knew Ghostwriter would have to be capable of being easily changed to fit alterations in report designs and requirements.

This was also important for releasing Ghostwriter as an open source tool. It is easier for others to adopt Ghostwriter as a member of their team when they do not need to invest hours into rewriting code to make a simple report design change.

In support of this tenet we included JSON reporting early on to make report alterations simple, allow bringing Ghostwriter projects into other reporting engines (if desired), and normalize report generation across all of the different types of reports (e.g. docx, xlsx, pptx). We use Office documents as templates for the report styles. This way no code needs to be edited to make adjustments to most of the fonts, colors, or header styles. The majority of all of that is managed using a familiar and ubiquitous tool, Microsoft Office.

Extensibility

Extensibility is equally important as flexibility. We knew we would want a solution that could be extended down the road. One of Ghostwriter’s primary objectives is keeping everything in one place. If Ghostwriter could not be easily extended then that might mean a second application or platform could come into play for tracking some new future requirement. That may sound like a little thing; however, it could lead to increased unhappiness over time.

We did not want Ghostwriter to end up so rigid in construction that it could only ever serve a limited set of functions. In our experience, even great custom applications go stale and, at times, become a burden to use. A burden that saves you time is still a joyless thing.

The ability to add new applications that could integrate into existing Ghostwriter applications is a blessing. We are already working towards adding new applications to Ghostwriter, such as an application for the blue team that leverages the reporting engine to generate Alerting and Detection Strategies (ADS).

Not only is an application that generates ADS reports useful for our team, the integration with the existing applications means the “Blue Team Application” is also aware of all of the “Red Team Applications” and accompanying client information, projects, reports, and activities recorded by other team members — and vice versa. Extending Ghostwriter in one direction not only adds new capabilities but also enriches the whole platform.

After all, Ghostwriter is a friendly purple ghost and just wants to help everyone.

Resilience

One of the early fears we had was building Ghostwriter using $Some_Library and then having $Some_Library change in a way that would be a negative for Ghostwriter. So, we committed to the concept of “resiliency” for the first version of everything. Everything in this current version of Ghostwriter is built using core Python and Django libraries. There are a few exceptions, but outside of Django Q (which is well supported) most of the third-party libraries support features that could change without breaking Ghostwriter. Even those are well-used and supported libraries, like Requests. Otherwise, the only other outside dependencies are web frameworks, jQuery and Bootstrap, which are both popular and well supported.

That all means Ghostwriter is easier to install and use and won’t break or act up if an outside library is changed or the wrong version is installed. This made development slightly more difficult in some cases. We chose to leverage jQuery instead of third-party Django libraries, for example. This added a learning curve and some initial difficulties, but the end product is ultimately better off as a result.

Perhaps the most noticeable trade-off is the lack of certain bells and whistles, like WYSIWYG text editors. WYSIWYG editors would certainly be nice, but we deliberately chose to avoid them to make sure there would be a strong foundation in case whichever editor plugin we chose were to vanish one day. At least then we could excise the non-functional WYSIWYG plugin and know the basic text boxes would work. (Also, WYSIWYG editors are coming.)

Automation

We wanted Ghostwriter to be able to automate certain processes that would be tedious or those we knew that humans would forget about or ignore. This tenet speaks for itself, but it was important for us to make this a priority early on in the development process. Every time we implemented a new action a user could take we had to ask, “Is any part of this process something we should just automate based on the user’s input?”

Once we had functioning automation using triggers and scheduled tasks (easily done thanks to Django Q and Redis), we could also have fun spitballing new ideas for automation, like Slack notifications and DNS checks for domains.

Eat the Dog Food

Finally, we knew the best — and only — way to make sure we were on the right track was to use Ghostwriter as early as possible. The goal was always to use Ghostwriter, but we started using it within two months instead of waiting for a “beta” or some sort of “v1.0” release. Back then it was just a Python library and a script run from a command line that would generate a report. It was a proof of concept for some of the earliest ideas. That grew into a basic web front end created using Python’s Flask web framework. Research and testing eventually led to the switch to Django.

During each phase we tried different interface designs, tested new ideas and concepts, and just generally experimented. There were quite a few duds, but this process was so important. To quote Mark Frauenfelder, founding editor in chief of Make magazine, from Adam Savage’s book Every Tool is a Hammer:

You’ve got to do at least six iterations, minimum, of any project before it starts getting good enough to share it with other people.

We passed Ghostwriter’s sixth iteration a while back. This gif shows what Ghostwriter, or SOAR as it was known at the time, looked like on September 19, 2018. We hope you will agree with us, the waiting and numerous iterations was worth it.