TL;DR: Ghostwriter now supports real-time collaborative editing for observations, findings, and report fields using the YJS framework, Tiptap editor, and Hocuspocus server, enabling multiple users to edit simultaneously without overwriting each other. This feature integrates seamlessly with Ghostwriter’s existing GraphQL API and infrastructure while introducing modern frontend tooling, React-based components, and upgrades to Django, PostgreSQL, and development workflows.

If you’re into cybersecurity, you’ve probably edited a Google Doc. In fact, you’ve probably edited a Google Doc at the same time as other people and enjoyed how you could see their edits appear on your screen in real-time! That is collaborative editing, and it’s a handy feature—not just because you can see what others are doing but also because you don’t have to worry about the changes you make overwriting anyone else when you save.

We’ve had many requests to add collaborative editing to Ghostwriter, and indeed, we at SpecterOps have also wanted it for a while. But there is a lot of infrastructure and code to make the magic work. Peers must constantly communicate with each other, not just to send their copies of the documents but also to track and send the history of each change so that other peers can successfully merge conflicting edits without issue—a big ask.

However, after a few months of experimentation and implementation, we’ve added collaborative editing to a few sections of Ghostwriter! Most notably, editing observations, findings, and extra fields on reports all sport this new feature.

Collaborative editing is in the spotlight, but it comes with many new changes—including inline image evidence previews, a new editor we’ve customized for Ghostwriter, and some back-end upgrades. Check it out below:

We’re releasing all of that as a release candidate for testing, with plans for a full release in a month following feedback. We’ve been testing this release for the past month and are excited to share it.

This post covers how we implemented collaborative editing in Ghostwriter and some critical information (see Upgrades to Django and Postgres) about the changes. If you only read one more thing here, make sure it’s the information in that section. Read on to learn more!

Selection of Technology

There are plenty of collaborative editing solutions—in fact, TinyMCE, our current rich text editor, offers one! However, many of the easy-to-use ones come with a price tag attached. This is partially because collaborative editing is a “premium” feature above and beyond simple text editing and also because of the infrastructure required to make collaborative editing work—often provided in The Cloud™.

Because Ghostwriter is an open-source project that manages sensitive data that shouldn’t be sent to untrusted parties, we needed a solution to maintain the infrastructure that was compatible with open-source, which ruled out many cloud-provided solutions. We wanted a solution that allowed us to extend it with custom elements for features such as embedding evidence. We also needed a solution that still allowed us to access the data on the server—not just in the web browser—so that Ghostwriter could still generate reports. And finally, we wanted something that is actively maintained and has an active userbase to ensure continued support.

After searching, we concluded that the YJS ecosystem was the most mature for collaborative editing. YJS itself is a data format designed for the task—it tracks a log of operations so each peer can make its edits and publish updates to the document to other peers. The updates can be applied in any order, resulting in everyone having the same data after all of them are applied.

Many text editors supporting YJS exist. We selected the Tiptap rich text editor due to its features, maturity, and extensibility with custom elements and formatting.

We selected Tiptap’s Hocuspocus server for sharing collaborative edits between peers. This server exposes a WebSocket endpoint that the client publishes and receives updates on in real-time. It also handles loading and saving the YJS documents from a data store.

Internals

YJS recommends that we store the document in the database since the document contains a log of all the operations, and having that history helps with reconciling edits. While we explored changing Ghostwriter’s data model to YJS and storing documents in its database, we determined it was not a practical solution. It required extensive rewriting of our data models, including bridging the YJS document with Django’s ORM. It would also have required completely replacing the Hasura GraphQL API with something that understood YJS. Additionally, we would have had to rework the report generator to take in YJS XML instead of HTML, and the Python bindings to YJS were not well suited to this task.

Instead, we decided to leverage the Hocuspocus server’s API for loading and saving documents to a database to load and save our existing models via our existing GraphQL API, converting them on the fly from and to YJS. This allows the Hocuspocus server to easily slot in with our existing infrastructure with minimal changes, as we don’t need to change our underlying database structure.

Our extension to Hocuspocus uses the Apollo GraphQL client to communicate with our Hasura GraphQL API. While we had to add a few more endpoints to the API around permissions checking and accessing tags, it uses the same API available to existing integrations. And since they both run in the same Docker Compose environment, they communicate via the internal Docker network for efficiency and security.

To convert rich text fields between HTML and YJS, we leverage Tiptap’s ability to import and export its own internal element tree to and from both HTML and YJS. As a side effect, this also lets us work with rich text fields edited with the old TinyMCE editor, which produces HTML (though this isn’t extensively tested!).

The downside of this approach is that the server does not preserve the history between document loads. In particular, if a client sends an update for an older version of a document than the server has loaded, the divergent histories will cause issues. This shouldn’t usually happen since the server keeps the document around while clients are connected, and the clients don’t persist with the document, but it may occur if the server has to restart and the client reconnects. To prevent this from causing issues, the server embeds a random UUID into the document every time it loads. The client, if reconnecting, provides this ID, and if it doesn’t match what the server is expecting, the server will reject the connection, preventing corruption.

The second is that edits made outside of the collaborative editor (for example, through the GraphQL API) will not propagate to the collaborative editor while it is running, and will be lost when the Hocuspocus server saves its state. This limitation required us to remove a few features from Ghostwriter, such as changing finding severities from the report detail page, to prevent conflicts. We will continue iterating on this. Our goal is feature parity between v6 and previous versions, but we didn’t want to hold back collaborative editing for this.

Related Work

While collaborative editing is the top billing change, a lot of infrastructure work also went into this, which should make Ghostwriter more maintainable in the future.

Upgrades to Django and Postgres

We have upgraded Django to version 4.2 and PostgreSQL to version 16.4.

This PostgreSQL upgrade requires additional steps to upgrade an existing Ghostwriter instance to v6. To automate the process, we’ve added the pg-upgrade subcommand to ghostwriter-cli. Make a database backup using ghostwriter-cli, pull the new update, and then run the subcommand to migrate the data (optionally passing --dev for a dev instance).

These upgrades will keep Ghostwriter on supported versions of our major software components, allowing it to continue receiving their fixes and improvements.

JS Infrastructure with Vite

Between needing custom editor components and handling connection state, working with plain JS like Ghostwriter has been was not going to cut it for collaborative editing. The update adds a javascript/ directory containing the Vite project for the collaborative editing JS code—both the frontend and the Hocuspocus server. This lets us use Typescript, Sass, graphql-codegen, React, and NPM to make writing and maintaining frontend code easier.

For production, Ghostwriter builds the frontend code as part of making the Django container. On development, you’ll see a new frontend container that watches the frontend and rebuilds it as needed, making development more convenient.

Only the JS related to collaborative editing is currently built with this system. Later, we may move some of the loose JS files currently with the Django static files into it as well.

React

Related to the above, we use React to render the collaborative editors—a step up from manual management of the DOM that we do in a few other places. We also implement a few collaborative-aware controls for non-rich-text fields—like checkboxes, titles, and selectors—in React as well. With the JS infrastructure now in place, we may leverage React for more complex UIs in the future.

Nginx on development

Production instances of Ghostwriter use Nginx as a reverse proxy in front of the Django app, to provide things like TLS. So far, development instances have excluded this container and exposed the Django app directly. However, collaborative editing requires us to route some traffic to Hocuspocus instead of Django—something that Nginx can easily do. Thus, we now set up the Nginx container on development as well.

In addition to assisting with routing, using Nginx in development also helps the development environment match the production one more closely.

Wrap Up

As mentioned above, this is a release candidate we are releasing for users to try. We’d love to hear your thoughts and feedback if you try it. Please remember to perform the necessary upgrade outlined in the Upgrades to Django and Postgres section.

You can grab the release candidate here:

https://github.com/GhostManager/Ghostwriter/releases/tag/v6.0.0-rc1