Being a Good Domain Shepherd: Part 2

Part 1 of this post discussed the data collection side of a solution for managing domain names used for red team operations and penetration testing and introduced the DomainCheck tool. As mentioned in that post, the SpecterOps team continued research and development on a solution that would combine DomainCheck’s data collection with team management capabilities. The first product to come out of that work is Shepherd.

Shepherd is designed to be used by a team of operators. It keeps track of domain names and each domain’s current DNS settings, categorization, project history, and status. The tracked statuses include which domains are: ready to be used, burned/retired, or in use, and which team member checked out each of the active domains.

Shepherd is the latest addition to the GhostManager collection. Check it out here:

Tending to the Flock

The more we examined what would be needed from a domain management solution, the more we realized our initial design, alluded to in Part 1, was inadequate. A solution that would streamline domain check-outs, automate check-ins, and track categorization would be a good start, but it would only work well for a team no larger than a few people. If someone has a question about a domain, a small team can be easily contacted to confirm information. That just does not work for a larger team spread across multiple time zones.

Who Bought This?

It’s a common question in SpecterOps chats. The team has a lot of domains purchased for different purposes but those purposes are not always clear. If a domain’s history is tracked, anyone could see when it was purchased, why it was purchased, if it was purchased for a specific project, and how it has been used since.

Ya Burnt

Project tracking led to the idea that a domain’s health status is not global. If a phishing link is reported to a security team, the domain is likely to be blacklisted within a short period of time, but that domain isn’t a goner yet. If the domain itself is not reported to a third-party, that domain has a second chance at life. Tracking the domain’s history means anyone can see they should not use the domain for that particular client anymore but not consider it universally “burned.”

A Lost Lamb

Like the above situation, if a domain’s categorization is changed to “Suspicious” or “Phishing,” that domain can be retired without a second thought, and Shepherd does this automatically. Some situations, though, are murkier and require a human to make the call. We decided to give administrative users (i.e. senior operators) the ability to quickly retire any domain and remove it from the pool.

An operator may also want to remove a domain from the pool for other reasons, such as holding it for a future project or aging the domain. New domains default to “Available” in the pool because they are not in use or tied to a project. We determined a “Reserved” status would be helpful for domains that were purchased for the future.

Command Your Domains

With a design and wishlist sketched out we moved forward with the creation of Shepherd. To toss in a small teaser here, we designed this new tool in such a way that we could add onto or extend it in the future for some other GhostManager projects that are in the works. The final interface is clean and easy to use.

We designed Shepherd to be simple. Once a user has authenticated, they are presented with a summary of the domain catalog and can view all of the team’s domains or filter the domains based on their status. The domain catalog tracks all domain names and their various statuses and project histories. There is also a notes field that can be used to record why a domain was purchased, provide details on how it was used in the past, or why it was temporarily reserved or permanently retired.

This is also where team members can check-out a domain for use on a project. When a domain is checked-out the user will see a second screen asking for some details for tracking that domain’s use. Shepherd tracks who is checking out the domain, the client name, project type (e.g. penetration test, red team), how the domain will be used (e.g. phishing, C2), and the date range the domain will be in use. Clicking a domain name in the catalog will present the user with all of the details recorded for that domain and that domain’s project history.

For now, Shepherd checks to see if the client name is already in the database. If the name is not found in the database then a new entry is created. This is not ideal because of typos and the potential for different names to be used for the same organization, but lays the foundation for a future update that will include a client manager and the ability to edit clients and view projects and domains tied to the client entries.

Users with more elevated access (e.g. senior operators) can flag domains as retired/burned or unavailable for use (i.e. reserved status). These users can temporarily pull domains out of the pool to let them age or mark them as burned so no one can check them out for use on future assessments.

Finally, users can view the list of unavailable domains to see which have been burned and removed and which are on the sidelines. No domains are ever fully removed from the database so the historical records are maintained.

A search bar is also always available that enables searching by domain name and by category.

Data Imports

Records can be imported from a csv file. Shepherd provides a template that can be downloaded from this page and edited. Only the basic information (e.g. domain name, registrar, purchase date, and expiration date) is required, but health and category information can be included as well if it is readily available.

Also, the latest DomainCheck outputs a csv that is ready to import into Shepherd. Make sure you have a version of DomainCheck updated after the publishing date of this post, 15 January 2019.

Automated Updates

Domain records can be manually updated with categorization data. As a better alternative, DomainCheck’s functionality has been baked into Shepherd. Shepherd has a task that updates the catalog’s DNS and categorization data. DNS updates can take a minute or two, but the full health check-up can take a while due to API limits (VirusTotal’s free API only allows four requests every 60 seconds, so check-ups take X minutes where X = TOTAL DOMAINS / 4). The default sleep time is 20 seconds.

If users have a paid VirusTotal API key the sleep time can be changed in Shepherd’s settings. This will help speed up the checks, but reducing it too much is not advisable because Shepherd will start encountering some reCAPTCHAs on a few sites.

The updates will run as a background task and update the domains as each one is reviewed.

This task can also be scheduled to be run at specific times. Then the updates pages become a way to request manual updates and view the status of the last update, whether it was manually requested or scheduled.

Additionally, the SpecterOps team is looking into conditional task execution. For example, a recurring category check could be scheduled for an individual domain whenever it is checked-out. It is simple to fire these tasks, but some of them, like category updates, can take a while and it is desirable to only fire them occasionally.

Slack Integration

Like DomainCheck, Shepherd has some Slack integration options. If Slack messages are enabled and the Slack configuration is filled-out in settings.py Shepherd will send messages to the specified channel. Messages are sent to alert users to various events, such as a domain health check completing and domains that have flipped from “Healthy” to “Burned.”

We are also considering some other options. Shepherd may eventually support sending emails and/or push notifications via services like Pushover.

The Wooly Technical Details

We started sketching out ideas using Python and the Flask web framework, but certain demands pushed us to use Django. Django offers a few advantages over Flask for a project like this, like a built-in admin console with “out of the box” support for authentication and account management. Django also supports several types of databases, with SQLite being its default. It is unlikely Shepherd users will require simultaneous database edits, so Shepherd uses the default SQLite database.

The Catalog App

To use Django’s terminology, Shepherd contains an “application” called catalog. This is the primary application that tracks the domain names and everything associated with them. The Django “models” used for catalog look like this when visualized.

It’s a simple design that allows for editing the database and migrating with little to no fuss. Django’s makemigrations and migrate commands make database tweaks a breeze. Those commands should also help any users who wish to modify the models to add new fields or change how data is presented in Django’s admin console.

Web Design

Shepherd uses jQuery, jQuery UI, Bootstrap, JavaScript, and Font Awesome for some of the web design, mostly the menu and datepicker widgets. These are managed inside the base_template.html file where they are loaded. Django uses templates to render web pages and makes it simple to generate content, like tables of domain names, on the fly.

The rest of the templates are built using HTML and Jinja2. Using Jinja2, a base template is created and maintained in one file. For Shepherd, this is where the sidebar menu and navigation bar lives and where scripts and stylesheets are loaded.

Many different forms had to be created for managing the domain catalog. Rather than creating individual HTML forms in the HTML files, forms are passed to templates in the back-end Python code and placed and rendered when the webpage is loaded using Jinja2 content blocks.

The other templates extend the base template and use Shepherd’s Django models and forms, so some are just a few lines of HTML and Jinja2 code but display whole forms for managing domains. Any individual page can be customized as desired while changes that need to be applied to all instances of a form, model, or style choice can be managed in one location.

It should be straightforward to customize Shepherd as desired.

App Dependencies

Shepherd runs on Django. A few additional libraries are needed for domain status updates:

• bs4
• cymon
• lxml
• pillow
• pytesseract
• requests
• django-q
• redis

The major dependency is Django-Q and its Automated Message Queuing Protocol (AQMP) “broker” Redis. AQMP is needed because Shepherd has to run a few different background tasks, including sending Slack messages, automatically releasing domain names on a project’s end date, and running domain status updates.

Releasing a domain requires a quick loop through domains that are marked as in-use and releasing any with an end date that is equal to the current date or earlier. That takes moments to complete and could be run on-demand as a web request, but we wanted this to happen automatically after midnight every day.

On the other hand, we want domain updates to be on-demand rather than scheduled, but they can require 20 minutes or more to complete. That just does not work for standard web server requests and so the task must be run in the background.

When a domain status update is requested that task is handed off to the broker for processing. The task is queued and executed in the background so the end user does not have to wait 20 minutes for a response from the server. The same thing happens with the task for releasing a domain back into the pool, except it happens at a scheduled time without any need for a request to be made.

Task Scheduling Options

We briefly considered using some of the more basic alternatives, e.g.creating custom admin tasks and using cron jobs to call the functions via Django’s manage.py at scheduled times. That option could have worked, but it is less elegant, would have limited options, and made monitoring difficult. Had we gone with that option we would have needed to switch to a more advanced solution down the road, so it was better to just do it now. As it turned out, we quickly found a number of additional uses for Django Q like the Slack message queuing and dynamic scheduled tasks. Avoiding cron jobs was a good call that paid off during later feature development.

We also looked at Celery, Huey, and django-background-tasks. Celery with Redis or RabbitMQ is still the standard package many projects use, but we could not see any future need for that level of queue management. The most complex job Shepherd needs to perform is the domain health check and that is performed infrequently and must be performed one domain at a time due to API limits and to avoid reCAPTCHAs. If we could spin up a task for each domain then Celery would have been an excellent option for managing something like 100+ tasks.

The django-background-tasks solution looked great, but was too simple for our desired product. It is basically a nice way of interacting with crontab programmatically and (per the documentation’s suggestion) still uses a cron job to ensure the django-background-task queue manager remains running in case of a failed task. That’s not a bad thing, but shares some of the same downsides as the full cron job solution.

We ultimately selected Django-Q because it is native to Django, integrates into the admin panel for easy monitoring and troubleshooting, and is simple to implement. It still requires a broker, but setting up the broker is simple. Django-Q uses Redis by default, there is a Python library for Redis, and it is a very simple broker to use. All you need to do to get started is install Redis (e.g. brew install redis) and then start the server. Should it be desired or needed, Django-Q also supports more advanced options like RabbitMQ and Amazon SQS. Overall it seemed like the best option.

Conclusion

Shepherd will hopefully improve communication between team members by automating much of the obnoxious and error-prone management that goes into recording the who/what/when behind a domain’s use. This allows team members to focus on the more interesting/technical aspects of a project rather than pinging Slack channels to find out if a domain is available.

What is even more exciting is Shepherd’s future potential. As teased above, we are already working on ways to expand this platform and we are excited to share the results in the near future. Shepherd is only the beginning; we expect this platform to really soar to new heights, so be on the lookout for more news soon!

Do note that, while we have included user authentication and limited privileges of basic user roles, Shepherd should not be put on the internet. Remember, Shepherd contains names of clients, your domain names, and project details. That is not the sort of thing you want accessible to the internet. We strongly encourage anyone using Shepherd to use it internally and require a VPN for external access.