Go Phishing (and Reporting)

I have used most of the phishing frameworks out there and none of them ever really grabbed me. That was true until Jordan Wright (@jw_sec) released Gophish. Jordan did the release so right. The platform is polished, there is documentation, and the whole package is just really slick. I liked Gophish so much I created a reporting tool for it called Goreport. That tool has somehow stood the test of time and continued working with Gophish through each new release; however, life happened and I couldn’t stay on top of the latest releases. Goreport fell behind on some Gophish features like the “Email Reported” events. It was also tough to do any development without a Gophish database, and I never seemed to have one available. That was until Gophish gained the awesome feature of generating sample databases.

I am very happy to release an updated version of Goreport, v3.0, that incorporates all of the new Gophish features, fixes some old issues, and includes a number of enhancements.

Testing the Waters

Gophish can be run inside a Docker container and with one short command Docker will run Gophish, forward local port 3333 to the container, and generate a demo database.

1. There are two things you will need to start:
2. Docker installed for your operating system
3. A git clone of this Github repo: https://github.com/MatteoGgl/docker-gophish

Once you have Docker installed and the container cloned it’s time to build the container and start Gophish. This is done in two commands:

1. docker build -t “<YOUR_DOCKER_ID>/gophish:<SOME_KEYWORD>”.
2. docker run -ti -p 3333:3333 — rm gophish/demo

Once the container is running you will begin to see the simulated events stream by in your terminal:

Gophish generates a decent sample size of a few hundred recipients and their activities so this will take a short time to complete. Once you are instructed to visit https://localhost:3333 you will know the demo database is ready. Login with admin:gophish to see your new campaign summary. Mine looked like this with 298 fake recipients:

Measuring the Catch

Gophish’s interface provides all kinds of data about your campaign but there’s no reporting function. The only report export option is a dump of event data you can download as csv files. This is actually a good thing. Other frameworks output PDF reports that you cannot change and it’s difficult to extract the data from the report and move it into your own report template. Reporting is where the Gophish API shines.

The API allows for users to create and execute entire campaigns using just the API. Users can do many interesting things with the API. Goreport leverages the API to access campaign and event details, enrich that data, and then generate reports. Goreport will generate spreadsheets, document files, and statistics. Thanks to the demo database, all of the API calls can be used for testing and experimentation, which makes development much easier.

Scaling the Phish

Goreport accomplishes all reporting tasks using Python 3.7, web requests, and the Gophish API. Goreport ingests the campaign data from Gophish and then processes each event to provide event totals (both overall totals and unique totals). The demo database does not simulate this but it is common for phishing campaigns to record more than one click for one recipient. The victim may visit the phishing page multiple times or forward the email to others who may then click the link. All the clicks get lumped together because they’re associated with just one recipient’s rid value (Gophish’s unique ID value assigned to each recipient).

For that reason Goreport compares IP addresses and displays a message if the recipient’s IP address changes between events. Such a change might indicate the recipient connected or disconnected a VPN or changed locations but by flagging these changes Goreport determines if timestamps and activities suggest the email was forwarded or is being analyzed.

For example, if you see the recipient clicked the link from New York City and then clicked again from Boston within minutes there might be something interesting happening. If the IP changed to Mountain View, CA and the IPs belong to Google you might be catching someone attempting some triage with VirusTotal. These are events that are easy to miss if you’re looking at Gophish’s raw data so Goreport helps you by flagging these mismatches.

Filleting the Phish

On the subject of IP addresses, Goreport geolocates each unique IP address. The previous release of Goreport did this using a messy mix of the MaxMind database and the Google Maps API to take the coordinates Gophish stores with the events, lookup the coordinates with Google Maps, and then compare the Google Maps results with MaxMind to try to get the most up-to-date and accurate IP location. This worked but it was error prone and horrible to debug. Also, Google changed their API and the Maps API is no longer free.

This release removes all of that and just uses Gophish’s geolocation information. Optionally, you can now provide an API token for ipinfo.io in the gophish.config file. The free tier of ipinfo.io offers 1,000 queries per 24 hour period which is likely to be more than enough for a GoPhish report. If you happen to be running a very large campaign you could split-up the emails into smaller Gohish campaigns of ~500 recipients and run reports against these individual campaigns (or pay for more requests for one month).

The Maps API is still included as well. Google reorganized the API and broke it into SKUs for billing. It still works the same way as it did when I first used it in Goreport but now it requires a Google account, the Geolocate API enabled on that account, and an API key that will incur charges for each request. It’s inexpensive at $0.005 per request or $5 for 100,000 requests but Goreport has this API disabled by default. You can enable it by providing a Geolocate-enabled API key in the gophish.config and using the new --google flag. The Geolocate API will only be used if an ipinfo.io API key is unavailable.

A final Maps tweak is the data collected from the API. The API returns results with the first result being the most specific location. Each result that follows “zooms out” from the location and gets less and less specific. It might begin with Seattle, WA, then just Washington, and then the United States. Goreport always took just the first result and then tried to piece together a sensible address from the address_components within the result. The API now includes a formatted_address key which provides a full address in a human-readable form. Goreport now returns that as the result.

There is one caveat: Google Maps is very specific and will return road names and even the name of the building at the site if those details are available. This could be good or bad depending on what someone wants from the location data. Rather than cut it down to just returning the city Goreport will leave it up to you to determine if you want to lump different locations within the same area together to count as one location.

All of that said, I would encourage trying the ipinfo.io route. That is enabled by default (as long as an API key is provided) and may be more accurate than Gophish’s coordinates. Sometimes the Gophish coordinates match-up with MaxMind and/or ipinfo.io and other times Gophish will place the recipient in another country. This is not an issue with Gophish; rather it is the result of IP addresses changing hands and moving around the world.

To help reduce potential costs (money or query credits) Goreport is now much smarter about geolocation lookups. Campaigns will contain multiple events tied to the same IP address and the old Goreport would lookup every IP address each time it appeared in an event. That meant each IP address for a “Submitted Data” event would be looked up a minimum of two times (once for the Clicked Link event and once for the Submitted Data event) just for that one recipient’s click and submit. This was silly even when lookups were free. Goreport now checks a list of IP addresses it has already queried before checking ipinfo.ip or Maps.

This also brought about the new IP and Locations table. It just made sense to convert this query-saving list into a new statistic so you can see which IP addresses match-up with each location. You could see this before in the detailed event tables but now it’s all in one table. This table makes it much easier to see IP addresses that share the same location.

Goreport also correlates user agent information from the event details with operating system versions and browser versions. The Gophish dashboard displays this information but it’s not in the details available from the API. Goreport uses a handy Python library to match-up user agents with the software.

Grilling the Phish

Goreport saves all of the location, operating system, browser, and IP address data and creates summaries at the end. Each report includes a list of all locations, operating system versions, browser versions, and IP addresses and a count of how many times each one was seen during the campaign. This provides a nice snapshot of the type of the environment your phish landed in.

These summaries and all of the data from each event are recorded in the final report in nicely formatted tables. Goreport uses the python-docx library to generate the tables and final document. You can customize the colors, font, and other formatting options used for the report by editing the template.docx file. The provided template has a table style simply named “Goreport” that you can edit as desired and then save as the new Goreport style. You can also change the header and “Normal” text styles to change how the rest of the report looks. The next Word report you generate will use your new styles.

The reports contain a summary of events (above) and the details for each event tied to each individual recipient.

Finally, Goreport has moved from the csv library to Xlsxwriter so the spreadsheet reports can be organized into worksheets with colored table headers. These new reports are much easier to browse and read and are now a nice alternative to the docx reports. To match this change the old --format csv option has been changed to --format excel.

Also, Xlsxwriter and Excel support data types that Goreport uses to assist with filtering. The old “Y/N” values that determined if a recipient had clicked or submitted data have been replaced by Xlsxwriter’s write_boolean() function and True/False values. This means you can now more easily sort/filter those values.

Dinner Time

The demo database and Docker container are great tools for developing automation scripts with Gophish. There’s never been a better time to begin tinkering with the API. It definitely helped me improve Goreport. Jordan released a couple of blog posts discussing the sample database feature and its development that are worth a read:

If you are a current (or future) user of Gophish, I hope you find Goreport to be a handy reporting tool.