Automation-Driven SecOps: Making Needles Pop-out of the Haystack

By Roland Dela Paz, Senior Information Security Engineer, Docusign

Recently, we talked about Pescatore – our in-house built Phishing classification system that was recently awarded as an honoree of the 2019 CSO50 Award from IDG’s CSO. This is something we are really grateful and proud of, as Pescatore represents one of our core focuses in Docusign’s Security Operations—automation.

In this post, we will talk about how we strive to achieve an automation-centric environment through Houston – our Security Orchestration Automation and Response (SOAR) system that we built in-house. We have been using Houston for over three years now and it has immensely helped us scale, allowing us to process about 250K security events per day.

Centralized events management

In our environment, we use a combination of third-party and in-house security tools to get the visibility and contextual data that we need. The events and alerts coming out of these tools are routed to and processed centrally by Houston.

Houston is our in-house Security Automation and Orchestration system and is essentially a collection of agents. Each agent is responsible for one specific task. However, these agents work collectively to perform our intended events processing before it reaches our Incident Response (IR) queue. This includes triaging alerts, deduplication, and data enrichment, to name a few.

Below is an overview of this environment:

Through this setup, we are able to micro-manage our alerts, almost independent from the limitations of third-party security software. For us, this is extremely important. It allows us to efficiently find “needles in a haystack” and ensure that Incident Response resources are focused only on alerts that are relevant!

But how is a centralized automation possible?

Centralizing data and alerts management are possible by interfacing with multiple Application Programming Interfaces (APIs) from various security tools. Houston allows for specific agents to be setup to interface with APIs from security tools as needed. These agents then feed information into what we call a Houston “flow” where the collected data will be processed according to its type. A Houston flow is a cluster of agents within Houston that work with each other to achieve a specific task. It can also work with other Houston flows to perform even more complex events management. Needless to say, our team set ups these Houston flows—and this is where the events micro-managing magic happens!

Let’s look at our Houston flow for phishing as an example. The flow chart below provides a rough overview of what happens when someone sends a phishing sample to spam@docusign.com:

When someone forwards a phishing email to spam@docusign.com, the email sample will be pulled from the mailbox by a Houston agent. Embedded URLs will then be extracted, vetted, and submitted to Pescatore for analysis. URLs confirmed by Pescatore as bad are then checked across our environment via our “Backtrace” flow. If an employee is detected to access a known bad URL, an incident will be created accordingly. Obviously, this is an oversimplified example, but you get the idea!

Clearing up the noise

Third-party security tools are important to any security operations environment. They provide the visibility you very much need, but at the same time they may also provide a significant number of irrelevant alerts that ultimately translates to noise in your event logs.

The centralized data and events management that Houston allows for means that you can manage all upcoming alerts in a central location. Simply put, you can simply process all those alerts that are known to be relevant and discard those that aren’t!

Automation-driven incident response

I presume by now you have a rough idea how this is possible through Houston. It is also worth noting that we have setup our independent incident queue management platform. This allows us to manage incident tickets in an automated fashion more easily, via its API.

Since our security alerts are processed by Houston before it reaches our IR queue, we can do a lot with them. Here are some examples.

1. Incident Response

Things that our Incident Responders used to do manually are already automated. For instance, when a machine is detected to run a suspicious Microsoft Office macro, it is automatically isolated from the network. Employees who click known phishing sites, on the other hand, receive an automated “you got phished” email alert from Houston.

2. Incident Enrichment

Incident tickets are enriched with applicable data including but not limited to:

• Affected employee’s details (department, office location, etc.)

• If relevant to the incident, any suspicious logins from an affected employee in the last 24 hours

• A link to the internal Standard Operating Procedure (SOP) page relevant to the incident

• Incident priority

3. Cross Team Collaboration

In some cases, we need to collaborate with other teams to resolve an incident i.e. pulling a machine for forensics. In those cases, we have to create a ticket to our IT Team so that they can collect the machine for us.

Houston is able to automatically generate those tickets, track the progress and update the corresponding incident ticket on our end whenever there is any update from our IT Team.

4. Enhanced Visibility using Threat Intelligence

We have Houston flows for pulling fresh Indicators of Compromise (IOC) from public and commercial sources. This then gets disseminated in our environment immediately, adding another layer of protection on top of what we already have.

Additionally, pulling Threat Intelligence through Houston allows us to be alerted when a malicious campaign is using the Docusign brand. This enables us to alert the public via our Trust Center site.

5. Threat Intelligence Sharing

It is important for us to participate in sharing IOCs to the security community. We have Houston flows that forwards IOCs that we found to Facebook ThreatExchange for example. Additionally, Docusign phishing sites are submitted to Google Safe Browsing.

6. Brand Abuse Takedowns

Domains verified by Pescatore as phishing, that are abusing the Docusign brand, are submitted for takedown through Houston. In the last six months, we have reported a total of 2,538 Docusign-themed phishing domains.

Closing Remarks

We hope that this post provided a little bit of insight into how we approach Information Security within Docusign.

By congregating security events to an in-house Security Orchestration and Automation system that we have fully customized, we are able to better manage the sea of events we pick up and accurately alert on relevant security events. The 250K events that Houston processes every day is impossible for humans to process manually. Yet, it highlights the importance of automation for a Security Operations Center to scale.

If you want to share a malicious email that is abusing the Docusign brand, please feel free to forward to us at spam@docusign.com.