- a virus is detected on a VM and is quarantined by the AV solution
- the AV solution tags the VM with an NSX security tag
- NSX places the VM in a new Security Group, whose network policies steer all VM traffic through an IPS
- vCenter Orchestrator monitors the security group for changes and when a VM is added
- a snapshot of the VM is taken for forensic purposes
- a vSpan session (RSPAN) is set up on the Distributed Virtual Switch to begin capturing inbound/outbound traffic on the VM
- once the VM has been removed from the security group, the vSpan session is removed
You will note that there is a portion of the workflow that is handled natively by NSX (Security Tag reaction, Security Group policy) but the snapshot and RSPAN are done via vCO workflow.
If you are interested in exploring this capability, I have provided the vCO workflow package for download. This is provided as-is and you should fully test it (and modify as needed) before using in your environment.
To use the workflow package you will need (assuming you also have NSX, vShield Endpoint and some third party integration already set up):
- vCO 5.5.2
- the NSX plugin for vCO (installed and configured)
- the REST plugin with your NSX manager added as a REST host
- vCenter plugin configured
The workflow package includes a good number of "helper" workflows which you will not need to run directly. The master workflow is in the root folder Security Reaction and is named "Set up VM Forensics RUN THIS" (just in case you had any doubt as to which one to run).
|The Security Reaction Master Workflow|
Running the master workflow will prompt you for three items -
- The NSX Security Group to monitor. This is why the NSX plugin is required, so that you can browse the vCO managed objects and locate the desired Security Group.
- A time to sleep in seconds. The master workflow will run continuously until manually stopped and will use a REST call to NSX to get the current membership for the Security Group. We have no recommendation on this poll time, although in testing we used 5-10 seconds. It would have been better to use some external event to kick off the vCO workflow but we could not find a way to do this from NSX. It may be possible to do via the partner solution, but we wanted this workflow package to be "partner neutral."
- Destination IPv4 address. This is the destination for the RSPAN (or vSpan session in vSphere API terms). The vSpan session is created with some defaults (for example sampling rate, normal traffic allowed, etc). If you want to change any of those properties, you will need to modify the Helper workflow named "Configure encapRemoteMirrorSource vSpan Session on DVS" (modify the "Create Port Mirror" script task).
Also note that this workflow doesn't support VMs with multiple vNICs - specifically, it will only create an RSPAN that includes the first vNIC found on a VM. You can modify the Helper workflow "Implement Forensics" and adjust the script task "Prep for Mirror Creation" so that the additional NICs (if any) are added to the sourcePorts array. It's something we intended to fix but forgot about until after our final testing and video production - so as they say in the textbooks "this is left as an exercise for the reader."
Of course, there are many other actions that can be taken besides setting up an RSPAN and getting a snapshot. This solution can be extended to practically any task required during such an event such as creating a ticket in your service desk software, spinning up additional workloads to replace the compromised VM, sending emails, guest OS file system operations... all of these and more can be accomplished using vCO in conjunction with NSX.
Hadar Freehling - @dfudsecurity - is a Security and Compliance Systems Engineer Specialist with VMware and jointly contributed to this solution and blog post.