How IntelliFilter Enables Fast, In-Depth Analysis of Endpoint Activity

If you’ve paid attention to the security software market in the past few years, you’ve heard of a class of products known as Endpoint Detection and Response, or EDR. 

These products go beyond traditional endpoint protection solutions and provide visibility and tools to help discover and remediate advanced threats that may not be stopped by traditional anti-malware or intrusion prevention software.  We’re talking here about adversaries that don’t rely on typical, mass-deployed malware or hacking toolkits.

Rather, they target a specific organization using custom tools and techniques specifically designed not to trigger existing detectors. EDR technology helps threat hunters within an organization look for these stealthy attacks by monitoring, recording, and analyzing fine-grained behavioral information about every process running on the endpoint. 

Access to this rich information allows security analysts to scrutinize behavior on endpoints across the organization, facilitating the forensic investigation and detection of malicious behavior that might otherwise fly under the radar.  But efficiently processing a vast quantity of behavioral data is challenging.

Technology developed in Symantec Research Labs enables the processing and analysis of this high-volume, information-rich stream of events while having minimal impact on the performance of the endpoint.  The component that processes the stream on the endpoint is called the IntelliFilter.

We borrowed an old technique, commonly used in business rules systems, called the Rete algorithm and adapted it to our purposes.  In its basic form, the Rete algorithm is a technique for translating a set of declarative pattern-matching rules into an executable discrimination network that applies conditional tests to “facts” passed through the network.

In the traditional setting, these facts would be drawn from a knowledge base representing the state of the world and matching a rule would cause a business rules planning system to take an action, for instance automatically re-ordering inventory, or adding a newly derived fact to the knowledge base.  In our EDR setting, the facts are the fine-grained behavioral events that are generated on an endpoint, such as a user logging into the system, or a process writing to a particular registry key. 

The action taken upon finding a matching pattern might be generating an alert for a security analyst or updating behavioral summaries that can be used for machine learning or anomaly detection.

The beauty of the Rete algorithm is its ability to avoid redundant computation by (i) finding common conditions from different rules in a large rule set and evaluating them only once; and (ii) tracking sets of facts that partially satisfy a rule in preparation for the arrival of a fact that completes the match. 

The IntelliFilter component of Symantec’s EDR maintains these benefits, while adapting the algorithm to operate efficiently in a streaming setting where potentially thousands of behavioral events pass through the discrimination network every second.  Our enhancements greatly reduce memory usage through specialized scheduling and storage logic that retains events only as long as the rules require, and optimizations that vastly reduce the size of the discrimination network by identifying opportunities for node reuse in the portion of the network that joins multiple events together.

The end result is a system that can evaluate thousands of complex pattern specifications on a real-time stream of thousands of fine-grained behavioral events per second —  all without making a standard laptop break a sweat. 

While the IntelliFilter allows Symantec EDR to find patterns based on specified rules, this is not just another set of “signatures” for finding malware.  Why?  First of all, the granularity and quantity of the data being processed is exceptional.  Every file write, network connection, registry modification and more is analyzed for every process on an endpoint.

Furthermore, the expressive language we designed to write IntelliFilter rules allows an analyst to build up high-level behaviors, within or across processes, by joining together these low-level events.  Some of these high-level behaviors may generate a security or informational alert.  But we don’t stop there.  Non-alerting behaviors can be tracked and summarized, creating a sort of behavioral fingerprint for processes and system users.  These fingerprints are concise but contain a wealth of information about how programs and users behave.  This presents new opportunities for viewing, sharing, and analyzing endpoint activity. 

IntelliFilter gives us a flexible engine, capable of real-time pattern detection and summarization over the high-volume stream of granular events generated on an endpoint every second.  We at Symantec Research Labs are excited to wield this powerful tool to fashion new security analytics that uncover stealthy attacks perpetrated by advanced adversaries.