What Is AI-Based Encrypted Traffic Analysis?

AI-based encrypted traffic analysis uses AI to identify suspicious or malicious encrypted network flows. This is generally accomplished by analyzing unencrypted attributes of encrypted network traffic, with minimal or no decryption of the payload.

Encrypted network traffic has been increasingly common as a means of enhancing online privacy and security. Google and other organizations have pushed strongly for the use of encrypted network protocols, resulting in over 95% of network traffic being encrypted today, according to Google’s Transparency Report. While this is a positive for network privacy, it limits security visibility for these network flows, potentially leaving organizations vulnerable to malware and other threats.

AI models can examine metadata, behavior, and flow patterns of encrypted traffic to analyze the likely risk associated with these flows. As part of an integrated SSE or SASE architecture, AI-based encrypted traffic analysis can enhance network performance and security by reducing the need to perform resource-intensive decryption of network traffic.

Why Encrypted Traffic Creates Blind Spots for Security

Transport Layer Security (TLS) is a network protocol that wraps insecure protocols within an encrypted wrapper for confidentiality and authentication. The use of TLS for web traffic has become commonplace, enhancing online privacy. However, this growing adoption of traffic encryption also benefits attackers who attempt to conceal malicious content within encrypted traffic flows. Network traffic containing malware, command and control traffic, and data exfiltration may be opaque to security tools and hide among legitimate traffic.

Many security tools depend on the ability to analyze the payload of network traffic, which TLS encrypts. As a result, the growing use of traffic encryption creates significant security blind spots.

How Do Traditional Decryption-Centric Approaches Work?

Traditionally, enterprises handle the challenge introduced by network traffic encryption by performing decryption and inspection of the traffic. This can be accomplished in various ways, such as the use of network proxies, on-prem appliances, or cloud gateways. This approach to managing encrypted network traffic offers deep visibility into the network traffic. By decrypting the entire payload, a security appliance can scan its entirety for potential malicious content.

However, full traffic decryption also has its tradeoffs. Full traffic decryption is computationally expensive and can introduce additional latency into the connection. Additionally, it requires the organization to break the traditional TLS paradigm using enterprise SSL certificates, which introduces security risks and concerns about user privacy.

What Is AI-Based Encrypted Traffic Analysis?

AI-based encrypted traffic analysis addresses the challenges associated with encrypted network traffic by focusing on the unencrypted aspects of the traffic. Packet metadata, such as IP addresses and packet sizes, and features of traffic flows can provide hints that help with differentiating between legitimate and malicious network traffic. The more high-quality telemetry collected and analyzed by these systems, the more accurate their conclusions.

AI-based encrypted traffic analysis systems can use various models to detect potential malicious traffic. Supervised machine learning (ML) can be used to detect known malicious traffic patterns, while unsupervised ML and anomaly detection help to identify novel behaviors.

Since encrypted traffic analysis doesn’t decrypt payloads, it protects user privacy while still identifying likely risky and suspicious connections. This allows security appliances to selectively decrypt and inspect TLS sessions that are identified as the most risky.

How Encrypted Traffic Analysis Protects Privacy

Encrypted traffic analysis focuses on analyzing metadata, including flow duration, packet sizes, timing, TLS handshake features, SNI, and certificate attributes. This information isn’t encrypted in TLS sessions because it is less sensitive than the packet payloads. By avoiding decryption of packet payloads, where user session data is located, these tools protect user privacy.

How Does AI Detect Threats in Encrypted Traffic Without Decryption?

AI can analyze traffic metadata to detect common forms of malicious network traffic, such as command and control traffic, data exfiltration, and lateral movement. Some key patterns that AI can recognize in encrypted traffic include:

• Beaconing
• Use of unusual TLS ciphers
• Unusual traffic destinations
• Mismatched SNI and certificates

These indicators can be used to identify likely malicious traffic within an enterprise environment. These network flows can be flagged for decryption by intrusion prevention systems (IPS) and anti-malware engines, balancing security, privacy, and performance.

Examples of AI Features and Models in Use

AI-based encryption traffic analysis solutions can use various features to identify malicious traffic. These include:

• TLS handshake fingerprints
• Flow statistics
• Server reputation scores
• Protocol anomalies
• Behavior baselines

AI-based tools can employ supervised or unsupervised learning to identify threats. Supervised models are trained on labeled malicious/benign traffic to identify known threats, and unsupervised models spot deviations from “normal” behavior for novel threat detection. Additionally, some platforms incorporate threat intel feeds and rule-based logic alongside AI.

Which TLS Handshake and Flow Attributes Matter Most?

The TLS handshake is used to negotiate a TLS session and contains a wealth of useful information that can be used to differentiate legitimate and malicious traffic. Some important fields that AI-based encrypted traffic analysis solutions examine include:

• Ciphers
• Versions
• Certificate fields
• SNI
• Session resumption behavior
• JA3-style fingerprints

Legitimate traffic from user sessions in web browsers will have certain values for these attributes. Therefore, any deviations or anomalies can indicate spoofed services, malware frameworks, or covert channels. By looking for these warning signs, AI-based encrypted traffic analysis tools can potentially differentiate C2 traffic and other malicious data flows.

How TLS Inspection Fits With AI-Based Encrypted Traffic Analysis

TLS inspection and AI-based encrypted traffic inspection are complementary solutions. AI can efficiently analyze unencrypted metadata to identify the network flows that require decryption and more in-depth analysis.

To balance performance and security considerations, a TLS inspection methodology should follow best practices and define policies to determine when decryption should be performed and when it can be skipped. For example, a system may mandate that a network flow with an AI-generated risk score above a particular threshold requires decryption. Additionally, policies may require or forbid decryption for certain types of traffic, such as connections to banking and healthcare sites.

Benefits and Limits Of AI-Based Encrypted Traffic Analysis

Adding AI-based encrypted traffic analysis alongside selective, safe TLS inspection provides several benefits to the organization and its users, including:

• Improved Performance: AI analysis of unencrypted metadata can eliminate the need for computationally-intensive decryption, improving network performance.
• Enhanced Privacy: Selection decryption eliminates the need to decrypt and analyze low-risk network flows.
• Reduced Overhead: Selective decryption requires less computational power than full decryption, reducing cost and resource requirements.

However, while AI-based decryption is a useful tool, it has its limitations. Without good data and models, AI systems can make critical mistakes. Additionally, systems may have false positives or false negatives, leading to unnecessary decryption or missed threats.

How Should Security Teams Operationalize These Capabilities?

AI-based encrypted traffic analysis should be rolled out carefully to minimize the potential impacts on security and the user experience. Best practices include:

• Enabling available encrypted traffic analytics features on existing platforms and carefully tuning policies.
• Building playbooks to triage high-risk encrypted traffic alerts and correlate them with endpoint, identity, and application logs.
• Performing ongoing model tuning, feeding confirmed incidents and benign cases back into the system to enhance models.
• Clearly defining ownership between network, security, and compliance teams when encrypted traffic and privacy are involved.

FAQs about AI-Based Encrypted Traffic Analysis

How Is AI-Based Encrypted Traffic Analysis Different from SSL Decryption?

AI-based encrypted traffic analysis attempts to identify malicious traffic without decryption by looking at session metadata and other unencrypted information in an encrypted session. This may be less computationally expensive and invasive than full SSL decryption, making it a good option for identifying those sessions that merit deeper analysis of decrypted traffic.

Does Encrypted Traffic Analysis Replace TLS Inspection?

No, encrypted traffic analysis and TLS inspection are complementary solutions. By identifying likely malicious sessions, encrypted traffic analysis allows organizations to avoid the need to decrypt all traffic, focusing on risky sessions instead. This can enhance performance by reducing the use of resource-intensive decryption, and improve user privacy without compromising security.

What Types of Threats Are Best Detected With Encrypted Traffic Analysis?

Encrypted traffic analysis is best suited to identifying malicious network traffic with common structural patterns. Common use cases include identifying command and control (C2) traffic, data exfiltration, botnet communications, and malware.

How Does AI-Based Encrypted Traffic Analysis Impact User Privacy?

AI-based encrypted traffic analysis eliminates the need to decrypt and inspect every network session. This enhances privacy since the data contained in network packets that aren’t marked for further review is never inspected by the security appliance.

What Should Organizations Look for When Evaluating Solutions?

Encrypted traffic analysis solutions are only as good as the telemetry that they collect and analyze, and the models that they use to assess it. Other critical features include coverage of an organization’s entire IT environment and integration with other security tools, ideally as part of a converged SSE/SASE deployment.

How Do AI Security Vulnerabilities Affect Encrypted Traffic Analysis?

AI-based encrypted traffic analysis can be a powerful tool, but it is also potentially vulnerable to the same threats as other AI systems. Attackers may poison training data or identify ways to evade detection by an AI model. Additionally, blind spots in telemetry fed to the system and overreliance on AI signals for malicious traffic detection can enable attacks to slip through the cracks.

To manage these threats, organizations must implement strong governance models for encrypted traffic analysis. Ongoing monitoring, feedback loops, and clear policy guardrails are essential to manage potential risks to the system and the threat of missed detections.

At the end of the day, AI-based encrypted traffic analysis is a complementary tool to TLS inspection, providing educated guidance regarding the network flows that require decryption. A well-designed network security program combines TLS inspection, encrypted traffic analytics, and strong governance.