Network Security Monitoring: Tools, Techniques, and Alerts

Perimeter security is no longer sufficient. In cloud-native environments, the "network perimeter" dissolves into API gateways, service meshes, VPCs, and hundreds of microservices that communicate across trust boundaries. Network Security Monitoring (NSM) provides the visibility to detect threats as they traverse these networks — not just at the edge.

Effective NSM answers three questions: What is connecting to what? What are they sending? Does it match known-bad patterns or anomalous baselines?

The NSM Framework

NSM operates on three data types:

Full packet capture (FPC) — Complete network sessions. Maximally detailed but storage-intensive. Used selectively for forensics.
Session/flow data — Metadata: who talked to whom, when, how long, how much data. Much cheaper to store. NetFlow/IPFIX.
Alert data — Events generated when traffic matches known signatures (IDS/IPS) or statistical anomalies.

A mature NSM program uses all three: flow data provides breadth and historical depth, FPC provides forensic detail, and alerts provide real-time signal.

Intrusion Detection vs. Intrusion Prevention

IDS (Intrusion Detection System) monitors traffic and generates alerts. It does not block traffic.

IPS (Intrusion Prevention System) monitors traffic and can block or drop malicious packets inline.

In practice, the distinction has blurred — most modern tools can operate in either mode. The choice between detect-only and prevent-and-detect depends on your tolerance for false positives causing legitimate traffic drops.

Deployment modes:

Network TAP — Passive copy of network traffic. Zero impact on traffic flow. Used for IDS.
Inline — Traffic flows through the sensor. Required for IPS. Introduces latency and a single point of failure if the sensor fails.
Cloud mirroring — AWS VPC Traffic Mirroring, GCP Packet Mirroring. Sends copies of VPC traffic to NSM tools.

Suricata: Open-Source Network IDS/IPS

Suricata is the leading open-source network IDS/IPS. It supports signature-based detection using Suricata rule syntax (compatible with Snort rules), protocol parsers for 50+ protocols, and file extraction.

Key Suricata capabilities:

Pattern matching on packet payloads
Protocol anomaly detection (malformed HTTP, DNS abuse, etc.)
TLS certificate inspection
File extraction and hashing for malware detection
Lua scripting for custom detection logic

Sample Suricata rule detecting potential SQL injection:

alert http any any -> $HTTP_SERVERS any (
  msg:"SQL Injection Attempt";
  flow:established,to_server;
  http.uri;
  content:"UNION SELECT";
  nocase;
  pcre:"/UNION\s+SELECT/i";
  classtype:web-application-attack;
  sid:1000001;
  rev:1;
)

Emerging Threats (from Proofpoint) provides free and paid Suricata/Snort rule sets updated daily based on current threat intelligence. Run suricata-update to pull the latest rules:

suricata-update update-sources
suricata-update enable-source et/open
suricata-update

NetFlow Analysis

NetFlow (Cisco) and IPFIX (the IETF standard) are flow export protocols. Network devices export metadata about each TCP/UDP flow — source/destination IP and port, bytes transferred, packet count, protocol, timestamps.

Why NetFlow matters:

Full packet capture of a 10 Gbps network link generates ~1 TB per day. NetFlow metadata for the same traffic generates ~100 MB. For long-term historical analysis, flow data is the practical choice.

Key NetFlow use cases:

Lateral movement detection — Unusual server-to-server connections that do not match normal topology
Data exfiltration detection — Large outbound transfers to unusual destinations
Port scanning — A host connecting to dozens of IPs on the same port in a short window
Beaconing — Periodic outbound connections to the same external IP (C2 communication)

Tools for NetFlow analysis:

ntopng — Real-time traffic analysis with NetFlow/IPFIX input
SiLK — CERT NetSA tools for large-scale flow analysis
Elastic/OpenSearch — Ingest NetFlow data and query with KQL

Beaconing detection query (Elasticsearch):

{
  "aggs": {
    "by_dest": {
      "terms": { "field": "destination.ip" },
      "aggs": {
        "connection_times": {
          "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "5m"
          }
        },
        "stddev": {
          "extended_stats": {
            "field": "@timestamp",
            "sigma": 2
          }
        }
      }
    }
  }
}

Low standard deviation in connection intervals (very regular timing) is a beaconing indicator.

SIEM Integration

A SIEM (Security Information and Event Management) system aggregates logs and events from across the environment, correlates them, and generates alerts.

Log sources for network security:

VPC flow logs (AWS, GCP, Azure)
Firewall logs (allow and deny)
DNS query logs
Proxy/web gateway logs
Load balancer access logs
IDS/IPS alerts (Suricata, Snort)
Network device syslogs

High-value SIEM correlation rules for network security:

Impossible travel — Same user account authenticating from two geographic locations within a timeframe inconsistent with travel
Port scan — Single IP connecting to 20+ distinct destination ports within 60 seconds
DNS tunneling — High-entropy DNS query names, unusually long subdomains, or high query frequency to a single domain
Beaconing to newly registered domain — Outbound connection to a domain registered within the last 30 days
Internal reconnaissance — Host making NetBIOS, LDAP, or SMB connections to dozens of internal hosts
Data exfiltration — Outbound transfer exceeding a threshold (e.g., 500 MB) to a destination not in an allowlist

Example Sigma rule (SIEM-agnostic format):

title: DNS Tunneling Detection
status: experimental
description: Detects potential DNS tunneling based on query length
logsource:
  category: dns
detection:
  selection:
    QueryName|re: '^[a-z0-9]{30,}\.'
  condition: selection
falsepositives:
  - Long DKIM or SPF records
  - CDN domains with encoded parameters
level: medium

Sigma rules can be converted to Splunk SPL, Elastic KQL, Sentinel KQL, and other SIEM query languages using sigmac or sigma-cli.

Managing Alert Fatigue

Alert fatigue is the single biggest threat to NSM effectiveness. When analysts receive 1,000 alerts per day and 990 are false positives, they stop paying attention.

Strategies to reduce noise:

Establish a baseline first. Run your detection rules in observe-only mode for 2 weeks before enabling alerting. Identify legitimate traffic patterns that generate false positives and add suppression rules.
Tier your alerts. Not every alert requires immediate response. Define:
- Critical — Immediate response (automated blocking + human review)
- High — Response within 1 hour
- Medium — Analyst review within 4 hours
- Low — Batch review daily
Use thresholds and aggregation. A single failed login is noise. Ten failed logins from the same IP in 5 minutes is an alert. Configure thresholds before alerting.
Enrich alerts automatically. Before an alert reaches an analyst, automatically enrich it with:
- IP reputation (AbuseIPDB, VirusTotal)
- Domain age and registrar
- Geolocation
- Historical context for the involved host
Track true positive rate per rule. Retire or retune rules with less than 10% true positive rate over 30 days.
Automate tier-1 triage. Use SOAR (Security Orchestration, Automation, and Response) to auto-close known false positives, gather enrichment data, and only escalate confirmed or ambiguous alerts to humans.

NSM is an ongoing practice, not a one-time deployment. Detection rules must evolve as the environment changes, new threats emerge, and attacker TTPs shift.