MSP Endpoint Management: Troubleshooting 'Guns Crew' and 'Lion Under the Table' Alert Behaviors

Introduction

How can MSPs reduce alert fatigue while accurately diagnosing endpoint incidents? Two perplexing alert patterns known colloquially as "Guns Crew" and "Lion Under the Table" behaviors often confuse IT ops engineers tasked with Remote Monitoring and Management (RMM). These terms describe specific alert characteristics that can indicate either legitimate threats or false positives.

This guide provides actionable steps to identify, troubleshoot, and resolve these alert behaviors. You'll learn how to improve alert quality, minimize noise, and streamline root cause analysis through log management and automation.

---

Prerequisites / What You Need

Before troubleshooting these alert behaviors, ensure you have:

• Access to your RMM platform's alert management dashboard (e.g., ConnectWise Automate, Datto RMM)
• Log management tools like Splunk, Graylog, or ELK Stack for deep diagnostics
• Remote access software capable of session auditing (e.g., TeamViewer, AnyDesk)
• Patch management system logs to review recent updates
• Network monitoring tools with alert correlation features (e.g., PRTG, SolarWinds)
• IT automation playbooks or scripts to reduce manual intervention

Do this now: Confirm your monitoring tools have alert tagging and categorization features to isolate 'Guns Crew' and 'Lion Under the Table' alerts effectively.

---

Step 1: Understand 'Guns Crew' and 'Lion Under the Table' Alert Behaviors

• Guns Crew alerts typically manifest as rapid-fire, repetitive notifications triggered by endpoint anomalies such as process crashes or sudden CPU spikes. They often inundate dashboards with noise.
• Lion Under the Table alerts are stealthy; they represent intermittent, low-volume warnings that hide in system logs or subtle endpoint behaviors, often escaping immediate detection.

Example: In a recent incident, a client's antivirus triggered over 200 'Guns Crew' alerts within an hour due to a misconfigured heuristic scan.

Do this now: Categorize your current alerts into these two groups by reviewing alert frequency and severity patterns.

---

Step 2: Filter and Reduce False Positives Using RMM Alerting Noise Reduction Techniques

1. Tune alert thresholds: Adjust CPU, disk, and memory usage thresholds to reflect normal operational baselines.
2. Implement alert suppression windows: Avoid repetitive alerts for the same event within short intervals.
3. Use alert correlation: Link related alerts across endpoints to identify true incidents versus duplicates.
4. Whitelist known benign processes: Exclude routine tasks causing false positives.

Tools & Metrics: Datto RMM's alert suppression reduced false positives by 35% in one MSP's environment.

Do this now: Review your RMM alert rules and apply suppression and correlation features to 'Guns Crew' type alerts.

---

Step 3: Perform Log Management Root Cause Analysis for 'Lion Under the Table' Alerts

• Collect endpoint logs from security, system, and application sources.
• Use centralized log management (e.g., Splunk) to search for subtle warning signs like failed authentications or process anomalies.
• Cross-reference timestamps of low-volume alerts with patch deployments and remote sessions.

Concrete Example: Splunk queries revealed that several 'Lion Under the Table' alerts coincided with failed patch installations on Windows 10 endpoints.

Do this now: Set up a log search query for anomalies occurring within 30 minutes before and after each 'Lion Under the Table' alert.

---

Step 4: Audit Remote Access Sessions to Verify Alert Validity

• Review session logs to identify unauthorized or unexpected remote connections.
• Confirm whether alert-triggering activities align with approved IT support interventions.
• Use session recording tools to capture suspicious behavior.

Example: TeamViewer session audits uncovered unauthorized access attempts triggering 'Guns Crew' alerts due to rapid credential failures.

Do this now: Schedule weekly audits of remote sessions focusing on endpoints generating frequent 'Guns Crew' alerts.

---

Step 5: Detect Patch Management Anomalies That Trigger Alert Behaviors

• Analyze patch deployment logs for failures, rollbacks, or delays.
• Identify if patch conflicts or incomplete installs correlate with alert spikes.
• Prioritize patch remediation on endpoints with recurrent 'Lion Under the Table' alerts.

Data Point: MSPs report a 20% decrease in endpoint alerts after implementing automated patch anomaly detection.

Do this now: Integrate patch management reports with your RMM alert system to flag endpoints with recent patch issues.

---

Step 6: Correlate Network Monitoring Alerts to Endpoint Behaviors

• Use network monitoring tools to detect unusual traffic patterns concurrent with alert occurrences.
• Correlate endpoint alerts with firewall logs or intrusion detection system (IDS) events.
• Identify lateral movement or external scanning attempts linked to 'Guns Crew' alert bursts.

Tool	Feature	Benefit
PRTG	Alert Correlation	Connect network & endpoint data
SolarWinds	Traffic Anomaly Detection	Early detection of threats

Do this now: Set up correlation rules between your network monitoring tool and RMM to automatically flag suspicious endpoint alert clusters.

---

Step 7: Implement IT Automation Playbooks to Expedite Response

• Develop automated remediation scripts for common 'Guns Crew' alert causes (e.g., restarting a service).
• Automate alert acknowledgment and suppression for known benign 'Lion Under the Table' events.
• Continuously update playbooks based on incident trends.

Example: Using ConnectWise Automate's scripting engine, one MSP automated resolution of 60% of 'Guns Crew' alerts within 10 minutes.

Do this now: Build or adapt playbooks targeting repetitive alert scenarios to reduce manual workload.

---

Common Mistakes to Avoid

• Ignoring low-volume alerts: 'Lion Under the Table' alerts may indicate stealthy threats; dismissing them risks missing root causes.
• Over-tuning alert thresholds: Excessive suppression can hide critical incidents.
• Neglecting session audits: Remote access remains a common vector for alert triggers.
• Failing to integrate data sources: Isolated endpoint or network data reduces diagnostic accuracy.

Do this now: Review your alert management policies quarterly to balance sensitivity and noise.

---

FAQ

Q1: What causes 'Guns Crew' alert behavior in endpoint monitoring? A1: Rapid, repetitive alerts often stem from misconfigured monitoring thresholds, runaway processes, or frequent process crashes that flood alert systems.

Q2: How can I distinguish between false positives and real threats in these alert types? A2: Correlate alerts with logs, remote access sessions, patch statuses, and network data to validate if an alert represents a genuine issue.

Q3: Are automation playbooks effective against these alert behaviors? A3: Yes, well-designed playbooks can automate common fixes and alert handling, reducing manual effort and improving response times.

Q4: Which tools are best for analyzing these alerts? A4: Tools like Splunk for log analysis, Datto RMM for alert management, TeamViewer for session auditing, and PRTG for network correlation provide comprehensive coverage.

Q5: How often should MSPs review alert tuning and automation rules? A5: At minimum, quarterly reviews ensure alert relevance and minimize noise as endpoint environments evolve.

---

Conclusion

Managing complex alert behaviors like 'Guns Crew' and 'Lion Under the Table' requires a multi-faceted approach combining alert tuning, robust log analysis, session auditing, patch anomaly detection, network correlation, and automation. By following these steps, MSPs can reduce false positives, improve incident accuracy, and optimize operational efficiency.

Do this now: Start by categorizing your alerts today, then gradually implement the outlined steps to build a resilient endpoint monitoring strategy that minimizes alert noise and maximizes actionable insights.