Since being created and open sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. What is Kafka?Īpache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Even with remote destinations and more elaborate processing the performance is usually considered “stunning”.
Rsyslog can deliver over one million messages per second to local destinations when limited processing is applied (based on v7, December 2013). While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to accept inputs from a wide variety of sources, transform them, and output to the results to diverse destinations. It offers high-performance, great security features and a modular design. Rsyslog is a rocket-fast system for log processing. Rather than approving each table as a separate pull request, Mitchell took the opportunity to add a native SQLite parsing method to osquery, which would allow adding any number of new virtual tables on a customizable basis. ATC was added to osquery by Mitchell Grenier (obelisk) in response to a number of virtual table pull requests which all functioned by parsing SQLite databases. What are Osquery ATC tables?ĪTC (automatic table construction) is a method which can expose the contents of local SQLite database file as an osquery virtual table. With Osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes. This allows you to write SQL-based queries to explore operating system data. Osquery exposes an operating system as a high-performance relational database. The “future improvements” section discusses various improvements for this implementation. Second, this blog post contains setups and configurations that may NOT be production-ready. This post will NOT cover how Osquery, Kafka, Rsyslog, VirusTotal, Spunk, Homebrew for macOS, or how Docker works, therefore this post assumes you know how these technologies work. This blog post is written to be a proof of concept and not a comprehensive post. Deploying and creating a logging pipeline with Kafka, Rsyslog, Python, and Spunk on Docker.Detect malicious URLs that users are browsing too with VirusTotal.Log user browsing activity with Osquery.In addition to URL collection, this PoC will use VirusTotal to enrich the logs with a ranking of the URL to detect malicious activity. This proof of concept (PoC) uses open-source software, it doesn’t require a certificate on each endpoint, just Osquery, is not affected by certificate pinning, and this PoC will work on or off the network. However, web proxies are costly, they require a cert on every device for SSL termination, certificate pinning prevents SSL inspection, and a web proxy will only work when devices are on the network. Many organizations will monitor the browser activity of their users using a web proxy. In this blog post, we will use Osquery to monitor the browser activity of users. Once this pipeline has been implemented, your security team will have the ability to protect your user’s from today’s most serious threats on the web. In addition to VirusTotal, this PoC will utilize Rsyslog, Osquery, Kafka, Splunk, Virustotal, Python3, and Docker as a logging pipeline. Not only will this PoC collect browser activity, but it will also use VirusTotal to rank each URL to detect malicious activity.
OSQUERY TABLES HOW TO
This proof-of-concept (PoC) will demonstrate how to use Osquery to monitor the browser activity of users.
0 kommentar(er)
