Early in the development of Sagan “xbits,” we noticed a couple of limitations.
For one, Sagan didn’t have the ability to “remember” xbits between process restarts meaning that that all the data Sagan had been collecting (xbits, threshold, etc.) would be lost during a reboot or process restart.
The second issue was that each Sagan process was its own “island”. That is, Sagan processes could not “share” xbit data. For example, if you had one Sagan process analyzing and processing Windows logs and another Sagan process analyzing Cisco logs, these two processes would have their own internal stack, memory resources, etc., and these resources could not be shared between the two processes. In short, if the Windows Sagan process saw something “bad” happening, there was no means to “share” that data with the Cisco Sagan process.
To address these two issues, we made Sagan use “memory mapped” files (https://en.wikipedia.org/wiki/Mmap) which allow Sagan to “remember” data between system reboots and process restarts. This also allowed for “Inter-Process Communications” (IPC) between Sagan processes.
Sagan’s island got a little larger. Rather than Sagan only being able to use xbit data within its own process space, it could now share xbit data with other processes. The size of the island went from the Sagan process, to the machine Sagan is running on.
This was a good start, but I almost immediately started thinking about sharing outside the new “island”. What if I have multiple Sagan servers and I want to share xbit data between the servers? What if some of those machines were on different networks on different continents? It is not uncommon for us to deploy Sagan in global networks.
Use cases for a “global xbit” are pretty easy to come up with. For example, if Sagan detects some suspicious traffic on our Japan sensor, it should be able to share that data with the United States sensors via xbits. If “recon” and “brute force” activity are detected to be occurring in Florida, and I see a successful login from that same source IP on a California system, I’d likely want to look into this activity. Ideally, Sagan could put the two events together regardless and generate an “alert” for this activity.
The idea we are now developing is to make the Sagan island larger; from the machine to the network: a “global” xbit.
To power this concept, we are using Redis (https://redis.io). To quote from the Redis site:
“Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries”
To be honest, when I first read about Redis years ago, I didn’t understand the concept. I already handle data traditionally in memory. Why would I use Redis?
The answer became apparent when I started thinking about “global” xbits. Sagan uses Redis as a shared memory and process communication platform. Xbits really end up being a great target for Redis storage engine.
Redis is really fast and simple to use. Sagan xbits are basically small memory objects; they simply store the xbit name, source IP, destination IP, signature ID, expiration time, and creation timestamp. In a very busy environment, Sagan typically only has a few dozen active xbits at any given moment.
Using a standard SQL engine would add too much complexity and overhead. While we want xbits to stay persistent, they aren’t really a target for long-term storage. A typical xbit is only active for a few seconds and 12-24 hours is usually the max lifetime of an xbit. Redis also stores data in RAM on the Redis server. Accessing data on the Redis server is very fast.
Sagan YAML Redis Configuration:
As previously stated, Sagan uses mmap files for storing xbits. If you’d like to use Redis as your back-end storage for xbits, you’ll need to do a couple of things.
First off, keep in mind that this is highly experimental code!
Second, you’ll need to get the latest Sagan development code from https://github.com/beave/sagan. This version (1.1.8-git) contains the experimental Redis code.
Next, you’ll need to install the HiRedis C library which can be found at https://github.com/redis/hiredis or by doing an “apt-get install libhiredis-dev” on Debian/Ubuntu systems.
Once you complete that, you’ll need to configure Sagan (“./configure”) with the “–enable-redis” option. You can then proceed with the standard “make && make install”.
After compiling, you’ll need to add a few lines to your “sagan.yaml” files. First, you’ll need to switch the xbit storage engine which can be done by adding the following line to the “core” section of your configuration:
xbit-storage: redis
You’ll then need to enable Redis which can be done by adding a new “redis-server” configuration. For example:
redis-server:
enabled: yes
server: {server IP address}
port: 6379
password: “mypassword” # Comment out to disable authentication.
To see a complete Sagan configuration with Redis options, see https://github.com/beave/sagan/blob/master/etc/sagan.yaml
That’s it! Repeat on all of your Sagan servers. As long as all your Sagan instances can communicate with the primary Redis server, they’ll exchange xbit data!
Using “global” xbits in the Future: Sagan is a log analysis engine but I sometimes hear myself referring to Sagan as “glue”. Xbits play an important role in “gluing” events together. One of my goals in this experiment is to extend xbits in Redis to Suricata (https://suricata-ids.org). If you aren’t already playing with Suricata, I highly advise doing so; it is my second favorite open source project!
;)
The idea would be to allow Sagan to gather data from Suricata processes and visa-versa.
Final Notes: We’d love to get feedback on using Redis as a xbit storage engine since this is very experimental code. I’m also new with working with Redis, so I’d love to get design and input from the Redis community. The xbit Redis code can be found at: https://github.com/beave/sagan/blob/master/src/xbit-redis.c
References:
https://suricata-ids.org
http://www.cipherdyne.org/blog/2013/07/crossing-the-streams-in-ids-signature-languages.html
https://github.com/beave/sagan/blob/master/etc/sagan.yaml https://redis.io https://github.com/redis/hiredis
https://github.com/beave/sagan/blob/master/src/xbit-redis.c https://en.wikipedia.org/wiki/Mmap
