New Packet Logging and File Carving

carvingPacket Logging and File Carving

Being able to go back and look at the payloads or files transmitted on a network is extremely useful for several reasons:

  1. If you do not have the payload, you cannot really prove malicious intent, and legally you are on the hook.
  2. Payloads/Files are the ultimate forensic tool to decide if a particular incident is a false positive or a true positive.
  3. In more advanced systems payloads can also be used to find false negatives (things should have caused a security event but did not).

Obviously logging all data transferring on a network is challenging because disk space is limited and disks are relatively slow.

The MetaFlows Security System Logging Approach

Our overall approach to overcoming logging limitation is:

  • We store Payloads/Files that are associated with a specific security alert (using the time and the source/destination addresses and ports for identification)
  • When logging proactively (to also see Payloads/Files that do not involve a security alert), keep the disk at 90% utilization or below a certain number of Gigabytes by deleting the older logs.

This scheme gives you certainty of access if there is an incident and a time window to go back in time to look for certain things that might have been overlooked.

Recent Improvements

The Logging and File carving system has been vastly improved by the following:

  1. We now index the packets based on IP addresses using a proprietary approach. Instead of looking for particular packets in a big bucket full of files, the files are divided in smaller buckets each representing a subset of the addresses. This indexing scheme slows down packet logging a bit but makes looking for packets about 200 times faster!
  2. We added the ability to specify user-defined logging policies. Once a policy hits, the logging system prioritizes all packets for the matching policy and stores the Files/payloads in a separate high-priority repository which takes precedence over the normal logging. We will make a separate announcement on the policy specification because it is quite powerful and complex, and requires a dedicated post. For now, the only logging policy is to prioritize any packets involved in high priority events. In the future users will be able to customize more precise ad-hoc policies based on IP addresses, ports, and type of alerts.
The new carving system is backward compatible and automatically converts the existing packet logs stored on the sensor hard drive to the new indexing scheme. This process can take from a few minutes to days depending on your disk size. While this conversion takes place, queries on older logs may not return any data.

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Posts

Recent Comments

Archives

Categories