Parsing atop files with python dissect.cstruct

Like you’ve probably read, Fox-IT released their incident response framework called dissect, but before that they released the cstruct part of their framework. Ever since they released it publicly I’ve been wanting to find an excuse to play with it on public projects. I witnissed the birth of cstruct back when I was still working at Fox-IT and am very happy to see it all has finally been made public, it sure has evolved since I had a look at the very first version! Special thanks to Erik Schamper (@Schamperr) for answering late night questions about some of the inner workings of dissect.cstruct.

This is one of those things that you can encounter during your incident response assignment and for which life is a bit easier if you can just parse the binary file format with python. Since with incident response you never know in which format exactly you want to receive the data for analysis or what you are looking for it really helps to work with tools that can be rapidly adjusted. python is an ideal environment to achieve this. An added benefit of parsing the structures ourselves with python is that we can avoid string parsing and thus avoid confusion and mistakes.

The atop tool is a performance monitoring tool that can write the output into a binary file format. The creator explains it way better than I do:

Atop is an ASCII full-screen performance monitor for Linux that is capable of reporting the activity of all processes (even if processes have finished during the interval), daily logging of system and process activity for long-term analysis, highlighting overloaded system resources by using colors, etc. At regular intervals, it shows system-level activity related to the CPU, memory, swap, disks (including LVM) and network layers, and for every process (and thread) it shows e.g. the CPU utilization, memory growth, disk utilization, priority, username, state, and exit code.
In combination with the optional kernel module netatop, it even shows network activity per process/thread.

The atop tool website

Like you can imagine, having the above information is of course a nice treasure throve to find during an incident response, even if it is based on a pre-set interval. For the most basic information, you can at least extract process executions with their respective commandlines and the corresponding timestamp.

Since this is an open source tool we can just look at the structure definitions in C and lift them right into cstruct to start parsing. The atop tool itself offers the ability to parse written binary files as well, for example using this commend:

atop -PPRG -r <file>

For the rest of this blog entry we will look at parsing atop binary log files with python and dissect.cstruct. Mostly intended as a walkthrough of the thought process as well.

You can also skip reading the rest of this blog entry and jump to the code if you are impatient or familiar with similar thought processes.

Continue reading “Parsing atop files with python dissect.cstruct”

Notes on ZFS / Solaris forensics

A while ago I wrote a script to perform what I called poor man’s forensics. The script was meant as a way to utilize the native operating system to extract some minimal data from exotic filesystems to be able to create a timeline and identify possible abnormalities. As a reminder to myself here are some additional raw notes, commands and resources on performing (forensic || incident response || compromise assessments) investigations on ZFS / Solaris environments. I encountered ZFS / Solaris during some of the FoxCert investigations I participated in.

These raw notes are by no means complete and you must definitely not follow these blindly and always ensure you are working on a copy of a copy of a copy of the real evidence.

Continue reading “Notes on ZFS / Solaris forensics”

[python] Poor man’s forensics

So after a period of ‘lesser technical times’ I finally  got a chance to play around with bits, bytes and other subjects of the information security world.  A while back I got involved in a forensic investigation and participated with the team to answer the investigative questions.  This was an interesting journey since a lot of things peeked my interest or ended up on one of my todo lists.

One of the reasons that my interest was peeked is that yes, you can use a lot of pre-made tools to process the disk images and after that processing is done you can start your investigation. However, there are still a lot of questions you could answer much quicker if you had a subset of that data available ‘instantly’. The other reason is that not all the tools understand all the filesystems out there, which means that if you encounter an exotic file system your options are heavily reduced. One of the tools I like and which inspired me for these quick & dirty scripts is ‘mac-robber‘ (be aware that it changes file times if the destination is not mounted read-only) since it’s able to process any file system as long as it’s mounted on an operating system on which mac-robber is able to run. An example of running mac-robber:

sudo mac-robber mnt/ | head

You can even timeline the output if you want with mactime:

sudo mac-robber mnt/ | mactime -d | head
Date,Size,Type,Mode,UID,GID,Meta,File Name
Thu Jan 01 1970 01:00:00,2048,…b,dr-xr-xr-x,0,0,0,”mnt/.disk”
Thu Jan 01 1970 01:00:00,0,…b,-r–r–r–,0,0,0,”mnt/.disk/base_installable”
Thu Jan 01 1970 01:00:00,37,…b,-r–r–r–,0,0,0,”mnt/.disk/casper-uuid-generic”
Thu Jan 01 1970 01:00:00,15,…b,-r–r–r–,0,0,0,”mnt/.disk/cd_type”
Thu Jan 01 1970 01:00:00,60,…b,-r–r–r–,0,0,0,”mnt/.disk/info”

Now that’s pretty useful and quick! One of the things I missed however was the ability to quickly extend the tools as well as focus on just files. From a penetration testing perspective I find files much more interesting in an forensic investigation than directories and their meta-data. This is of course tied to the type of investigation you are doing, the goal of the investigation and the questions you need answered.

I decided to write a mac-robber(ish) python version to aid me in future investigations as well as learning a thing or two along the way. Before you continue reading please be aware that:

  1. The scripts have not gone through extensive testing
  2. Thus should not be blindly trusted to produce forensically sound output
  3. The regular ‘professional’ tools are not perfect either and still contain bugs ;)

That being said, let’s have a look at the type of questions you can answer with a limited set of data and how that could be done with custom written tools. If you don’t care about my ramblings, just access the Github repo here. It has become a bit of a long article, so here are the ‘chapters’ that you will encounter:

  1. What data do we want?
  2. How do we get the data?
  3. Working with the data, answering questions
    1. Converting to body file format
    2. Finding duplicate hashes
    3. Permission issues
    4. Entropy / file type issues
  4. Final thoughts

Continue reading “[python] Poor man’s forensics”