Try Buy

Efficient forensic workflow: Is your bridge a bottleneck?

Over the last few years we have been presenting seminars on accelerating forensic workflow at a range of conferences such as HTCIA, DFRWS, ACSC, & F3. This blog series aims to extract and expand the key themes in the seminar.

Is your bridge a bottleneck?

For around 4 years now, we have been benchmarking the throughput of a range of storage devices: SSD’s RAID’s, NVMe drives, regular spinning rust drives, and USB bridges (the subject of this post). The table below summarises our results for interconnects.

Not all bridges are created equal

The test methodology was to connect a drive with known performance characteristics via various interconnects. We used a benchmarking tool to see how fast we could read data through the various bridges from the drive. Through earlier testing with the same tools, we know that the test drive is capable of reading data at upwards of 500 MB/s when directly connected to a motherboard via SATA3.

In all cases the bridges are a bottleneck, limiting throughput.

The results fall into roughly three groups, being bridges that limit throughput at around 200 MB/s, 300 MB/s and 400 MB/s. For the USB3 bridges running at 200 MB/s the cause appears to be that they are using the USB Bulk Only Transport (BOT) standard (up until recently this was the only game in town for USB storage). The bridges with 400 MB/s figures employ the next-generation USB Attached SCSI Protocol (UASP), which enables faster speeds through more parallelism. The final group of bridges, the forensic write blockers, limit speeds to around 300MB/s.

When is your bridge NOT a bottleneck?

If you are reading single spinning rust disks the bridges aren’t a significant bottleneck. Today’s commodity 3.5” SATA drives pump data at a maximum of around 200 MB/s.

The bottleneck hits in both directions.

Consider acquiring a Samsung T5 (400 MB/s), via a Tableau T8u (325 MB/s). If the Tableau is connected via USB3 to a forensic workstation, and we are imaging to RAID volume inside, there is likely to be enough throughput to keep up with a read stream of 325 MB/s. However, if we image to a single spinning disk (200 MB/s) via a BOT based USB3 bridge (200 MB/s), the bottleneck is either the disk or the BOT based bridge. Substituting an SSD won’t give a speed improvement as we are still limited to 200 MB/s by the bridge.

How to go faster.

On the read side, choose an interconnect that mates the native storage protocol of the drive (eg. SATA or SAS or NVMe) with the PCIe bus on your imaging device. This means either using a forensic live OS or a forensic duplicator to assure write blocking and evidence integrity.

On the image storage side, find an interconnect that gives the most throughput. Use native SAS or SATA if possible, with USB3 preferenced over USB2. If using USB3, use a UASP bridge. Onwards, make sure your storage device has sufficient write throughput to match the rest of the pipeline.

There are a range of ways to do the above. My next post will focus on how the acquisition technique chosen can become the become a bottleneck.


2018 in Review

The close of 2018 provides a perfect milestone to reflect on what we accomplished and contributed in the last year. We shared novel research and tools that advanced the field of digital forensics, worked with third-party vendors to improve tool interopability, and brought game-changing forensic tools to the market.

Here’s the reasons why we are proud of the year and looking forward to 2019.

Brought a game-changing approach to in-lab forensic workflow to market

In May we announced Evimetry Lab: a game changing approach to accelerating in-lab workflow. Lab fuses our existing analyse-while-you-acquire technology with high-end networking & storage. This enables acquisition and processing to occurr at the same time with negligible slowdown. Our early adopter testing is showing evidence progressing into the analysis phase hours and days earlier, especially for multi-terabyte spinning disks.

Evimtry Lab gets processing underway as acquisition proceeds

Advanced the state of the art in evidence preservation

Third party interest in and support for the AFF4 evidence container really began to pick up. With it we attracted funding to continue our research and development and contribute back to the open source forensic stack.

Shared a refined AFF4 logical imaging implementation

A big-4 forensic team needed an open source logical imaging solution for an in-house forensic tool. We extended the Python AFF4 implementation to support logical imaging and released the code to the pyAFF4 Github Repository, for all to use.

The images produced are able to be browsed using regular ZIP64 implementations such as 7Zip (seen below).

AFF4 Logical Images are generally interopable with ZIP64

Created a new method of de-duplicated logical imaging

NIST have for years been archiving software distributions and compiling and releasing a hash set known as the National Software Reference Library (NSRL). With software-as-a-service, the game has changed, and NIST are currently looking at how to adapt their processes to Steam games.

We were funded by NIST to help solve their storage scaling issues with this work. Our solution was the addition of deduplication to AFF4 logical imaging, by adapting our earler AFF4 Hash Based Disk Imaging work to task.

We integrated the work with pyAFF4 and shared the code.

Grew the AFF4 tool ecosystem

Shifting the field to using AFF4 requires buy-in from the existing tool vendors, so making it easy to adopt the AFF4 evidence format has been a big part of our work to date. In August 2018 NUIX released v7.6, which includes our open sourced Java AFF4 read implementation. We closed out 2018 by helping Arsenal Consulting integrate AFF4 read support into the widely used Image Mounting software Arsenal Image Mounter.

Looking Ahead

We are excited by what 2019 has in store for us. We will release new features targeting in-lab efficiency and continue to refine Evimetry, making it simpler than ever to use. We look forward to sharing new research, continuing to contribute to the open source forensic stack, and supporting new vendors in adopting AFF4.