How to notify unused blocks to the flash storage controller over the USB interface

Research the usability of the unused blocks notification operation against flash storage controllers over the USB interface.

https://gitlab.com/brlin/trim-over-usb-howto

Table of contents

[TOC]

Problem

A few important characteristics of flash memory storage devices is that:

• Once a memory cell has data written on it(bit flips from 0→1), it can only be rewritten after it is erased(1→0).
• Memory cells can only be erased in batch(called erase segments), erase segments are usually bigger than memory blocks(e.g. 4MiB).
• When the count of erased memory cells are low the memory controller will need to do the erase on-the-fly, which results in a very poor write performance.

Some flash storage drives supports notifying unused memory blocks to the controller from the Operating System to do garbage collection(GC) using the following protocol commands:

• ATA: TRIM
• NVM Express(NVMe): DEALLOCATE
• SCSI: UNMAP
• SD/MMC: ERASE/DISCARD

in order to let the storage controller do the garbage collection in the background to maintain write performance. However, the operation may no be available if:

• It is not implemented (common in lower-end flash storage drives)

• There's a translation layer between the storage controller and the operating system, like a USB external hard-drive enclosure/cable that translates the SATA/NVMe/MMC... protocol commands into the counterpart SCSI commands which the USB mass storage/UASP specification uses.

  Another issue is that even if the translation layer support such operation, the Operating System may not utilize it due to the storage drive not complying to the relevant specifications.

Solution

We'll need to know:

• How to check whether the storage drive supports such operation.
• If the storage drive supports such operation but the Operating System didn't enables it due to uncompliance, how to override it.
• If the storage drive claimed to support such operation, how to check whether it works or not.

The answer of these questions varies based on the following factors:

• The manufacturer and model of the storage drive/enclosure
• The firmware version of the storage drive/enclosure

We'll individually collect related info of each device to avoid bias.

Research process

This section documents the process of determining the support:

Determine the kernel name of the storage device

Run the following commands in a text terminal to determine the kernel name of the storage device:

lsblk_opts=(
    # Don't print sub-device nodes
    --nodeps

    # Exclude loopback devices(e.g. snaps)
    --exclude 7

    # Specify output columns that are useful in determining the kernel
    # name of the storage device block device
    --output NAME,VENDOR,MODEL,SERIAL,SIZE
)
lsblk "${lsblk_opts[@]}"

Determine whether the unused block notification feature is enabled by the operating system by default

Run the following commands in a text terminal to determine the kernel name of the storage device:

device_kernel_name=_kernel_name_
device="/dev/${device_kernel_name}"
lsblk_opts=(
    # Don't print sub-device nodes
    --nodeps

    # Print information about the unused block notification capabilities
    # for each device.
    --discard
)
lsblk "${lsblk_opts[@]}" "${device}"

Check whether the storage device has declared implementation of the Logical Block Provisioning Management feature specified by SBC-4

Run the following command as root in a text terminal to query the response of the SCSI READ CAPACITY (16) command of the storage device:

device_kernel_name=_kernel_name_
device="/dev/${device_kernel_name}"
sg_readcap_opts=(
    # Use the 16 byte cdb variant of the READ CAPACITY command, allow
    # proper results for storage drives capacity over 2TiB
    # (2**32 - 2) * 512 / 1024 / 1024 / 1024
    --long
)
sg_readcap "${sg_readcap_opts[@]}" "${device}"

Check the SCSI vital product data(VPD) pages supported by the SCSI device

Run the following command as root in a text terminal to determine which SCSI vital product data(VPD) pages supported by the SCSI device:

device_kernel_name=_kernel_name_
device="/dev/${device_kernel_name}"
sg_vpd "${device}"

Check the supported parameters of the SCSI UNMAP command of the storage device

Run the following command as root in a text terminal to query the Block limits VPD page of the storage device:

device_kernel_name=_kernel_name_
device="/dev/${device_kernel_name}"
sg_vpd_opts=(
    --page=bl
)
sg_vpd "${sg_vpd_opts[@]}" "${device}"

Enable unused block notification support by force

Although the missing declaration of the SCSI Logical Block Provisioning Management feature, there's still possibility that the drive actually implemented the unused block notification functionality in the firmware. Let's try to find out.

Warning: There's a reason why the functionality is not enabled by default as there's a chance that the controller have a problomatic reaction when facing the UNMAP SCSI command, which may results in problems including but not limited to:

• The drive simply fails and no longer functions properly, rendering it no longer usable for data storage/access.
• The drive erratically respond to the command and erases memory blocks that are not requested, leads to data loss.

Only continue if you can take the responsibility of device failure/data recovery.

In order to force enable the unused block notification feature, we need to first determine the address of the SCSI device of the storage device , which can be queried by running the following command in a text terminal:

lsscsi

We can force enable the unused block notification support feature using the UNMAP SCSI command by running the following commands as root after setting the proper scsi_device_address:

scsi_device_address=_address_
echo unmap > "/sys/class/scsi_disk/${scsi_device_address}/provisioning_mode"

Now that the UNMAP command is enabled, run the following commands to check whether the data size of each UNMAP SCSI command set by the kernel is sane:

device_kernel_name=_kernel_name_
cat "/sys/block/${device_kernel_name}/queue/discard_max_bytes"

Test whether unused block notification actually work

Now we can test whether the unused block notification really work by triggering a whole drive/partition block device discard operation by running the following commands as root:

device_kernel_name=_kernel_name_
device="/dev/${device_kernel_name}"
blkdiscard_opts=(
    # Disable safeguard checks
    --force

    # Print details of the operation
    --verbose
)
blkdiscard "${blkdiscard_opts[@]}" "${device}"

Research results

We currently have research results for the following products:

• 伽利略 M2NVU31 M.2(NVMe) PCI-E SSD to USB3.1 Gen2
• OWC USB-C Travel Dock E(SD card reader)
• Transcend JetFlash 790 Series 64GB

References

The following material are referenced during the development of this project:

• Discard over USB - Gentoo Wiki
  Explains how to enable discard/trim operation for block devices via the USB bus.
• External SSD with TRIM support - Solid state drive - ArchWiki
  Explains how to check and enable TRIM operation on solid storage drives over a USB-to-SATA bridge chips.
• Hardware support - Trim (computing) - Wikipedia
  Explains the equivalent commands of unused block notification operation for each protocols.
• sd: disable logical block provisioning if 'lbpme' is not set - Patchwork
  Explains the reasoning that Linux Kernel requires the LBPME bit in order to enable unused memory blocks notification support.
• SCSI Commands Reference Manual 100293068, Rev. J
  Documents the Serial Attached SCSI(SAS) commands in the SCSI specifications, sections that may include useful information are:
  • 5.4 Vital product data parameters
• Enabling TRIM on an external SSD on a Raspberry Pi | Jeff Geerling
• Trying to get SSD boot working on pi4 - Page 2 - Raspberry Pi Forums
• The lsblk(8) manual page
  Explains how to use the --nodeps and --output command-line options.
• The sg_vpd(8) manual page
  Explains how to use the --long sg_vpd command-line option.
• ‎Gemini - LBP in SCSI
  Introduces what Logical Block Provisioning Management (LBP) in SCSI do.
• SCSI device location codes - IBM Documentation
  Explains the format of the SCSI device location code.
• SCSI Addressing
  Explains the format of the SCSI device addresses.
• Queue sysfs files | Info on the Block I/O (BIO) layer | Linux kernel plaintext documentation
  Explains the definition of the discard_max_bytes sysfs file.