When a failing Lexar NS100 SSD stopped OPNsense from booting: CAM, TRIM, and the 31-second timeout

The firewall was down, and the disk was the reason. A client’s OPNsense system would not boot from its internal SSD, and it would not boot from alternative media while that SSD remained attached. The disk reported:

Attempt to query device size failed not ready
device not ready after 31000ms

The failed drive in the client’s system was a Lexar NS100 512 GB SATA SSD (model LNS100-512RB). Removing it allowed the system to boot from an installer. We reinstalled OPNsense on a USB 3 Gen2 drive, restored the previous configuration, reinstalled the missing plugins, and returned the firewall to service. The USB drive is an interim recovery measure; the failed SSD is no longer in use.

The business impact

An edge firewall is a single point of failure for connectivity, DNS, VPN, and remote access. When its boot disk fails, the problem is not just a disk: the whole network perimeter goes with it. In this case the configuration backup did its job, but the hardware failure still turned into an unplanned recovery project.

That is the practical lesson. A backup is only useful if you have tested the restore path and have bootable media ready when the system will not start.

Why the whole system refused to boot

OPNsense runs on FreeBSD. Disk I/O passes through FreeBSD’s Common Access Method (CAM) subsystem, which talks to ATA and SATA devices, sends commands, and enforces timeouts. During boot the kernel asks the disk for basic information, including its size. If the drive does not answer, CAM waits, retries, and eventually declares the device not ready. On this system that timeout appeared as 31000ms.

The important part is that the failure was below OPNsense. The firewall software was not corrupted; FreeBSD could no longer communicate reliably with the storage device. Because the boot process depends on that device, the system could not continue, and the firmware did not successfully fall through to another boot device while the unresponsive SSD was attached.

Where TRIM comes in

SSDs need to know which blocks are no longer in use. TRIM is the ATA command that tells the drive it can erase those blocks in the background. OPNsense’s filesystem, whether ZFS or UFS, can issue TRIM when it is enabled.

Low-cost SSD controllers vary in how robustly they handle TRIM and related ATA commands, especially around power states. In this case the evidence points to a CAM-level command timeout: the drive stopped responding, and FreeBSD could not recover it. TRIM is a plausible contributor or trigger, not a proven root cause from this incident alone. The drive itself was the failing component; the CAM and TRIM interaction explains how the failure presented itself.

This is also why the usual “reinstall the operating system” instinct would not have helped. The installer could not reliably use the drive either. Physical removal was the first recovery step.

Recovery, step by step

1. Removed the Lexar NS100. The system would not boot from other media while it was attached.
2. Booted the OPNsense VGA installer from a microSD card.
3. Used a USB 3 Gen2 drive as the installation target and a separate USB 2 drive for the configuration backup (conf/config.xml).
4. Logged in to the installer as installer using the default opnsense password.
5. Selected the previous configuration during installation and imported it.
6. Completed the installation to the USB drive and rebooted.
7. Confirmed the system was reachable with the restored configuration. All previously installed plugins were reported missing.
8. The GUI refused to install the missing plugins until a firmware update was completed. We deferred that update because a previous test on a nano image had ended in an importer error after updating.
9. Installed the missing plugins from the CLI with pkg install, covering the mail, DNS, reverse-proxy, backup, monitoring, and ACME roles the firewall had before.
10. Rebooted. The services did not all start automatically after the package installation, but came up normally after a full reboot.
11. Updated the firmware through the GUI, rebooted again, and verified that the system stayed stable.

Why the GUI blocked the plugins

Restoring an older config.xml brings back plugin settings, but not the plugin packages themselves. The web interface wanted the base system on a matching firmware level before it would install them. The CLI path bypassed that sequencing problem. After the packages were present and the firmware was updated, the system was back in a supported state.

The previous importer error is worth calling out. It is why we did not simply run the firmware update first and hope for the best. On a production firewall, recovery order matters: restore service, verify it, then make the next change.

What we recommend for any OPNsense deployment

• Test the restore, not just the backup. A config.xml on a USB stick is good. A restore you have actually performed is better.
• Keep boot media separate from the system. The microSD installer was the difference between a recovery and a much longer outage.
• Choose storage with FreeBSD in mind. Budget SATA SSDs are not all equally well behaved under CAM. Validate the exact model before putting it in a firewall.
• Monitor SMART and replace suspect drives early. A drive that disappears under load is not a drive to keep in production.
• Treat TRIM changes as a workaround. On ZFS, vfs.zfs.trim.enabled=0 can help diagnose or avoid the timeout with a problematic drive, but it is not a repair. You trade background cleanup for performance and longevity, and you still own a suspect disk.
• Do not leave USB storage as the permanent boot device. It got the firewall back online, but internal storage is the right long-term home.
• Stage firmware updates. Test on a non-production image when you can, and keep the configuration backup current before every update.

The takeaway

A firewall outage can begin with something as small as a disk that stops answering one ATA command. In this case the diagnosis was a FreeBSD CAM timeout after 31 seconds, the trigger was consistent with a low-cost SSD struggling with TRIM or a related command, and the recovery depended on two things: removing the failed drive and having a usable configuration backup.

If your firewall has never had its restore path tested, that is the next maintenance task worth doing.

Need help reviewing your firewall’s recovery plan? Talk to us or explore our IT services.