Emulex: The Cheapest 10gbe for Your Homelab

Years ago, the hunt for the cheapest 10gbe NICs resulted in buying Mellanox ConnectX-2 single-port 10gbe network cards from eBay for around $10. Nowadays those cards have increased in cost to around $20-30. While still cheap, not quite the cheapest. There are now alternatives!

Before diving into details, let’s get a something very clear. If you want the absolute simplest plug-and-play 10gbe LAN for your homelab, pay the extra for Mellanox. If you’re willing to go hands-on, do some simple manual configuration and installation, read on for my experiences with Emulex 10gbe NICs.

Emulex NICs can often be had for around $15 on eBay, sometimes even cheaper. I recently picked up a set of 4 of these cards, which came bundled with 6 SFP+ 10g-SR modules for a grand total of $47.48. Considering I can usually find SFP+ modules for about $5/ea, these alone were worth $30.

Card model numbers to look out for

• Emulex OCE10102
• Emulex OCE11102
• HP NC550SFP
• IBM 49Y7952
• IBM 96Y3766

FreeNAS 11.3

FreeNAS is the easiest OS to work with. These cards will work out of the box with the latest driver back-ported from FreeBSD 13-CURRENT, and enabled by default. Just put the card in, and configure like you would any other NIC on FreeNAS.

FreeBSD 12.1-RELEASE

FreeBSD 12 includes the latest driver, much like FreeNAS, however it is not enabled to load automatically. One small file edit is all this will take! Add the following line to your /boot/loader.conf file.

root@OPNsense:~ # echo 'if_oce_load="YES"' >> /boot/loader.conf

After that, reboot, and configure your NIC like normal.

FreeBSD 11.2-RELEASE

If all you want is basic network connectivity and VLAN support, simply follow the directions above for FreeBSD 12.1-RELEASE. However, do note that this FreeBSD releases uses an outdated driver that does not work with features such as CARP.

FreeBSD 11.3-RELEASE and FreeBSD 11.4-RELEASE

These versions of FreeBSD include a broken version of the Emulex oce(4) driver. SOME cards will work, SOME cards will cause a kernel panic initializing the driver. Please continue reading on for details on custom compiling the Emulex oce(4) driver.

If you’re already experienced at compiling and implementing FreeBSD driver changes, simply head over to this commit for the patch.

https://github.com/freebsd/freebsd/commit/b44d7086a16f737935e1eec7c7995adbc077081e

OPNsense 20.1 and HardenedBSD 11.2-RELEASE

These OSes come with the same driver included with FreeBSD 11.2-RELEASE. Out of the box, the driver has basic networking and VLAN functionality, but is also disabled by default, and also lacks support for advanced networking features like CARP. If you don’t need the more advanced networking features, simply follow the instructions above for FreeBSD 12.1-RELEASE to enable the Emulex oce(4) driver at boot.

If you want the quick and easy solution, I’ve provided the pre-compiled binary for the patched driver at the following URL. https://vincerants.com/public/opnsense/if_oce.ko

root@OPNsense:~# cd /boot/kernel
root@OPNsense:/boot/kernel# fetch https://vincerants.com/public/opnsense/if_oce.ko
if_oce.ko                                     100% of  128 kB 9307 kBps 00m00s

Custom Compiling the Emulex oce(4) Driver – Prepping the /usr/src Tree

Because my intended use case for these NICs are a series of OPNsense routers, this part of the guide is going to be custom tailored for building the driver natively on this OS.

First, install nano for easier text file editing. Next, enable the FreeBSD package repository. Use this repository to install git so that way we may clone the source tree.

root@OPNsense:~ # pkg install -y nano
Updating OPNsense repository catalogue...
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01
Fetching packagesite.txz: 100%  183 KiB 187.0kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 708 packages processed.
All repositories are up to date.
Updating database digests format: 100%
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.12.0 -> 1.12.0_1

Number of packages to be upgraded: 1

3 MiB to be downloaded.
[1/1] Fetching pkg-1.12.0_1.txz: 100%    3 MiB 903.4kB/s    00:04
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.12.0 to 1.12.0_1...
[1/1] Extracting pkg-1.12.0_1: 100%
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        nano: 4.9.2

Number of packages to be installed: 1

276 KiB to be downloaded.
[1/1] Fetching nano-4.9.2.txz: 100%  276 KiB 282.6kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing nano-4.9.2...
[1/1] Extracting nano-4.9.2: 100%

root@OPNsense:~ # echo 'FreeBSD: { enabled: yes }' > /usr/local/etc/pkg/repos/FreeBSD.conf

root@OPNsense:/usr/src # pkg install -y git
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    916 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   6.5MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 31592 packages processed.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.12.0_1 -> 1.13.2 [FreeBSD]

Number of packages to be upgraded: 1

3 MiB to be downloaded.
[1/1] Fetching pkg-1.13.2.txz: 100%    3 MiB   3.5MB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.12.0_1 to 1.13.2...
[1/1] Extracting pkg-1.13.2: 100%
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following 10 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        cvsps: 2.1_2 [OPNsense]
        fontconfig: 2.13.92_2,1 [FreeBSD]
        freetype2: 2.10.1 [FreeBSD]
        git: 2.26.2 [OPNsense]
        libfontenc: 1.1.4 [FreeBSD]
        libunistring: 0.9.10_1 [FreeBSD]
        libunwind: 20170615 [FreeBSD]
        p5-Error: 0.17029 [OPNsense]
        pcre2: 10.35 [OPNsense]
        png: 1.6.37 [FreeBSD]

Number of packages to be installed: 10

The process will require 50 MiB more space.
9 MiB to be downloaded.
[1/10] Fetching git-2.26.2.txz: 100%    5 MiB   1.3MB/s    00:04
[2/10] Fetching libfontenc-1.1.4.txz: 100%   20 KiB  20.8kB/s    00:01
[3/10] Fetching freetype2-2.10.1.txz: 100%    1 MiB   1.4MB/s    00:01
[4/10] Fetching fontconfig-2.13.92_2,1.txz: 100%  382 KiB 390.9kB/s    00:01
[5/10] Fetching png-1.6.37.txz: 100%  288 KiB 294.8kB/s    00:01
[6/10] Fetching libunwind-20170615.txz: 100%  118 KiB 121.2kB/s    00:01
[7/10] Fetching libunistring-0.9.10_1.txz: 100%  522 KiB 535.0kB/s    00:01
[8/10] Fetching pcre2-10.35.txz: 100%  879 KiB 449.8kB/s    00:02
[9/10] Fetching p5-Error-0.17029.txz: 100%   27 KiB  27.4kB/s    00:01
[10/10] Fetching cvsps-2.1_2.txz: 100%   41 KiB  41.5kB/s    00:01
Checking integrity... done (0 conflicting)
[1/10] Installing freetype2-2.10.1...
[1/10] Extracting freetype2-2.10.1: 100%
[2/10] Installing p5-Error-0.17029...
[2/10] Extracting p5-Error-0.17029: 100%
[3/10] Installing cvsps-2.1_2...
[3/10] Extracting cvsps-2.1_2: 100%
[4/10] Installing git-2.26.2...
===> Creating groups.
Creating group 'git_daemon' with gid '964'.
===> Creating users
Creating user 'git_daemon' with uid '964'.
[4/10] Extracting git-2.26.2: 100%
[5/10] Installing libfontenc-1.1.4...
[5/10] Extracting libfontenc-1.1.4: 100%
[6/10] Installing fontconfig-2.13.92_2,1...
[6/10] Extracting fontconfig-2.13.92_2,1: 100%
Running fc-cache to build fontconfig cache...
Font directories:
        /usr/local/share/fonts
        /usr/local/lib/X11/fonts
/usr/local/share/fonts: skipping, no such directory
/usr/local/lib/X11/fonts: skipping, no such directory
/var/db/fontconfig: cleaning cache directory
fc-cache: succeeded
[7/10] Installing png-1.6.37...
[7/10] Extracting png-1.6.37: 100%
[8/10] Installing libunwind-20170615...
[8/10] Extracting libunwind-20170615: 100%
[9/10] Installing libunistring-0.9.10_1...
[9/10] Extracting libunistring-0.9.10_1: 100%
[10/10] Installing pcre2-10.35...
[10/10] Extracting pcre2-10.35: 100%
=====
Message from freetype2-2.10.1:

--
The 2.7.x series now uses the new subpixel hinting mode (V40 port's option) as
the default, emulating a modern version of ClearType. This change inevitably
leads to different rendering results, and you might change port's options to
adapt it to your taste (or use the new "FREETYPE_PROPERTIES" environment
variable).

The environment variable "FREETYPE_PROPERTIES" can be used to control the
driver properties. Example:

FREETYPE_PROPERTIES=truetype:interpreter-version=35 \
        cff:no-stem-darkening=1 \
        autofitter:warping=1

This allows to select, say, the subpixel hinting mode at runtime for a given
application.

If LONG_PCF_NAMES port's option was enabled, the PCF family names may include
the foundry and information whether they contain wide characters. For example,
"Sony Fixed" or "Misc Fixed Wide", instead of "Fixed". This can be disabled at
run time with using pcf:no-long-family-names property, if needed. Example:

FREETYPE_PROPERTIES=pcf:no-long-family-names=1

How to recreate fontconfig cache with using such environment variable,
if needed:
# env FREETYPE_PROPERTIES=pcf:no-long-family-names=1 fc-cache -fsv

The controllable properties are listed in the section "Controlling FreeType
Modules" in the reference's table of contents
(/usr/local/share/doc/freetype2/reference/site/index.html, if documentation was installed).
=====
Message from git-2.26.2:

--
If you installed the GITWEB option please follow these instructions:

In the directory /usr/local/share/examples/git/gitweb you can find all files to
make gitweb work as a public repository on the web.

All you have to do to make gitweb work is:
1) Please be sure you're able to execute CGI scripts in
   /usr/local/share/examples/git/gitweb.
2) Set the GITWEB_CONFIG variable in your webserver's config to
   /usr/local/etc/git/gitweb.conf. This variable is passed to gitweb.cgi.
3) Restart server.


If you installed the CONTRIB option please note that the scripts are
installed in /usr/local/share/git-core/contrib. Some of them require
other ports to be installed (perl, python, etc), which you may need to
install manually.

root@OPNsense:/usr/src # git clone https://github.com/HardenedBSD/hardenedBSD.git -b hardened/11.2-releng/master --single-branch --depth=1 /usr/src
Cloning into '/usr/src'...
remote: Enumerating objects: 77339, done.
remote: Counting objects: 100% (77339/77339), done.
remote: Compressing objects: 100% (68364/68364), done.
remote: Total 77339 (delta 16296), reused 29974 (delta 6113), pack-reused 0
Receiving objects: 100% (77339/77339), 252.74 MiB | 9.96 MiB/s, done.
Resolving deltas: 100% (16296/16296), done.
Updating files: 100% (74050/74050), done.

Custom Compiling the Emulex oce(4) Driver – Updating the Driver Source Code

Now that we have the HardenedBSD 11.2-RELEASE source code, we need to update the driver to a more recent version, and patch it to ensure the kernel will not segfault on boot.

root@OPNsense:~ # cd /usr/src/sys/net/
root@OPNsense:/usr/src/sys/net # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/net/ethernet.h
ethernet.h                                    100% of   21 kB   34 MBps 00m00s

root@OPNsense:~ # cd /usr/src/sys/dev/oce/

root@OPNsense:/usr/src/sys/dev/oce # ll
total 340
-rw-r--r--  1 root  wheel  14018 Jun 21 20:36 oce_hw.c
-rw-r--r--  1 root  wheel  86688 Jun 21 20:36 oce_hw.h
-rw-r--r--  1 root  wheel  53329 Jun 21 20:36 oce_if.c
-rw-r--r--  1 root  wheel  33583 Jun 21 20:36 oce_if.h
-rw-r--r--  1 root  wheel  59884 Jun 21 20:36 oce_mbox.c
-rw-r--r--  1 root  wheel  26826 Jun 21 20:36 oce_queue.c
-rw-r--r--  1 root  wheel  49140 Jun 21 20:36 oce_sysctl.c
-rw-r--r--  1 root  wheel   6182 Jun 21 20:36 oce_util.c

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_hw.c
oce_hw.c                                      100% of   13 kB   37 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_hw.h
oce_hw.h                                      100% of   97 kB   26 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_if.c
oce_if.c                                      100% of   73 kB   37 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_if.h
oce_if.h                                      100% of   36 kB   48 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_mbox.c
oce_mbox.c                                    100% of   64 kB   40 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_queue.c
oce_queue.c                                   100% of   33 kB   45 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_sysctl.c
oce_sysctl.c                                  100% of   52 kB   49 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_user.h
oce_user.h                                    100% of 3497  B   16 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_util.c
oce_util.c                                    100% of 6182  B   23 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://github.com/freebsd/freebsd/commit/b44d7086a16f737935e1eec7c7995adbc077081e.patch
b44d7086a16f737935e1eec7c7995adbc077081e.patch 100% of 1603  B 9043 kBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # ll
total 408
-rw-r--r--  1 root  wheel   1603 Jun 21 20:51 b44d7086a16f737935e1eec7c7995adbc077081e.patch
-rw-r--r--  1 root  wheel  14174 Jun 21 20:44 oce_hw.c
-rw-r--r--  1 root  wheel  99367 Jun 21 20:44 oce_hw.h
-rw-r--r--  1 root  wheel  75369 Jun 21 20:44 oce_if.c
-rw-r--r--  1 root  wheel  37330 Jun 21 20:44 oce_if.h
-rw-r--r--  1 root  wheel  65545 Jun 21 20:44 oce_mbox.c
-rw-r--r--  1 root  wheel  33799 Jun 21 20:44 oce_queue.c
-rw-r--r--  1 root  wheel  53522 Jun 21 20:44 oce_sysctl.c
-rw-r--r--  1 root  wheel   3497 Jun 21 20:48 oce_user.h
-rw-r--r--  1 root  wheel   6182 Jun 21 20:44 oce_util.c

Custom Compiling the Emulex oce(4) Driver – Applying the Patch and Compiling

With all of the files downloaded, it is time to apply the patch and compile our new driver.

root@OPNsense:/usr/src/sys/dev/oce # patch oce_if.c b44d7086a16f737935e1eec7c7995adbc077081e.patch
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From b44d7086a16f737935e1eec7c7995adbc077081e Mon Sep 17 00:00:00 2001
|From: mav <mav@FreeBSD.org>
|Date: Tue, 28 May 2019 18:32:04 +0000
|Subject: [PATCH] Fix array out of bound panic introduced in r306219.
|
|As I see, different NICs in different configurations may have different
|numbers of TX and RX queues.  The code was assuming 1:1 mapping between
|event queues (interrupts) and TX/RX queues.  Since number of interrupts
|is set to maximum of TX and RX queues, when those two are different, the
|system is doomed.
|
|I have no documentation or deep knowledge about this hardware, so this
|change is based on general observations and code reading.  If some of my
|guesses are wrong, please do better.  I just confirmed HP NC550SFP NICs
|are working now.
|
|MFC after:     2 weeks
|Sponsored by:  iXsystems, Inc.
|---
| sys/dev/oce/oce_if.c | 18 ++++++++++++++----
| 1 file changed, 14 insertions(+), 4 deletions(-)
|
|diff --git a/sys/dev/oce/oce_if.c b/sys/dev/oce/oce_if.c
|index 221a871a7ceb5..44fb04fef791e 100644
|--- a/sys/dev/oce/oce_if.c
|+++ b/sys/dev/oce/oce_if.c
--------------------------
Patching file oce_if.c using Plan A...
Hunk #1 succeeded at 2394.
done

root@OPNsense:/ # cd /usr/src/sys/modules/oce

root@OPNsense:/usr/src/sys/modules/oce # make
machine -> /usr/src/sys/amd64/include
x86 -> /usr/src/sys/x86/include
awk -f /usr/src/sys/tools/makeobjops.awk /usr/src/sys/kern/bus_if.m -h
awk -f /usr/src/sys/tools/makeobjops.awk /usr/src/sys/kern/device_if.m -h
awk -f /usr/src/sys/tools/makeobjops.awk /usr/src/sys/dev/pci/pci_if.m -h
Warning: Object directory not changed from original /usr/src/sys/modules/oce
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_if.o -MToce_if.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_if.c -o oce_if.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_hw.o -MToce_hw.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_hw.c -o oce_hw.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_mbox.o -MToce_mbox.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_mbox.c -o oce_mbox.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_util.o -MToce_util.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_util.c -o oce_util.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_queue.o -MToce_queue.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_queue.c -o oce_queue.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_sysctl.o -MToce_sysctl.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_sysctl.c -o oce_sysctl.o
ld -d -warn-common -r -d -o if_oce.ko oce_if.o oce_hw.o oce_mbox.o oce_util.o oce_queue.o oce_sysctl.o
:> export_syms
awk -f /usr/src/sys/conf/kmod_syms.awk if_oce.ko  export_syms | xargs -J% objcopy % if_oce.ko
objcopy --strip-debug if_oce.ko

root@OPNsense:/usr/src/sys/modules/oce # ll
total 392
-rw-r--r--  1 root  wheel    4039 Jun 21 20:56 .depend.oce_hw.o
-rw-r--r--  1 root  wheel    4098 Jun 21 20:56 .depend.oce_if.o
-rw-r--r--  1 root  wheel    4043 Jun 21 20:56 .depend.oce_mbox.o
-rw-r--r--  1 root  wheel    4049 Jun 21 20:56 .depend.oce_queue.o
-rw-r--r--  1 root  wheel    4051 Jun 21 20:56 .depend.oce_sysctl.o
-rw-r--r--  1 root  wheel    4043 Jun 21 20:56 .depend.oce_util.o
-rw-r--r--  1 root  wheel     335 Jun 21 20:36 Makefile
-rw-r--r--  1 root  wheel   40423 Jun 21 20:56 bus_if.h
-rw-r--r--  1 root  wheel   11902 Jun 21 20:56 device_if.h
-rw-r--r--  1 root  wheel       0 Jun 21 20:56 export_syms
-rw-r--r--  1 root  wheel  131288 Jun 21 20:56 if_oce.ko
lrwxr-xr-x  1 root  wheel      26 Jun 21 20:56 machine@ -> /usr/src/sys/amd64/include
-rw-r--r--  1 root  wheel    9432 Jun 21 20:56 oce_hw.o
-rw-r--r--  1 root  wheel   48512 Jun 21 20:56 oce_if.o
-rw-r--r--  1 root  wheel   23912 Jun 21 20:56 oce_mbox.o
-rw-r--r--  1 root  wheel   19280 Jun 21 20:56 oce_queue.o
-rw-r--r--  1 root  wheel   37160 Jun 21 20:56 oce_sysctl.o
-rw-r--r--  1 root  wheel    3128 Jun 21 20:56 oce_util.o
-rw-r--r--  1 root  wheel      37 Jun 21 20:56 opt_inet.h
-rw-r--r--  1 root  wheel      16 Jun 21 20:56 opt_inet6.h
-rw-r--r--  1 root  wheel   18641 Jun 21 20:56 pci_if.h
lrwxr-xr-x  1 root  wheel      24 Jun 21 20:56 x86@ -> /usr/src/sys/x86/include

Custom Compiling the Emulex oce(4) Driver – Installing the Driver

With the driver built the only thing left to do is install it on the system. Copy the driver to the /boot/kernel/ folder and enable the driver in /boot/loader.conf

root@OPNsense:/usr/src/sys/modules/oce # cp if_oce.ko /boot/kernel/

root@OPNsense:/usr/src/sys/modules/oce # ll /boot/kernel/ | grep oce
-r-xr-xr-x  1 root  wheel    131288 Jun 22 04:21 if_oce.ko*

root@OPNsense:~ # echo 'if_oce_load="YES"' >> /boot/loader.conf