open-vm-tools on FreeBSD under VMware ESXi ARM Fling

vince4 Comments

Over the past week or so, I’ve been working with the VMware team to solve compatibility issues with open-vm-tools running under ARM. As of right now, I’m excited to report that initial tests are showing that open-vm-tools is now working great! However, these changes have not been pushed upstream yet either to the FreeBSD Ports collection or into the upstream open-vm-tools repository. There are active requests for this that are under review, but if you’d like to get up and running today, please follow along.

There are two different options to get open-vm-tools on FreeBSD for ARM right now. I’ve provided pre-compiled packages for a few key FreeBSD versions. Alternatively, you can compile them yourself from the custom ports that I’ve provided.

Pre-compiled Packages

Download the appropriate package for your FreeBSD version. Both open-vm-tools and open-vm-tools-nox11. The former is if you’re intending to have a full X11 graphical desktop, the latter for command line only systems such as servers.

Use one of the following commands for your version of FreeBSD.

# open-vm-tools-nox11 dependencies
pkg install -y libmspack libdnet fusefs-libs gettext-runtime glib


# FreeBSD 12.1-RELEASE (desktop)
pkg add https://vincerants.com/public/freebsd-arm/12.1-RELEASE/emulators/open-vm-tools-11.1.5.3%2C2.txz

# FreeBSD 12.1-RELEASE (no-X11)
pkg add https://vincerants.com/public/freebsd-arm/12.1-RELEASE/emulators/open-vm-tools-nox11-11.1.5.3%2C2.txz


#FreBSD 12.2-RC3 (desktop)
pkg add https://vincerants.com/public/freebsd-arm/12.2-RC3/emulators/open-vm-tools-11.1.5.3%2C2.txz

#FreBSD 12.2-RC3 (no-X11)
pkg add https://vincerants.com/public/freebsd-arm/12.2-RC3/emulators/open-vm-tools-nox11-11.1.5.3%2C2.txz


# FreeBSD 13.0-CURRENT 20201015 (desktop)
pkg add https://vincerants.com/public/freebsd-arm/13.0-20201015/emulators/open-vm-tools-11.1.5.3%2C2.txz

# FreeBSD 13.0-CURRENT 20201015 (no-X11)
pkg add https://vincerants.com/public/freebsd-arm/13.0-20201015/emulators/open-vm-tools-nox11-11.1.5.3%2C2.txz

Manually Compiling open-vm-tools

I’ve created a supplementary ports collection which has the needed patches to get open-vm-tools running on FreeBSD under VMware ESXi ARM Fling. Before running the following commands, ensure you have the latest version of pkg built and installed. The version bundled with FreeBSD is out of date.

# Update the version of pkg installed
cd /usr/ports/ports-mgmt/pkg
make
make reinstall

# Build and install open-vm-tools
mkdir /code
cd /code
git clone https://github.com/darkain/ports.git
cd /code/ports/emulators/open-vm-tools
make
make install

Enabling open-vm-tools

After installing open-vm-tools, the associated services need to be enabled. Add the following to your /etc/rc.conf file and reboot your virtual machine.

vmware_guest_vmblock_enable="YES"
vmware_guest_vmhgfs_enable="YES"
vmware_guest_vmmemctl_enable="YES"
vmware_guest_vmxnet_enable="YES"
vmware_guestd_enable="YES"

What Code Changes Were Needed

https://github.com/vmware/open-vm-tools/pull/474

This pull request documents the files that have changed. The custom ports collection listed above simply references these changes. Please feel free to review these changes and comment on the github pull request if you have more to add to make FreeBSD ARM support better!

FreeBSD on ESXi ARM Fling: Fixing Virtual Hardware

vince2 Comments

With the current state of FreeBSD on ARM in general, a number of hardware drivers are either set to not auto-load on boot, or are entirely missing altogether. This page is to document my findings with various bits of hardware, and if possible, list fixes.

USB 2.0 Controller (uhci)

https://www.freebsd.org/cgi/man.cgi?query=uhci&sektion=4

UPDATE 2020-10-22: Today’s FreeBSD snapshot ISO image now includes the uhci driver, meaning you no longer need to worry about which USB controller is selected in VMware (you still need at least one though, as the virtual keyboard is USB based)

If you switch the USB Controller for the FreeBSD virtual machine from the 3.1 controller to the 2.0 controller, you’ll lose all USB support. This also means that virtual USB devices like keyboard and mouse will stop functioning as well. To get the virtual USB 2.0 controller to function, add the following to your /boot/loader.conf file and reboot your FreeBSD VM.

uhci_load="YES"

NOTE: If you create the virtual machine through a remote connection on VMware Workstation, it will default to using the USB 2.0 controller rather than the USB 3.1 controller. Because of this, FreeBSD without the uhci driver will not have a functional keyboard. Change the controller type from USB 2.0 to USB 3.1 and reboot the virtual machine to get keyboard functionality again.

USB Virtual Mouse (ums)

https://www.freebsd.org/cgi/man.cgi?query=ums&sektion=4

The vmmouse driver, or virtual machine mouse driver, will not work by default due to missing the USB Mouse driver. To load the USB Mouse driver, add the following to your /boot/loader.conf file and reboot your FreeBSD VM.

ums_load="YES"

Virtual Network Card (vmxnet)

https://www.freebsd.org/cgi/man.cgi?query=vmx&sektion=4

UPDATE 2020-10-22: Today’s FreeBSD snapshot ISO image now includes the vmxnet driver pre-compiled. However, it is not yet loaded by default. You’ll still need to update your /boot/loader.conf

The precompiled vmxnet driver is not included with FreeBSD on ARM, however compiling it manually works without any modification to source code to make it work on ARM. To do this, you must ensure you either install the system source tree when installing FreeBSD, or download the source matching your FreeBSD version after install.

Next, compile and install the vmxnet driver.

cd /usr/src/sys/modules/vmware/vmxnet3
make
make install

After building and installing the vmxnet driver, add the following to your /boot/loader.conf file and reboot your FreeBSD VM to load the driver.

if_vmx_load="YES"

Alternatively, you can download the pre-compiled kernel module for any of the following FreeBSD versions. Place the if_vmx.ko driver file in /boot/modules/ and then follow the instructions above to for modifying /boot/loader.conf to load the driver at boot time.

Paravirtual SCSI Controller (pvscsi)

UPDATE 2020-10-22: Today’s FreeBSD snapshot ISO image now includes the pvscsi driver pre-compiled. However, it is not yet loaded by default. You’ll still need to update your /boot/loader.conf

The pvscsi driver is new to FreeBSD 13.0-CURRENT, so new in fact it doesn’t have a man page to link to yet. The driver, however, compiles and runs fine on ARM64, however I’ve yet to get it to work as the boot device controller. Currently the ARM64 UEFI inside of the virtual machine doesn’t appear to query the pvscsi controller on boot for potential boot devices.

The instructions are virtually the same as for the vmxnet driver above. Ensure that you have the system source tree option enabled when installing FreeBSD, and then do the following.

cd /usr/src/sys/modules/vmware/pvscsi
make
make install

After building and installing the pvscsi driver, add the following to your /boot/loader.conf file and reboot your FreeBSD VM to load the driver.

pvscsi_load="YES"

Alternatively, you can download the pre-compiled kernel module for any of the following FreeBSD versions. Place the if_vmx.ko driver file in /boot/modules/ and then follow the instructions above to for modifying /boot/loader.conf to load the driver at boot time.

Virtual Machine Communication Interface (vmci)

This driver currently contains i386/AMD64 specific assembly macros that I’ve yet to convert to ARM/aarch64. I also don’t have any way to test working vmci afterwards yet. So for the time being, this driver isn’t working.

CD-ROM (cd)

https://www.freebsd.org/cgi/man.cgi?query=cd&sektion=4

UPDATE 2020-10-16: Yesterday’s snapshot ISO now contains the cd-rom driver. With this, the ISO now boots and installs perfectly as expected. You can get the 2020-10-15 or newer ISO from the following URL. This driver, however, is missing from 12.2-RELEASE. It should be included in future 12-SNAPSHOT releases though!

CD-ROM support was added to the FreeBSD ARM64 GENERIC kernel a few days ago, however, that was after the most recent FreeBSD 13.0-CURRENT snapshot as of this blog post. The next snapshot should have CD-ROM support, meaning the next snapshot’s ISO installer should be bootable under ESXi ARM Fling.

https://download.freebsd.org/ftp/snapshots/arm64/aarch64/ISO-IMAGES/13.0/

FreeBSD under VMware ESXi on Arm Fling

vinceLeave a comment

UPDATE 2020-10-17

As of the FreeBSD 13.0-CURRENT snapshot from 2020-10-15, the installation ISO is now bootable and fully works to install FreeBSD on ARM just as you would do under VMware on i386 or AMD64.

Download the latest disk1.iso image from the following URL, load it into the virtual CD-ROM drive, and boot your new VM like normal. It should boot off of the ISO image and enter the FreeBSD installer as expected.

https://download.freebsd.org/ftp/snapshots/arm64/aarch64/ISO-IMAGES/13.0/

Additional information on getting individual drivers to work under FreeBSD on ARM can be found at the following link.

https://vincerants.com/freebsd-on-esxi-arm-fling-fixing-virtual-hardware/

Earlier this week, VMware released ESXi on Arm Fling, their hypervisor for the ARM platform. Here are instructions to get a FreeBSD virtual machine up and running under VMWare ESXi on Arm Fling.

These instruction assume you’ve followed VMware’s documentation on setting up the hypervisor on your ARM platform, and are familiar with the basics of how ESXi/vSphere functions.

For my hardware configuration, I’m using a Raspberry Pi 4 (8GB) with whatever old MicroSD card I could find for the UEFI firmware, and an equally old USB drive to install the hypervisor on to. The ESXi install is less than 200MiB. For actual VM storage, I am using an existing x86-64 iSCSI VMFS from my NAS.

These instructions follow the process of downloading the official ARM VMDK files from FreeBSD. I’ve as of yet to get the installer ISO to boot properly, so for the time being, this is how we’ll work. (see update above)

Both FreeBSD 12.1-RELEASE and 13.0-CURRENT will work for this process, though I personally recommend 13.0 as it has a more complete collection of pre-compiled 3rd part ports software.

Downloading and preparing the VMDK

Download the compressed VMDK file from one of the following URLs, and then extract it locally.

https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/aarch64/Latest/

https://download.freebsd.org/ftp/snapshots/VM-IMAGES/13.0-CURRENT/aarch64/Latest/

Use the Datastore browser to create a location to store the FreeBSD VMDK file.
Click on the Upload button and select the extracted VMDK file from your system to begin the upload process.
Take note of the upload progress in the top-right of the window. Due to the size of the VMDK file, it may take a few minutes to upload.
The VMDK file provided by FreeBSD is designed for VMware Workstation. We’ll need to convert it for use on ESXi. This is also a good time to expand the size of the VMDK, so we have room to install additional applications.
  1. Navigate to the folder where the VMDK file was uploaded. In my case, I have it on my tank volume inside of the arm-bsd-base-13 folder.
    • cd /vmfs/volumes/tank/arm-bsd-base-13
  2. Convert the VMDK file from Workstation to ESXi (optionally making it thin provisioned)
    • vmkfstools -i FreeBSD-13.0-CURRENT-arm64-aarch64.vmdk -d thin arm-freebsd.vmdk
  3. Expand the size of the newly created VMDK file (in my case, I’m making it 50GiB)
    • vmkfstools -X 50g arm-freebsd.vmdk

Creating a FreeBSD ARM virtual machine

Now we can start creating our new virtual machine like normal.
Enter the name for your new virtual machine. Select Other as the Guset OS family and then FreeBSD 12 or later version (64-bit) as the Guest OS version.
Select where you want to store this new virtual machine.
NOTE: As of this writing, anything higher than 1 for the CPU setting will prevent FreeBSD from booting.
Remove Hard disk 1, as we’ll be using our custom VMDK instead.
Select Add hard disk and then Existing hard disk.
Navigate the file system, and select our recently created VMDK file. Don’t worry about the reported disk size on this screen, as that doesn’t reflect the expanded size of the disk here.
Our virtual machine configuration is complete, we’re ready to click Finish!
Our virtual machine is now ready. We can interact with it in just about any way that you would expect from a standard ESXi install.
For the sake of safety, this would be a good time to take a virtual machine snapshot. This can be done by selection the Action menu, highlighting Snapshots and then select Take snapshot.
Enter a name and description for this snapshot. I chose “pre-boot” to let me know that this snapshot was taken before the first time I booted the guest FreeBSD operating system.
We can now power on our FreeBSD virtual machine. During the first boot process, FreeBSD will autodetect our new VMDK size and expand the UFS file system automatically.
Once fully booted, simply log in as the root user, no password needed.
NOTE: it would be wise to add a password at this point.

kom-pew-pew-pew-t0r

vinceLeave a comment

Welcome to the kom-pew-pew-pew-t0r!

This is the build log for this super awesome amazing magical PC case mod!

My motivations for this were simple. PDXLAN Summer 2020 @ Home is having a case mod contest, I really freaggin love memes, and I wanted to see how crazy I could go with only spare parts on hand. There was zero purchases for this build, and zero asking others for any parts. All of these came from things I had laying around my apartment.

Specs

  • Case: Amazon Cardboard Box
  • CPU: AMD Athlon II X4 640 (4-core, 4-thread, 3.0GHz)
  • Motherboard: M4A88T-V EVO/USB3
  • RAM: 4x4GB G.Skill RipJaws DDR3 in Dual-Channel configuration
  • Storage: 64GB Kingston SATA SSD
  • Storage: USB 1.44MB Floppy
  • Sound: Creative Labs SoundBlaster Audigy (on 1x PCIe riser)
  • Sound: Onboard AC97
  • Network: Dual-Port SFP+ 10gbe Emulex OCE10102 (on 1x PCIe riser)
  • Network: Onboard RJ45 1gbe RealTek
  • Network: USB WiFi N
  • Network: USB BlueTooth 2.0
  • GPU: Nvidia GeForce 8400 GS 512 MB (on 1x PCIe riser)
  • GPU: Integrated ATI Radeon HD 4250 128 MB
  • PSU: 10+ year old modular 650w with ripped off tag (sorry, don’t know model anymore)
  • RGB: Custom made ATTiny85 based LED controllers
  • Keyboard: Logitech Internet Navigator Keyboard
  • Mouse: Microsoft IntelliMouse Explorer 2.0
  • Monitor: Dell LCD @ 1280x1024px
Devotion to Duty
https://xkcd.com/705/

Right near the start of the build, #DayJob happened. The Equinix LD8 datacenter in London suffered a near total power failure. My company houses a significant amount of infrastructure in this location, and being on the TechOps team, we went into around-the-clock emergency mode to bring out systems back online after power was restored. Sadly, this ate into my build time. Many of the planned memes and featured games are missing from the final case design and video. At the end of this document, I’ll leave an unedited copy of my personal Windows Notepad brainstorming notes for what I wanted to include. Most is written in short-hand, but it’ll at least give an idea!

All of the extra cardboard used to hold components in place are pieces from this same box, repurposed after being cut out!

The motherboard is held in by 9 traditional mount screws. I cut small holes into the cardboard and screwed these in. They were not quite stable enough, so I added some hot glue to each. Now that motherboard is rock solid, and not moving. 🙂

Next, the PSU. This thing is massively heavy. Using larger thumb screws, these acted as pseudo washers against the cardboard. This mostly keeps the power supply in place. I wanted a little extra stability, so I added some extra cardboard internally to stabilize it.

Next came the I/O. I had a bunch of spare 1x PCIe risers, and I didn’t give a damn about performance!!! So yup, just had to use these for the LAWLz. This is how the GPU, Sound Card, and 10gbe NIC are attached.

One of the random items I had recently purchased for another project was a USB floppy drive. I needed internal USB ports, so I rummaged through my old wire box. I found a 4-port USB 2.0 bracket, and this motherboard luckily has a pair of USB 2.0 headers ready to go for it. The bracket was also stored with a USB WiFi dongle and USB BlueTooth dongle already attached, so SCORE! Bonus features.

The RGB LEDs are not your typic kit. They’re not attached or controlled by the motherboard at all. These are WS2812b LEDs, just like other ARGB LED strips. But by designing my own LED controller and knowing my power budget, it means I can do some extra things with it. I attached each of the two strip’s power sources directly to the PSU’s 5v output. This means I don’t have the normal 500ma power limitation that USB or motherbaord LED strips have. Because of this, I can run them at max brightness without worrying of damaging components (consumer purchased LED controllers wont allow for max brightness usually, due to fear of straining power budgets, which can harm components or even catch fire… which yes, I’ve done while building my own controllers!!)

As you can tell from the video, all of the gaming was done on Windows 98 SE! … but wait … Win98 doesn’t support more than 512MB RAM, or wifi, or bluetooth, or 10gbe, or more than 1 CPU core, or….

**** SPOILERS ****

The host OS is actually Windows 10 1909, with a Windows 98 SE VMware Workstation guest. When I went to set this system, I couldn’t find my Windows 10 2004 USB drive, so I pulled an older 1909 out of my shelf of stuff… then an hour later when I needed my house keys, realized that the 2004 USB drive was attached to my key ring 😛 GO ME.

If you watch the video, you can actually see the VMWare BIOS boot screen during the Windows 98 SE install process while the system reboots. You can also notice the gray bar at the top of the video. Also, after a few days, Windows 10 started showing the “ACTIVATE WINDOWS” always-on-top text, and these shows up in the last few screen recordings.

SORRY for the repeated audio. The last couple clips that have sound were recorded in a hurry before I left the apartment for the weekend (I’m actually publishing this from another location on my laptop and barely meeting the deadline WHOOPSS!). Things got rushed, because once again, the massive server outage at work ate a LOT of time unexpectedly this week!

All in all, this was a fun little build. It runs. It plays games. It looks cool. It memes hard. It has already brought smiles to the faces of those that have seen the work in progress. In the end, that’s all that matters, gaming, and our gaming hardware, bringing smiles of joy to people’s faces!

Windows Notepad Brainstorming

“NOW PARTS ARE IN CASE!”

r-g-bees – printed out pics of bees, colored with red/green sharpie

videos:
* installing windows 98
* each of the games
* dxdiag?
* 3dmark 2000!
* fr-08

k0m-pew-pew-pew-t0r
“infinite” mega-hurtz
million-80-p rezo1ushunz
LCD-9001 (printed piece of paper with “LCD” on it)

fan goes BBBBRRRRR

over a clock (overclocked)

print out of 3d rendering of a character named mark
^^^ Mark the Minion

DONT COPY THAT FLOPPY


case provided by charity: cases for chris


cboard xtreme 2020 edition

cable manager, inspector, supervisor, worker, cheif cable officer
^^^ xkcd characters
^^^ printed out and pasted inside case

iconic sounds from games
wolf3d (whatever the German phase the guards say)
diablo “AHHHH, FRESH MEAT”
doom chainsaw
ut99 “HEADSHOT”
FPS Doug (need I say more!?)
skyroads
space pinball


use knife to start computer, because everything runs faster with a knife


equinix LD8 / xkcd sys admin
rgb power budget / brightness
FLOPPY! becase WHY NOT!?
10gbe! because WHY NOT !?

Emulex: The Cheapest 10gbe for Your Homelab

vince5 Comments

Years ago, the hunt for the cheapest 10gbe NICs resulted in buying Mellanox ConnectX-2 single-port 10gbe network cards from eBay for around $10. Nowadays those cards have increased in cost to around $20-30. While still cheap, not quite the cheapest. There are now alternatives!

Before diving into details, let’s get a something very clear. If you want the absolute simplest plug-and-play 10gbe LAN for your homelab, pay the extra for Mellanox. If you’re willing to go hands-on, do some simple manual configuration and installation, read on for my experiences with Emulex 10gbe NICs.

Emulex NICs can often be had for around $15 on eBay, sometimes even cheaper. I recently picked up a set of 4 of these cards, which came bundled with 6 SFP+ 10g-SR modules for a grand total of $47.48. Considering I can usually find SFP+ modules for about $5/ea, these alone were worth $30.

Card model numbers to look out for

  • Emulex OCE10102
  • Emulex OCE11102
  • HP NC550SFP
  • IBM 49Y7952
  • IBM 96Y3766

FreeNAS 11.3

FreeNAS is the easiest OS to work with. These cards will work out of the box with the latest driver back-ported from FreeBSD 13-CURRENT, and enabled by default. Just put the card in, and configure like you would any other NIC on FreeNAS.

FreeBSD 12.1-RELEASE

FreeBSD 12 includes the latest driver, much like FreeNAS, however it is not enabled to load automatically. One small file edit is all this will take! Add the following line to your /boot/loader.conf file.

root@OPNsense:~ # echo 'if_oce_load="YES"' >> /boot/loader.conf

After that, reboot, and configure your NIC like normal.

FreeBSD 11.2-RELEASE

If all you want is basic network connectivity and VLAN support, simply follow the directions above for FreeBSD 12.1-RELEASE. However, do note that this FreeBSD releases uses an outdated driver that does not work with features such as CARP.

FreeBSD 11.3-RELEASE and FreeBSD 11.4-RELEASE

These versions of FreeBSD include a broken version of the Emulex oce(4) driver. SOME cards will work, SOME cards will cause a kernel panic initializing the driver. Please continue reading on for details on custom compiling the Emulex oce(4) driver.

If you’re already experienced at compiling and implementing FreeBSD driver changes, simply head over to this commit for the patch.

https://github.com/freebsd/freebsd/commit/b44d7086a16f737935e1eec7c7995adbc077081e

OPNsense 20.1 and HardenedBSD 11.2-RELEASE

These OSes come with the same driver included with FreeBSD 11.2-RELEASE. Out of the box, the driver has basic networking and VLAN functionality, but is also disabled by default, and also lacks support for advanced networking features like CARP. If you don’t need the more advanced networking features, simply follow the instructions above for FreeBSD 12.1-RELEASE to enable the Emulex oce(4) driver at boot.

If you want the quick and easy solution, I’ve provided the pre-compiled binary for the patched driver at the following URL. https://vincerants.com/public/opnsense/if_oce.ko

root@OPNsense:~# cd /boot/kernel
root@OPNsense:/boot/kernel# fetch https://vincerants.com/public/opnsense/if_oce.ko
if_oce.ko                                     100% of  128 kB 9307 kBps 00m00s

Custom Compiling the Emulex oce(4) Driver – Prepping the /usr/src Tree

Because my intended use case for these NICs are a series of OPNsense routers, this part of the guide is going to be custom tailored for building the driver natively on this OS.

First, install nano for easier text file editing. Next, enable the FreeBSD package repository. Use this repository to install git so that way we may clone the source tree.

root@OPNsense:~ # pkg install -y nano
Updating OPNsense repository catalogue...
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01
Fetching packagesite.txz: 100%  183 KiB 187.0kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 708 packages processed.
All repositories are up to date.
Updating database digests format: 100%
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.12.0 -> 1.12.0_1

Number of packages to be upgraded: 1

3 MiB to be downloaded.
[1/1] Fetching pkg-1.12.0_1.txz: 100%    3 MiB 903.4kB/s    00:04
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.12.0 to 1.12.0_1...
[1/1] Extracting pkg-1.12.0_1: 100%
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        nano: 4.9.2

Number of packages to be installed: 1

276 KiB to be downloaded.
[1/1] Fetching nano-4.9.2.txz: 100%  276 KiB 282.6kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing nano-4.9.2...
[1/1] Extracting nano-4.9.2: 100%

root@OPNsense:~ # echo 'FreeBSD: { enabled: yes }' > /usr/local/etc/pkg/repos/FreeBSD.conf
root@OPNsense:/usr/src # pkg install -y git
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    916 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   6.5MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 31592 packages processed.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.12.0_1 -> 1.13.2 [FreeBSD]

Number of packages to be upgraded: 1

3 MiB to be downloaded.
[1/1] Fetching pkg-1.13.2.txz: 100%    3 MiB   3.5MB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.12.0_1 to 1.13.2...
[1/1] Extracting pkg-1.13.2: 100%
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following 10 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        cvsps: 2.1_2 [OPNsense]
        fontconfig: 2.13.92_2,1 [FreeBSD]
        freetype2: 2.10.1 [FreeBSD]
        git: 2.26.2 [OPNsense]
        libfontenc: 1.1.4 [FreeBSD]
        libunistring: 0.9.10_1 [FreeBSD]
        libunwind: 20170615 [FreeBSD]
        p5-Error: 0.17029 [OPNsense]
        pcre2: 10.35 [OPNsense]
        png: 1.6.37 [FreeBSD]

Number of packages to be installed: 10

The process will require 50 MiB more space.
9 MiB to be downloaded.
[1/10] Fetching git-2.26.2.txz: 100%    5 MiB   1.3MB/s    00:04
[2/10] Fetching libfontenc-1.1.4.txz: 100%   20 KiB  20.8kB/s    00:01
[3/10] Fetching freetype2-2.10.1.txz: 100%    1 MiB   1.4MB/s    00:01
[4/10] Fetching fontconfig-2.13.92_2,1.txz: 100%  382 KiB 390.9kB/s    00:01
[5/10] Fetching png-1.6.37.txz: 100%  288 KiB 294.8kB/s    00:01
[6/10] Fetching libunwind-20170615.txz: 100%  118 KiB 121.2kB/s    00:01
[7/10] Fetching libunistring-0.9.10_1.txz: 100%  522 KiB 535.0kB/s    00:01
[8/10] Fetching pcre2-10.35.txz: 100%  879 KiB 449.8kB/s    00:02
[9/10] Fetching p5-Error-0.17029.txz: 100%   27 KiB  27.4kB/s    00:01
[10/10] Fetching cvsps-2.1_2.txz: 100%   41 KiB  41.5kB/s    00:01
Checking integrity... done (0 conflicting)
[1/10] Installing freetype2-2.10.1...
[1/10] Extracting freetype2-2.10.1: 100%
[2/10] Installing p5-Error-0.17029...
[2/10] Extracting p5-Error-0.17029: 100%
[3/10] Installing cvsps-2.1_2...
[3/10] Extracting cvsps-2.1_2: 100%
[4/10] Installing git-2.26.2...
===> Creating groups.
Creating group 'git_daemon' with gid '964'.
===> Creating users
Creating user 'git_daemon' with uid '964'.
[4/10] Extracting git-2.26.2: 100%
[5/10] Installing libfontenc-1.1.4...
[5/10] Extracting libfontenc-1.1.4: 100%
[6/10] Installing fontconfig-2.13.92_2,1...
[6/10] Extracting fontconfig-2.13.92_2,1: 100%
Running fc-cache to build fontconfig cache...
Font directories:
        /usr/local/share/fonts
        /usr/local/lib/X11/fonts
/usr/local/share/fonts: skipping, no such directory
/usr/local/lib/X11/fonts: skipping, no such directory
/var/db/fontconfig: cleaning cache directory
fc-cache: succeeded
[7/10] Installing png-1.6.37...
[7/10] Extracting png-1.6.37: 100%
[8/10] Installing libunwind-20170615...
[8/10] Extracting libunwind-20170615: 100%
[9/10] Installing libunistring-0.9.10_1...
[9/10] Extracting libunistring-0.9.10_1: 100%
[10/10] Installing pcre2-10.35...
[10/10] Extracting pcre2-10.35: 100%
=====
Message from freetype2-2.10.1:

--
The 2.7.x series now uses the new subpixel hinting mode (V40 port's option) as
the default, emulating a modern version of ClearType. This change inevitably
leads to different rendering results, and you might change port's options to
adapt it to your taste (or use the new "FREETYPE_PROPERTIES" environment
variable).

The environment variable "FREETYPE_PROPERTIES" can be used to control the
driver properties. Example:

FREETYPE_PROPERTIES=truetype:interpreter-version=35 \
        cff:no-stem-darkening=1 \
        autofitter:warping=1

This allows to select, say, the subpixel hinting mode at runtime for a given
application.

If LONG_PCF_NAMES port's option was enabled, the PCF family names may include
the foundry and information whether they contain wide characters. For example,
"Sony Fixed" or "Misc Fixed Wide", instead of "Fixed". This can be disabled at
run time with using pcf:no-long-family-names property, if needed. Example:

FREETYPE_PROPERTIES=pcf:no-long-family-names=1

How to recreate fontconfig cache with using such environment variable,
if needed:
# env FREETYPE_PROPERTIES=pcf:no-long-family-names=1 fc-cache -fsv

The controllable properties are listed in the section "Controlling FreeType
Modules" in the reference's table of contents
(/usr/local/share/doc/freetype2/reference/site/index.html, if documentation was installed).
=====
Message from git-2.26.2:

--
If you installed the GITWEB option please follow these instructions:

In the directory /usr/local/share/examples/git/gitweb you can find all files to
make gitweb work as a public repository on the web.

All you have to do to make gitweb work is:
1) Please be sure you're able to execute CGI scripts in
   /usr/local/share/examples/git/gitweb.
2) Set the GITWEB_CONFIG variable in your webserver's config to
   /usr/local/etc/git/gitweb.conf. This variable is passed to gitweb.cgi.
3) Restart server.


If you installed the CONTRIB option please note that the scripts are
installed in /usr/local/share/git-core/contrib. Some of them require
other ports to be installed (perl, python, etc), which you may need to
install manually.
root@OPNsense:/usr/src # git clone https://github.com/HardenedBSD/hardenedBSD.git -b hardened/11.2-releng/master --single-branch --depth=1 /usr/src
Cloning into '/usr/src'...
remote: Enumerating objects: 77339, done.
remote: Counting objects: 100% (77339/77339), done.
remote: Compressing objects: 100% (68364/68364), done.
remote: Total 77339 (delta 16296), reused 29974 (delta 6113), pack-reused 0
Receiving objects: 100% (77339/77339), 252.74 MiB | 9.96 MiB/s, done.
Resolving deltas: 100% (16296/16296), done.
Updating files: 100% (74050/74050), done.

Custom Compiling the Emulex oce(4) Driver – Updating the Driver Source Code

Now that we have the HardenedBSD 11.2-RELEASE source code, we need to update the driver to a more recent version, and patch it to ensure the kernel will not segfault on boot.

root@OPNsense:~ # cd /usr/src/sys/net/
root@OPNsense:/usr/src/sys/net # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/net/ethernet.h
ethernet.h                                    100% of   21 kB   34 MBps 00m00s

root@OPNsense:~ # cd /usr/src/sys/dev/oce/

root@OPNsense:/usr/src/sys/dev/oce # ll
total 340
-rw-r--r--  1 root  wheel  14018 Jun 21 20:36 oce_hw.c
-rw-r--r--  1 root  wheel  86688 Jun 21 20:36 oce_hw.h
-rw-r--r--  1 root  wheel  53329 Jun 21 20:36 oce_if.c
-rw-r--r--  1 root  wheel  33583 Jun 21 20:36 oce_if.h
-rw-r--r--  1 root  wheel  59884 Jun 21 20:36 oce_mbox.c
-rw-r--r--  1 root  wheel  26826 Jun 21 20:36 oce_queue.c
-rw-r--r--  1 root  wheel  49140 Jun 21 20:36 oce_sysctl.c
-rw-r--r--  1 root  wheel   6182 Jun 21 20:36 oce_util.c

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_hw.c
oce_hw.c                                      100% of   13 kB   37 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_hw.h
oce_hw.h                                      100% of   97 kB   26 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_if.c
oce_if.c                                      100% of   73 kB   37 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_if.h
oce_if.h                                      100% of   36 kB   48 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_mbox.c
oce_mbox.c                                    100% of   64 kB   40 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_queue.c
oce_queue.c                                   100% of   33 kB   45 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_sysctl.c
oce_sysctl.c                                  100% of   52 kB   49 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_user.h
oce_user.h                                    100% of 3497  B   16 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://raw.githubusercontent.com/freebsd/freebsd/release/11.4.0/sys/dev/oce/oce_util.c
oce_util.c                                    100% of 6182  B   23 MBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # fetch https://github.com/freebsd/freebsd/commit/b44d7086a16f737935e1eec7c7995adbc077081e.patch
b44d7086a16f737935e1eec7c7995adbc077081e.patch 100% of 1603  B 9043 kBps 00m00s

root@OPNsense:/usr/src/sys/dev/oce # ll
total 408
-rw-r--r--  1 root  wheel   1603 Jun 21 20:51 b44d7086a16f737935e1eec7c7995adbc077081e.patch
-rw-r--r--  1 root  wheel  14174 Jun 21 20:44 oce_hw.c
-rw-r--r--  1 root  wheel  99367 Jun 21 20:44 oce_hw.h
-rw-r--r--  1 root  wheel  75369 Jun 21 20:44 oce_if.c
-rw-r--r--  1 root  wheel  37330 Jun 21 20:44 oce_if.h
-rw-r--r--  1 root  wheel  65545 Jun 21 20:44 oce_mbox.c
-rw-r--r--  1 root  wheel  33799 Jun 21 20:44 oce_queue.c
-rw-r--r--  1 root  wheel  53522 Jun 21 20:44 oce_sysctl.c
-rw-r--r--  1 root  wheel   3497 Jun 21 20:48 oce_user.h
-rw-r--r--  1 root  wheel   6182 Jun 21 20:44 oce_util.c

Custom Compiling the Emulex oce(4) Driver – Applying the Patch and Compiling

With all of the files downloaded, it is time to apply the patch and compile our new driver.

root@OPNsense:/usr/src/sys/dev/oce # patch oce_if.c b44d7086a16f737935e1eec7c7995adbc077081e.patch
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From b44d7086a16f737935e1eec7c7995adbc077081e Mon Sep 17 00:00:00 2001
|From: mav <mav@FreeBSD.org>
|Date: Tue, 28 May 2019 18:32:04 +0000
|Subject: [PATCH] Fix array out of bound panic introduced in r306219.
|
|As I see, different NICs in different configurations may have different
|numbers of TX and RX queues.  The code was assuming 1:1 mapping between
|event queues (interrupts) and TX/RX queues.  Since number of interrupts
|is set to maximum of TX and RX queues, when those two are different, the
|system is doomed.
|
|I have no documentation or deep knowledge about this hardware, so this
|change is based on general observations and code reading.  If some of my
|guesses are wrong, please do better.  I just confirmed HP NC550SFP NICs
|are working now.
|
|MFC after:     2 weeks
|Sponsored by:  iXsystems, Inc.
|---
| sys/dev/oce/oce_if.c | 18 ++++++++++++++----
| 1 file changed, 14 insertions(+), 4 deletions(-)
|
|diff --git a/sys/dev/oce/oce_if.c b/sys/dev/oce/oce_if.c
|index 221a871a7ceb5..44fb04fef791e 100644
|--- a/sys/dev/oce/oce_if.c
|+++ b/sys/dev/oce/oce_if.c
--------------------------
Patching file oce_if.c using Plan A...
Hunk #1 succeeded at 2394.
done
root@OPNsense:/ # cd /usr/src/sys/modules/oce

root@OPNsense:/usr/src/sys/modules/oce # make
machine -> /usr/src/sys/amd64/include
x86 -> /usr/src/sys/x86/include
awk -f /usr/src/sys/tools/makeobjops.awk /usr/src/sys/kern/bus_if.m -h
awk -f /usr/src/sys/tools/makeobjops.awk /usr/src/sys/kern/device_if.m -h
awk -f /usr/src/sys/tools/makeobjops.awk /usr/src/sys/dev/pci/pci_if.m -h
Warning: Object directory not changed from original /usr/src/sys/modules/oce
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_if.o -MToce_if.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_if.c -o oce_if.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_hw.o -MToce_hw.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_hw.c -o oce_hw.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_mbox.o -MToce_mbox.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_mbox.c -o oce_mbox.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_util.o -MToce_util.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_util.c -o oce_util.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_queue.o -MToce_queue.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_queue.c -o oce_queue.o
cc -O2 -pipe -DHARDENEDBSD -DSMP  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc  -I/usr/src/sys/dev/oce -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer   -MD  -MF.depend.oce_sysctl.o -MToce_sysctl.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c /usr/src/sys/dev/oce/oce_sysctl.c -o oce_sysctl.o
ld -d -warn-common -r -d -o if_oce.ko oce_if.o oce_hw.o oce_mbox.o oce_util.o oce_queue.o oce_sysctl.o
:> export_syms
awk -f /usr/src/sys/conf/kmod_syms.awk if_oce.ko  export_syms | xargs -J% objcopy % if_oce.ko
objcopy --strip-debug if_oce.ko
root@OPNsense:/usr/src/sys/modules/oce # ll
total 392
-rw-r--r--  1 root  wheel    4039 Jun 21 20:56 .depend.oce_hw.o
-rw-r--r--  1 root  wheel    4098 Jun 21 20:56 .depend.oce_if.o
-rw-r--r--  1 root  wheel    4043 Jun 21 20:56 .depend.oce_mbox.o
-rw-r--r--  1 root  wheel    4049 Jun 21 20:56 .depend.oce_queue.o
-rw-r--r--  1 root  wheel    4051 Jun 21 20:56 .depend.oce_sysctl.o
-rw-r--r--  1 root  wheel    4043 Jun 21 20:56 .depend.oce_util.o
-rw-r--r--  1 root  wheel     335 Jun 21 20:36 Makefile
-rw-r--r--  1 root  wheel   40423 Jun 21 20:56 bus_if.h
-rw-r--r--  1 root  wheel   11902 Jun 21 20:56 device_if.h
-rw-r--r--  1 root  wheel       0 Jun 21 20:56 export_syms
-rw-r--r--  1 root  wheel  131288 Jun 21 20:56 if_oce.ko
lrwxr-xr-x  1 root  wheel      26 Jun 21 20:56 machine@ -> /usr/src/sys/amd64/include
-rw-r--r--  1 root  wheel    9432 Jun 21 20:56 oce_hw.o
-rw-r--r--  1 root  wheel   48512 Jun 21 20:56 oce_if.o
-rw-r--r--  1 root  wheel   23912 Jun 21 20:56 oce_mbox.o
-rw-r--r--  1 root  wheel   19280 Jun 21 20:56 oce_queue.o
-rw-r--r--  1 root  wheel   37160 Jun 21 20:56 oce_sysctl.o
-rw-r--r--  1 root  wheel    3128 Jun 21 20:56 oce_util.o
-rw-r--r--  1 root  wheel      37 Jun 21 20:56 opt_inet.h
-rw-r--r--  1 root  wheel      16 Jun 21 20:56 opt_inet6.h
-rw-r--r--  1 root  wheel   18641 Jun 21 20:56 pci_if.h
lrwxr-xr-x  1 root  wheel      24 Jun 21 20:56 x86@ -> /usr/src/sys/x86/include

Custom Compiling the Emulex oce(4) Driver – Installing the Driver

With the driver built the only thing left to do is install it on the system. Copy the driver to the /boot/kernel/ folder and enable the driver in /boot/loader.conf

root@OPNsense:/usr/src/sys/modules/oce # cp if_oce.ko /boot/kernel/

root@OPNsense:/usr/src/sys/modules/oce # ll /boot/kernel/ | grep oce
-r-xr-xr-x  1 root  wheel    131288 Jun 22 04:21 if_oce.ko*
root@OPNsense:~ # echo 'if_oce_load="YES"' >> /boot/loader.conf