Editing Main Page (section)

== FreeBSD ==
=== PAM Error ===
<pre>
su
su: pam_start: System error
</pre>
remove OPIE in /etc/pam.d/system
<pre>
# auth
#auth            sufficient      pam_opie.so             no_warn no_fake_prompts
#auth            requisite       pam_opieaccess.so       no_warn allow_local
</pre>

=== Packages ===

https://pkg-status.freebsd.org/builds/foo:default:142amd64:2025-04-09_07h07m39s:gohan06#new_failed

=== CUDA ===
https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Rocky&target_version=9&target_type=rpm_local
<pre>
pkg install nvidia-driver  
kldload nvidia
kldload nvidia-modeset 
 # sysrc kld_list+=nvidia-modeset 
kldload linux64
pkg install linux_base-rl9  
pkg install linux-rl9-libxml2
pkg install gcc #gcc13

fetch https://developer.download.nvidia.com/compute/cuda/12.5.1/local_installers/cuda_12.5.1_555.42.06_linux.run
./cuda_12.5.1_555.42.06_linux.run

fetch https://developer.download.nvidia.com/compute/cuda/12.5.1/local_installers/cuda-repo-rhel9-12-5-local-12.5.1_555.42.06-1.x86_64.rpm

rpm2cpio cuda-repo-rhel9-12-5-local-12.5.1_555.42.06-1.x86_64.rpm > cuda.cpio
cpio -idmv < cuda.cpio

var/cuda-repo-rhel9-12-5-local# rpm2cpio nvidia-driver-cuda-libs-555.42.06-1.el9.x86_64.rpm > nvidia-driver-cuda-libs-555.42.06-1.el9.x86_64.cpio
cpio -idmv < nvidia-driver-cuda-libs-555.42.06-1.el9.x86_64.cpio


brandelf -t Linux <path_to_CUDA_installer>
./cuda_<version>_linux.run

LD_LIBRARY_PATH=/usr/lib64 ffmpeg -i TheBear-S03E01.ts -c:v h264_nvenc -preset slow -b:v 5000k -maxrate 5000k -bufsize 10000k -pix_fmt yuv420p -c:a aac output.mp4

</pre>
==== Davinci Resolve on Freebsd ====
https://www.youtube.com/watch?v=uQPNyQmvOIQ&t=222s

https://github.com/NapoleonWils0n/davinci-resolve-freebsd

Make sure Nvidia and Cuda is set up on the Freebsd host

<pre>sudo pkg install nvidia-driver linux-nvidia-libs libc6-shim libvdpau-va-gl libva-nvidia-driver
</pre>
run nv-sglrun nvidia-smi

<pre>
nv-sglrun nvidia-smi
</pre>
nv-sglrun nvidia-smi output
<pre>
shim init
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.146.02             Driver Version: 535.146.02   CUDA Version: 12.2     |
</pre>

=== Containers ===
containerd / runj

https://productionwithscissors.run/2022/09/04/containerd-linux-on-freebsd/

<pre>

https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda/tags
nvcr.io/nvidia/cuda:12.5.1-cudnn-devel-ubuntu20.04

https://github.com/canonical/microk8s/issues/1239
#ctr run --rm --gpus 0 docker.io/nvidia/cuda:9.0-base nvidia-smi nvidia-smi

ctr image pull --platform=linux nvcr.io/nvidia/cuda:12.5.1-cudnn-devel-ubuntu20.04
</pre>

=== VMs ===
==== qemu / quickemu ====
https://github.com/quickemu-project/quickemu?tab=readme-ov-file
<pre>
pkg install qemu-nox11 coreutils socat usbutils swtpm zsync samba416 qemu-nox11 edk2-qemu-x64
git clone https://github.com/quickemu-project/quickemu
cd quickemu
./quickget alpine v3.11

</pre>
<pre>cat freebsd.patch</pre>
<pre>
diff --git a/quickemu b/quickemu
index f2bd9df..2ad1933 100755
--- a/quickemu
+++ b/quickemu
@@ -652,6 +652,8 @@ function configure_bios() {
             if command -v brew &>/dev/null; then
                 SHARE_PATH="$(brew --prefix qemu)/share"
             fi
+       elif [ "${OS_KERNEL}" == "FreeBSD" ]; then
+               SHARE_PATH=/usr/local/share
         fi
 
         # https://bugzilla.redhat.com/show_bug.cgi?id=1929357#c5
@@ -675,7 +677,9 @@ function configure_bios() {
                         "${SHARE_PATH}/edk2-ovmf/OVMF_CODE.fd","${SHARE_PATH}/edk2-ovmf/OVMF_VARS.fd" \
                         "${SHARE_PATH}/qemu/ovmf-x86_64-4m-code.bin","${SHARE_PATH}/qemu/ovmf-x86_64-4m-vars.bin" \
                         "${SHARE_PATH}/qemu/edk2-x86_64-code.fd","${SHARE_PATH}/qemu/edk2-x86_64-code.fd" \
-                        "${SHARE_PATH}/edk2-ovmf/x64/OVMF_CODE.fd","${SHARE_PATH}/edk2-ovmf/x64/OVMF_VARS.fd"
+                        "${SHARE_PATH}/edk2-ovmf/x64/OVMF_CODE.fd","${SHARE_PATH}/edk2-ov
mf/x64/OVMF_VARS.fd" \
+                        "${SHARE_PATH}/edk2-qemu/QEMU_UEFI_CODE-x86_64.fd","${SHARE_PATH}
/edk2-qemu/QEMU_UEFI_VARS-x86_64.fd"
+
                     );;
             esac
             # Attempt each EFI_CODE file one by one, selecting the corresponding code and
 vars
</pre>
=== Linuxulator ===
==== nginx linuxulator ====
<pre>
2025/04/09 17:52:30 [emerg] 51722#51722: io_setup() failed (38: Function not implemented)
2025/04/09 17:52:30 [alert] 51721#51721: epoll_ctl(1, 6) failed (22: Invalid argument)
/var/log/nginx # grep fail error.log 
</pre>
<pre>
# podman run --os=linux -d --name nginx nginx:1.27.4
# dmesg | grep linux | tail
linux: jid 106 pid 92741 (nginx): syscall io_setup not implemented
linux: jid 106 pid 92739 (nginx): epoll_ctl unsupported flags: 0x10000001
</pre>
<pre>
# kldunload linux_common
kldunload: can't unload file: Device busy
kldstat
Id Refs Address                Size Name
77    1 0xffffffff8ba2e000     73c0 linprocfs.ko
78    1 0xffffffff8baa7000     c2a8 linux_common.ko
# kldunload linprocfs
</pre>
<pre>
procstat -ak | grep linux
</pre>
https://github.com/jiixyj/epoll-shim
This is a small library that implements epoll on top of kqueue.

https://www.freshports.org/devel/libepoll-shim/
===== io_setup =====
https://github.com/TritonDataCenter/illumos-joyent/blob/master/usr/src/uts/common/brand/lx/syscall/lx_aio.c

==== netcat ====
nc -kl 8080
telnet localhost 8080
==== Linux Sys Calls ====
https://man7.org/linux/man-pages/man2/syscalls.2.html

AIO 
https://elixir.bootlin.com/linux/v6.13.7/source/fs/aio.c#L1382

==== SmartOS LX Sys Calls ====
https://github.com/TritonDataCenter/illumos-joyent/tree/master/usr/src/uts/common/brand/lx

https://github.com/TritonDataCenter/illumos-joyent/blob/master/usr/src/uts/common/brand/lx/os/lx_syscall.c

AIO
https://github.com/TritonDataCenter/illumos-joyent/blob/master/usr/src/uts/common/brand/lx/syscall/lx_aio.c

==== FreeBSD Sys Calls ====
https://cgit.freebsd.org/src/tree/sys/amd64/linux/linux_syscall.h
https://cgit.freebsd.org/src/tree/sys/compat/linux/linux_dummy.c

https://alfonsosiciliano.gitlab.io/posts/2021-01-02-freebsd-system-calls-table.html

https://alfonsosiciliano.gitlab.io/posts/2023-08-28-freebsd-15-system-calls.html


https://docs.freebsd.org/en/articles/linux-emulation/#syscall-handling
=== FreeBSD dev ===
<pre>pkg install lua54-luafilesystem lua54-posix</pre>
==== cscope ====
<pre>
cd /usr/src/sys
find . -name '*.c' -o -name '*.h' > cscope.files
cscope -b -q -k
</pre>
start cscope
<pre>cscope -d</pre>
To quit: Ctrl+D


==== differences ====

Yes, I’m familiar with both '''SmartOS LX zones''' and '''FreeBSD’s Linuxulator'''. These are two different approaches to providing Linux binary compatibility on non-Linux systems, and their differing abilities to run '''systemd''' highlight some key technical distinctions. Let’s break this down and address what’s missing in FreeBSD’s Linuxulator to support systemd, as well as how you might approach enabling it by leveraging SmartOS/Illumos source code.
===== SmartOS LX Zones and FreeBSD Linuxulator =====

* '''SmartOS LX Zones''': LX zones are a feature of Illumos-based systems (like SmartOS) that allow running Linux binaries in a lightweight containerized environment called a "branded zone." These zones provide a near-native Linux kernel ABI (Application Binary Interface) by translating Linux system calls into Illumos equivalents. LX zones were originally developed by Joyent to support running Linux applications, including Docker, on SmartOS. Over time, they’ve been extended to support more complex Linux-specific features, including systemd, which is the default init system for many modern Linux distributions.

* '''FreeBSD Linuxulator''': The Linuxulator is FreeBSD’s Linux binary compatibility layer. It allows FreeBSD to run unmodified Linux ELF binaries by translating Linux system calls to FreeBSD equivalents. Unlike LX zones, it’s not a containerization technology but a kernel-level compatibility layer that operates alongside native FreeBSD processes. It currently supports a subset of Linux kernel functionality, with compatibility levels pegged at Linux 3.2.0 in FreeBSD 12-STABLE and Linux 5.15.0 in FreeBSD 14-STABLE (as of recent documentation). However, it lacks support for several Linux-specific features required by systemd.

==== Why Systemd Works in LX Zones but Not Linuxulator ====
Systemd is a complex init system that relies heavily on Linux-specific kernel features, such as:
* '''cgroups (Control Groups)''': Used for resource management and process grouping.
* '''Namespaces''': Particularly PID, mount, and user namespaces, which provide isolation and scoping for processes.
* '''Fanotify''': A file system notification system used by systemd for monitoring file events.
* '''Specific System Calls''': Systemd depends on calls like <code>pidfd_open()</code>, <code>clone3()</code>, and others introduced in newer Linux kernels.
* '''udev Integration''': Systemd integrates tightly with udev, which relies on Linux-specific device management mechanisms.

'''SmartOS LX Zones''' have evolved to support these features. The Illumos kernel, with contributions from Joyent and the SmartOS community, has implemented enough of the Linux kernel ABI—including cgroups, namespaces, and additional system calls—to allow systemd to function within an LX-branded zone. This was driven by the need to support Docker and other modern Linux workloads, which often assume systemd is present.

'''FreeBSD Linuxulator''', on the other hand, does not yet support these features fully. While it can run many Linux binaries (e.g., statically linked programs or those with simpler dependencies), its compatibility layer is incomplete for systemd’s requirements. The Linuxulator’s development has historically focused on supporting popular applications (like Steam or proprietary tools) rather than replicating the full Linux kernel feature set, and systemd support has not been a priority.

==== What’s Missing in FreeBSD Linuxulator? ====
To run systemd in FreeBSD’s Linuxulator, the following gaps need to be addressed:

===== 1. cgroups Support =====
* '''Current State''': FreeBSD has no native cgroups implementation. It uses its own resource control mechanisms (e.g., <code>rctl</code> for resource limits and jails for isolation), which are philosophically and technically different from Linux’s cgroups.
* '''Requirement''': Systemd expects cgroups v1 or v2 for process grouping, resource limiting, and service management. Without this, systemd cannot initialize properly.

===== 2. Namespaces =====
* '''Current State''': FreeBSD supports some namespace-like features via jails and VNET (virtual networking), but these are not equivalent to Linux’s PID, mount, or user namespaces. The Linuxulator does not translate Linux namespace-related system calls (e.g., <code>unshare()</code>, <code>clone()</code> with namespace flags) into FreeBSD equivalents.
* '''Requirement''': Systemd uses namespaces for service isolation and scoping, particularly PID namespaces to manage process trees.

===== 3. System Call Gaps =====
* '''Current State''': The Linuxulator implements a subset of Linux system calls (see <code>sys/compat/linux/linux_sysent.c</code> in the FreeBSD source). Missing calls include newer ones like <code>pidfd_open()</code> (Linux 5.3+), <code>clone3()</code> (Linux 5.3+), and others that systemd may use.
* '''Requirement''': Full compatibility with the system calls systemd depends on, which may require extending the Linuxulator’s syscall table.

===== 4. Fanotify and File System Events =====
* '''Current State''': FreeBSD has <code>kqueue</code> for event notification, but it’s not compatible with Linux’s <code>fanotify</code> API, which systemd uses for file system monitoring.
* '''Requirement''': An emulation layer to map <code>fanotify</code> calls to <code>kqueue</code> or a custom implementation.

===== 5. udev and Device Management =====
* '''Current State''': FreeBSD uses <code>devd</code> for device management, which differs significantly from Linux’s <code>udev</code>. The Linuxulator does not fully emulate <code>udev</code>’s behavior or its integration with the kernel.
* '''Requirement''': Systemd expects <code>udev</code> to manage device events, so a compatibility layer or partial <code>udev</code> emulation is needed.

===== 6. Kernel ABI Stability and Versioning =====
* '''Current State''': The Linuxulator mimics a specific Linux kernel version (e.g., 5.15.0 in FreeBSD 14-STABLE), but systemd may expect features from newer kernels or specific behaviors not yet emulated.
* '''Requirement''': Ensuring the emulated ABI matches a Linux kernel version that systemd supports (e.g., at least 4.4 or later for modern versions).

==== Where to Start: Leveraging SmartOS/Illumos Source ====
Since both SmartOS (Illumos) and FreeBSD are open source, you can study the LX zone implementation to guide enhancements to the Linuxulator. Here’s a step-by-step approach:

===== 1. Study Illumos LX Zone Source Code =====
* '''Location''': The relevant code is in the Illumos gate repository, specifically under <code>usr/src/uts/common/brand/lx/</code> and related directories. Key files include:
** <code>lx_syscalls.c</code>: Handles Linux system call emulation.
** <code>lx_cgroups.c</code>: Implements cgroups support.
** <code>lx_namespace.c</code>: Manages Linux namespace emulation.
** <code>lx_fanotify.c</code>: Provides fanotify support.
* '''Key Insight''': Illumos has extended its kernel to natively support these features for LX zones, rather than just translating them to existing Solaris/Illumos mechanisms. This suggests FreeBSD may need a hybrid approach—either adding native support or building a more comprehensive translation layer.

===== 2. Map Illumos Features to FreeBSD =====
* '''cgroups''': Start by reviewing <code>lx_cgroups.c</code>. You could either:
** Implement a minimal cgroups shim in the Linuxulator that maps to FreeBSD’s <code>rctl</code> or a new subsystem.
** Port Illumos’s cgroups implementation directly, adapting it to FreeBSD’s kernel architecture.
* '''Namespaces''': Look at <code>lx_namespace.c</code>. FreeBSD jails and VNET provide a foundation, but you’d need to extend the Linuxulator to handle <code>clone()</code> and <code>unshare()</code> with namespace flags, possibly by mapping them to jail-like constructs.
* '''System Calls''': Compare Illumos’s <code>lx_syscalls.c</code> with FreeBSD’s <code>sys/compat/linux/linux_sysent.c</code>. Add missing syscalls (e.g., <code>pidfd_open()</code>, <code>clone3()</code>) by adapting Illumos’s implementations or writing new ones based on Linux kernel source (e.g., from kernel.org).
* '''Fanotify''': Use <code>lx_fanotify.c</code> as a model to build a translation layer from <code>fanotify</code> to <code>kqueue</code>.

===== 3. Modify FreeBSD Linuxulator =====
* '''Source Location''': FreeBSD’s Linuxulator code is in <code>sys/compat/linux/</code> in the FreeBSD source tree (e.g., <code>/usr/src/sys/compat/linux/</code> if you’ve cloned the repo).
* '''Steps''':
# Extend the syscall table (<code>linux_sysent.c</code>) with missing calls, using Illumos as a reference.
# Add new kernel modules or hooks for cgroups and namespaces, potentially under <code>sys/compat/linux/</code>.
# Implement fanotify emulation by integrating with <code>kqueue</code>.
# Test with a minimal systemd binary (e.g., from a lightweight distro like Alpine) in a chroot environment.

==== 4. Test and Iterate =====
* '''Setup''': Use a FreeBSD jail with Linuxulator enabled (e.g., via <code>sysctl compat.linux.emulation=1</code> and a Linux base system in <code>/compat/linux</code> or <code>/compat/ubuntu</code>).
* '''Test Case''': Try running <code>systemd --version</code> or a simple <code>systemd</code> init process. Debug failures using <code>ktrace</code> or <code>truss</code> to identify missing syscalls or features.
* '''Reference''': Illumos’s LX zone testing scripts (e.g., in <code>usr/src/test/</code>) can inspire a test suite.

===== 5. Community and Documentation =====
* '''FreeBSD Resources''': Check the FreeBSD Wiki’s Linuxulator page and the <code>linux(4)</code> man page for current capabilities and known gaps.
* '''Illumos Resources''': The Illumos wiki and SmartOS documentation provide insights into LX zone internals.
* '''Collaboration''': Engage with the FreeBSD and Illumos communities (e.g., via mailing lists like <code>freebsd-hackers@</code> or the Illumos IRC) to validate your approach and get feedback.

==== Challenges and Considerations ====
* '''Philosophical Differences''': FreeBSD avoids adopting Linux-specific features natively (e.g., cgroups) to maintain its distinct identity. You may need to argue for a compatibility-only approach rather than kernel bloat.
* '''Performance''': Emulating complex features like cgroups and namespaces could introduce overhead, unlike Illumos’s native integration.
* '''Upstream Acceptance''': Contributions to FreeBSD must align with project goals. Systemd support might face resistance unless justified by broad use cases (e.g., Docker compatibility).

==== Starting Point ====
Clone the FreeBSD source (<code>git clone https://git.freebsd.org/src.git</code>) and Illumos gate (<code>git clone https://github.com/illumos/illumos-gate.git</code>). Begin by prototyping cgroups support—perhaps a minimal shim in <code>sys/compat/linux/</code>—and test it with a systemd binary. Use Illumos’s <code>lx_cgroups.c</code> as a blueprint, adapting it to FreeBSD’s kernel structure (e.g., replacing Illumos’s <code>zone_t</code> with FreeBSD’s <code>struct prison</code> for jail integration).

This is a significant undertaking, but with both projects being open source, you have the raw materials to bridge the gap. Good luck, and feel free to ask if you need more specific guidance as you dive in!