How to Configure Jails with VNET Networking on FreeBSD

FreeBSD’s jail subsystem is a powerful virtualization technology that allows system administrators to partition a computer system into multiple semi-independent mini-systems called jails. When combined with VNET (virtual network stack) capabilities, jails become even more flexible and isolated. This article provides a detailed guide on implementing jails with VNET networking in FreeBSD.

Understanding VNET Jails

Traditional FreeBSD jails share the host’s network stack, which means they’re limited to using IP aliases on the host’s network interfaces. VNET jails, on the other hand, have their own independent network stack, including:

• Virtual network interfaces
• Routing tables
• Firewalls
• Socket infrastructure

This enhanced isolation provides significant advantages:

1. Each jail can have its own firewall rules
2. Jails can use conflicting IP configurations
3. Network services can be completely confined within the jail
4. Jails can be connected to different network segments

Prerequisites

Before configuring VNET jails, ensure your FreeBSD system meets these requirements:

• FreeBSD 12.0 or newer (VNET has been stable since FreeBSD 12.0)
• A kernel with VIMAGE support (included in the GENERIC kernel since FreeBSD 12.0)
• Root access to the host system
• Basic understanding of FreeBSD networking
• Sufficient system resources for the jails you plan to create

Step 1: Enable Required Kernel Modules

First, ensure the necessary kernel modules are loaded by adding these lines to /boot/loader.conf:

if_bridge_load="YES"
if_epair_load="YES"

If you’re not using the GENERIC kernel, make sure your custom kernel has these options:

options         VIMAGE
options         RACCT
options         RCTL

Step 2: Configure the Host Network

We’ll create a bridge interface to connect our jails. Edit /etc/rc.conf to add:

# Enable packet forwarding
gateway_enable="YES"

# Bridge configuration
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 10.0.0.1/24 description jailnet up"

# Add firewall rules if needed
pf_enable="YES"
pf_rules="/etc/pf.conf"

This creates a bridge interface bridge0 with the IP address 10.0.0.1/24, which will serve as the gateway for your jails.

Step 3: Set Up Basic Jail Configuration

Create a base jail configuration in /etc/jail.conf:

# Global parameters
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
allow.raw_sockets;
allow.sysvipc;

# VNET-specific parameters
vnet;
vnet.interface = "epair${jid}b";

# Path parameters
path = "/usr/jails/${name}";
host.hostname = "${name}.jail";

The vnet parameter enables the virtual network stack, and vnet.interface automatically assigns an epair interface to the jail. The ${jid} will be replaced with the jail ID at runtime.

Step 4: Create a Jail Template

You’ll need a base system for your jails. There are multiple approaches:

Using bsdinstall

mkdir -p /usr/jails/template
bsdinstall jail /usr/jails/template

Using make buildworld and make installworld

mkdir -p /usr/jails/template
cd /usr/src
make buildworld
make installworld DESTDIR=/usr/jails/template
make distribution DESTDIR=/usr/jails/template

Don’t forget to configure basic settings in the template:

echo 'sshd_enable="YES"' > /usr/jails/template/etc/rc.conf
echo 'sendmail_enable="NONE"' >> /usr/jails/template/etc/rc.conf
cp /etc/resolv.conf /usr/jails/template/etc/

Step 5: Create Individual Jail Configurations

Add jail-specific configurations to /etc/jail.conf. Here’s an example for a jail named “webserver”:

webserver {
    # VNET specific setup
    exec.prestart = "ifconfig epair${jid} create up";
    exec.prestart += "ifconfig epair${jid}a up descr ${name}";
    exec.prestart += "ifconfig bridge0 addm epair${jid}a";
    
    # Setup jail's network when started
    exec.start += "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start += "/sbin/ifconfig epair${jid}b 10.0.0.10/24 up";
    exec.start += "/sbin/route add default 10.0.0.1";
    
    # Clean up on jail stop
    exec.poststop = "ifconfig bridge0 deletem epair${jid}a";
    exec.poststop += "ifconfig epair${jid}a destroy";
}

This configuration:

1. Creates an epair interface (epair${jid})
2. Connects one end to the bridge (epair${jid}a)
3. Assigns the other end (epair${jid}b) to the jail
4. Configures the jail’s network with an IP (10.0.0.10/24) and default route

Step 6: Create and Start Your First VNET Jail

First, clone the template for your new jail:

mkdir -p /usr/jails/webserver
cp -R /usr/jails/template/* /usr/jails/webserver/

Then update the jail’s specific configuration:

echo 'hostname="webserver.jail"' > /usr/jails/webserver/etc/rc.conf.local
echo 'ifconfig_epair0b="inet 10.0.0.10/24"' >> /usr/jails/webserver/etc/rc.conf.local
echo 'defaultrouter="10.0.0.1"' >> /usr/jails/webserver/etc/rc.conf.local

Start the jail:

service jail start webserver

Step 7: Configure NAT for Internet Access

If you want your jails to access the internet, you’ll need to set up NAT. Here’s a simple PF configuration for /etc/pf.conf:

# Define interfaces
ext_if = "em0"  # Your external interface, replace with yours
jail_net = "10.0.0.0/24"

# NAT rule for jail network
nat on $ext_if from $jail_net to any -> ($ext_if)

# Allow all traffic (customize as needed)
pass in all
pass out all

Reload PF to apply the changes:

service pf reload

Step 8: Advanced VNET Configurations

Multiple Network Interfaces

You can assign multiple interfaces to a jail by configuring additional epair devices:

myjail {
    # ... other settings ...
    
    # First network interface
    vnet.interface = "epair${jid}b";
    exec.prestart = "ifconfig epair${jid} create up";
    exec.prestart += "ifconfig epair${jid}a up";
    exec.prestart += "ifconfig bridge0 addm epair${jid}a";
    
    # Second network interface
    exec.prestart += "ifconfig epair${jid}x create up";
    exec.prestart += "ifconfig epair${jid}xa up";
    exec.prestart += "ifconfig bridge1 addm epair${jid}xa";
    exec.start += "/sbin/ifconfig epair${jid}xb 192.168.1.10/24 up";
    
    # ... other settings ...
}

VLAN Configuration

VNET jails can participate in VLANs:

exec.prestart += "ifconfig vlan100 create vlandev em0 vlan 100 up";
exec.prestart += "ifconfig bridge0 addm vlan100";
# ... configure jail with appropriate IP on the VLAN network ...

Jail-Specific Firewall

Since each VNET jail has its own network stack, you can run a separate firewall instance within the jail:

# Inside the jail:
echo 'pf_enable="YES"' >> /etc/rc.conf
echo 'pf_rules="/etc/pf.conf"' >> /etc/rc.conf

Create a jail-specific /etc/pf.conf and start the service.

Step 9: Troubleshooting VNET Jails

Common Issues and Solutions

1. Jail fails to start with “vnet” errors:

   • Ensure kernel modules are loaded: kldstat | grep if_epair
   • Make sure your kernel has VIMAGE support: sysctl kern.features.vimage

2. No network connectivity from jail:

   • Check interface status: ifconfig (both in host and jail)
   • Verify routing table: netstat -rn (both in host and jail)
   • Ensure bridge is properly configured: ifconfig bridge0

3. Can’t connect to services in the jail:

   • Check firewall rules in both host and jail
   • Verify service is running in the jail: service <service> status

Debugging Commands

Use these commands for troubleshooting:

# On the host
jls -v                # View detailed jail information
ifconfig              # Check network interfaces
tcpdump -i bridge0    # Monitor traffic on the bridge
ping 10.0.0.10        # Test connectivity to jail

# In the jail
ifconfig              # Verify jail's interfaces
netstat -rn           # Check routing table
ping 10.0.0.1         # Test connectivity to host

Best Practices

1. Resource Limits: Use rctl to set resource limits for your jails to prevent one jail from consuming all system resources:

myjail {
    # ... other settings ...
    exec.prestart += "rctl -a jail:${name}:memoryuse:deny=2G";
    exec.prestart += "rctl -a jail:${name}:pcpu:deny=20";
}

2. Jail Management Tools: Consider using jail management tools like ezjail, iocage, or bastille which can simplify jail creation and management with VNET support.

3. Automation: Use scripts or configuration management tools to automate jail creation, especially if you’re managing multiple jails.

4. Backup Strategy: Regularly backup your jail configurations and data, as jails often contain critical services.

5. Security Considerations:

   • Keep host and jail systems updated
   • Minimize installed packages in jails
   • Use separate jails for separate services
   • Consider implementing additional security measures like mandatory access control (MAC)

Conclusion

VNET jails provide a powerful virtualization capability in FreeBSD, offering network isolation that approaches that of a full virtual machine while maintaining the efficiency advantages of container-like technologies. While the initial setup requires careful planning, the resulting system provides exceptional flexibility for network service deployment.

By following this guide, you should be able to create and manage jails with independent network stacks, allowing for complex multi-jail deployments with sophisticated network topologies.