Nov 082008

Found excellent solution about the nasty problem described in subject and I’m reporting it here for the records.

Short description of the issue I’ve written on OpenFiler Forum:

Hi all!
I’m using Openfiler 2.3 as iSCSI server for 2 ESXi 3.5.
The system is configured with one 80 Gb internal disk where I defined a 72 GB partition as iSCSI volume, plus a 500 MB external disk connected via USB2 defined as another iSCSI volume.
Both are formatted as VMFS and accesses simultaneously on a Gbps network by the 2 ESXs.
Everything works fine at initial setup, until you reboot.
Then what happens is:
1) in boot process you see the following errors:
iSCSI Enterprise Target Software – version 0.4.16
iscsi_trgt: Registered io type fileio
iscsi_trgt: Registered io type blockio
iscsi_trgt: Registered io type nullio
iscsi_trgt: blockio_open_path(166) Can’t open device /dev/raidusb2/ofraid, error -2
iscsi_trgt: blockio_attach(340) Error attaching Lun 1 to Target

2) ESX servers are still seeing the OpenFiler internal disk volume but not anymore the external USB disk.
Check in OF Administration interface, I’ve noticed that the USB2 disk is detected at boot and appears in the “Status” screen but unfortunately it does not appear anymore in “Volumes” screen as a volume group.
Bizarre thing is that if I click on iSCSI target from Volume screen, then I re-click on Volume screen the Volume Group appears back.
Despite this, iSCSI scan made by ESX is still detecting OF Internal Volume Group but not anymore the external USB disk.
Only way to fix it is to delete the volume group and recreate it from scratch, losing all data on it.

There is a solution I’ve found in a long post, therefore I’ve decided to sum it up here in five easy steps:

  1. from /root directory on standard OpenFiler installation, issue the following command (all on one line):
    for file in `dir /lib/modules`; do mkinitrd --allow-missing --preload
     ohci-hcd --preload ehci-hcd --with=usb-storage --with=async_tx
    --with=xor --with=async_xor --with=async_memcpy --with=raid456 -f
    /boot/initrd-$file.img $file; done
  2. Reboot
  3. To retrieve your volume structure you have to create a command (I named it remake_volinfo) on OpenFiler containing the following lines:
    # strip the /mnt lines from fstab as we will be rebuilding them
    grep /mnt /etc/fstab -v > _fstab
    # create the new volumes.xml file
    echo -e "<?xml version=\"1.0\" ?>\n<volumes>" > _volumes.xml
    # find all logical volumes and loop
    for i in `lvdisplay | grep "LV Name" | sed 's/[^\/]*//'`; do
    	fstype=`vol_id $i -t 2> /dev/null`;
    	mntpoint=`echo $i | sed 's/\/dev\//\/mnt\//'`/
    	vgname=`echo $i | cut -d '/' -f3`
    	volid=`echo $i | cut -d '/' -f4`
    	if [ "$fstype" == "" ]; then
    		# assume iscsi since filesystem is unknown
    	if [ $fstype == ext3 ] ; then
    	if [ $fstype == reiserfs ] ; then
    	if [ $fstype != "iscsi" ]; then
    		echo "$i $mntpoint $fstype defaults,usrquota,grpquota$args 0 0" >> _fstab
    		echo "	<volume id=\"$volid\" name=\"$volid\" mountpoint=\"$mntpoint\"
    vg=\"$vgname\" fstype=\"$fstype\" />" >> _volumes.xml
    		echo "Mounting $mntpoint"
    		mkdir -p $mntpoint > /dev/null 2> /dev/null
    		umount $mntpoint 2> /dev/null
    		mount $mntpoint
    		echo "$i - assuming iSCSI"
    		echo "	<volume id=\"$volid\" name=\"$volid\" mountpoint=\"\"
    vg=\"$vgname\" fstype=\"$fstype\" incominguser=\"\" incomingpassword=\"\"
    outgoinguser=\"\" outgoingpassword=\"\" />" >> _volumes.xml
    echo "</volumes>" >> _volumes.xml
    mv -f _fstab /etc/fstab
    mv -f _volumes.xml /opt/openfiler/etc/volumes.xml
    chown openfiler.openfiler /opt/openfiler/etc/volumes.xml
  4. Chmod it to 700
  5. Run command

Once you’ve done this, your USB disk volume groups will survive the reboots and you’ll have back your volume structure.

Oct 272008

OpenFiler is a great system to implement NAS (Network Attached Storage) in a network using general purpose hardware.

On the ‘Net you can find great tutorial about how to setup with VMWare ESX servers, but it contains one important mistake:

When you define the ACL the netmask for every host is /32 or Using /24 or will prevent connection to OpenFiler.

I’ve tested and found it working with VMWare ESXi 3.5 free server.