0.94.2 volume mount issue

From: Lyn Gerner <no email>
Date: Tue, 5 Nov 2013 12:59:55 -1000

Hi All,

I've been using 0.93.3 with only minor, known operational issues. Decided
it was time to move to 0.94.2 about a week ago. Haven't been operational
since, because of new, incorrect behavior in the way a volume is getting
attached. Actually it's the way that starcluster is trying to mount it
--referring to the wrong /dev/xvd* --when trying to mount the vol.

I have the following specified for the volume I'm telling starcluster to
mount:

[volume jobspoolse1d]
VOLUME_ID = vol-f1a0d380
MOUNT_PATH = /usr/share/jobs/
DEVICE = /dev/sdq

Here is the excerpt of the failing vol attachment:

>>> Configuring cluster...
>>> Attaching volume vol-f1a0d380 to master node on /dev/sdq ...
>>> Waiting for vol-f1a0d380 to transition to: attached...
>>> Running plugin starcluster.clustersetup.DefaultClusterSetup
>>> Configuring hostnames...
2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100%
*** WARNING - Cannot find device /dev/xvdq for volume vol-f1a0d380
*** WARNING - Not mounting vol-f1a0d380 on /usr/share/jobs/
*** WARNING - This usually means there was a problem attaching the EBS
volume to the master node

Here is how the master node looks:

[root_at_master ~]# ls -l /dev/sd*
lrwxrwxrwx 1 root root 4 Nov 5 22:31 /dev/sda -> xvde
lrwxrwxrwx 1 root root 5 Nov 5 22:31 /dev/sda1 -> xvde1
lrwxrwxrwx 1 root root 5 Nov 5 22:31 /dev/sda2 -> xvde2
lrwxrwxrwx 1 root root 5 Nov 5 22:31 /dev/sda3 -> xvde3
lrwxrwxrwx 1 root root 5 Nov 5 22:31 /dev/sdaa -> xvdaa
lrwxrwxrwx 1 root root 4 Nov 5 22:32 /dev/sdq -> xvdu
[root_at_master ~]# mount
/dev/xvde1 on / type ext4 (rw,relatime)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
[root_at_master ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvde1 4909772 3344868 1315500 72% /
tmpfs 847740 0 847740 0% /dev/shm

So, the vol is getting attached at /dev/sdq, but this actually points to
/dev/xvdu, instead of the /dev/xvdq that starcluster is looking for. This
"multiletter offset" (sdq -> xvdu) in the device mapping was present in
.93.3 also, but was handled therein correctly.

I can manually mount /dev/xvdu and confirm that it's got the right stuff in
it:

[root_at_master ~]# mount /dev/xvdu /usr/share/jobs
[root_at_master ~]# ls -l /usr/share/jobs
total 32
drwxrwsr-x 2 prod prod 4096 Jun 20 19:31 ami-bridge
drwxrwsr-x 3 prod prod 4096 Mar 20 2013 common
drwxrwsr-x 10 prod prod 4096 Jun 19 19:19 demo
drwxr-sr-x 3 root prod 4096 Mar 21 2013 etc
drwxrwsr-x 8 prod prod 4096 Feb 20 2013 internal
drwxrwsr-x 10 prod prod 4096 Jun 19 19:19 live
drwxrwsr-x 2 prod prod 4096 Jun 20 18:58 log
drwxrwsr-x 10 prod prod 4096 Jun 19 19:20 test

Just what I wanted to see there.

So, I'll appreciate any support regarding a fix for this, whether it's a
patch or a workaround.

Thanks much,
Lyn
Received on Tue Nov 05 2013 - 17:59:58 EST