Getting started with ZFS on Linux: starting the f’n thing

I am running Ubuntu 08.10 (yeah, i know, waiting for 09-10) and just got a Rosewill RSV-S8 to replace my Drobo, and want to run ZFS on the disks in it, because ZFS is badass. I thought I had to run OpenSolaris or at least Nexenta (ubuntu on top of the solaris kernel, and yes, it’s real) but it turns out there’s a project called zfs-fuse which lets you run ZFS on Linux. FUSE is the “Filesystem in Userspace” project, which lets you run any filesystem as a regular system user. Fuse is necessary because ZFS licensing prohibits the GPL’d linux kernel from using its code, and normal linux filesystems need to be made a part of the kernel.

Now that the background is out of the way, I thought I’d start to pastedump the warnings I run into along the way, so that the greater googlenet may be able to search for them.

How to rename files in a directory to lowercase

Ever want to take a massive tree of files and rename them all lowercase? (Ever work between case-sensitive and case-insensitive systems? :\) Here’s my first attempt:


find . -type f -and -not -type d -print0 | xargs -0 -n 1 echo | while read line; do target=$(echo $line | tr A-Z a-z); echo mv \"$line\" \"$target\"; done | less


which looks fine, until you realize that the path is made lowercase too, despite only looking at files. So I googled, and found a StackOverflow page: http://stackoverflow.com/questions/152514/how-to-rename-all-folders-and-files-to-lowercase-on-linux with this bit:


for SRC in `find my_root_dir -depth`
do
DST=`dirname "${SRC}"`/`basename "${SRC}" | tr '[A-Z]' '[a-z]'`
if [ "${SRC}" != "${DST}" ]
then
[ ! -e "${DST}" ] && mv -T "${SRC}" "${DST}" || echo "${SRC} was not renamed"
fi
done


which isn’t careful about spaces in the filename. But otherwise, it’s on to something with dirname and basename. Revisit mine!


find . -type f -and -not -type d -print0 | xargs -0 -n 1 echo | while read line; do target=$(basename "$line" | tr A-Z a-z); target=$(dirname "$line")/$target; mv "$line" "$target"; done


Mercurial – SVN hg convert un-fubar script

Another nerd post! Turns out I’m using my blog as more of a script repository. Oh well!

If you’ve ever imported an svn repository into hg, then you may have discovered you need to keep the *exact* repo around… unless you know some magic! (thanks bos!)

The secret comes in knowing that hg convert uses a hidden, unversioned file to do its thang, .hg/shamap. The bigger secret is revealed when you run hg tip --debug – the same information is versioned, you just need to squeeze that metadata format back into a file. Because this involves string manipulation, and because I’m old school like that, I used perl to do this.

poppy@host:~/project$ hg log –debug | perl -e ‘my (@x,$l); while(){if(/extra: convert_revision=(.*)/){$l=$1.” $l”;push(@x,$l);}if(/changeset: \d+:(.*)/){$l=$1} }; print join(“\n”,reverse(@x)).”\n”;’ > .hg/new-shamap

poppy@host:~/project$ mv .hg/shamap .hg/old-shamap # if you have one…

poppy@host:~/project$ mv .hg/new-shamap .hg/shamap

BAM! Now you can clone and re-sync your code anywhere, and push it around however you like. It may be dirty but it’s expedient and works for me! Let me know if it works for you.

My little Arduino! Isn’t he cute?

I’m not sure it’s a he. But I made a thing! With a light! I am so excited, I wanted to share my simple code. The tweet:

I made the first “Getting Started with Arduino” example into the “breathing LED” on sleeping apple laptops. I like this little thing.

and the code:

Drobo and my typical luck

I’m not gonna lie; this is pretty dry and mostly here for google fodder. Still, this might tickle your nerd-bone.

Not-so-recently I bought a Drobo for storage. I was tired of fake on-motherboard RAID, and I didn’t want to spring for a 3ware RAID card and the associated learning curve. I heard about the Drobo from a friend, and got it and the DroboShare. After having a poor experience with the DroboShare (I wrote a review on NewEgg but the short version: it does less than a linux server) I decided I’d get rid of my existing file server and use my desktop and the Drobo as the server.

Eventually. I was about to move, had projects at work that needed love, and had some planned / unplanned time off. I did the rsync when I had the DroboShare, and didn’t get to play again on my linux box until recently.

I re-ran the rsync to get it back up-to-date. I noticed the rsync had file errors – all of the filenames with non-ascii characters (a not-insignificant amount of my music) always showed up in the transfer list. That was bad news, but I blamed the DroboShare. Who knows how it formatted the Drobo disk pack? Charset could be some obscure ext3 option, and I didn’t care to invest the time to find out. I reformatted using droboview in the linux drobo-utils package. For good measure, I set the LUN size to 4 (meaning the max size would be 4 TB, instead of segmenting into many virtual 2TB disks).

I mounted with errors=remount-ro and ran an rsync. After a couple dozen GB, I started to see “transfer failed – target is read only” and had a looksee in dmesg. Here’s what I got:


[131174.845284] kjournald starting. Commit interval 5 seconds
[131174.817576] EXT3 FS on sdb1, internal journal
[131174.817778] EXT3-fs: mounted filesystem with ordered data mode.
[136948.130539] journal_bmap: journal block not found at offset 12 on sdb1
[136948.130545] Aborting journal on device sdb1.
[136948.140801] EXT3-fs error (device sdb1) in ext3_new_blocks: Journal has aborted
[136948.140895] Remounting filesystem read-only
[136948.143414] __journal_remove_journal_head: freeing b_committed_data
[136948.143438] __journal_remove_journal_head: freeing b_committed_data
[136948.143454] __journal_remove_journal_head: freeing b_committed_data
[136952.123413] __journal_remove_journal_head: freeing b_committed_data
[136952.123672] __journal_remove_journal_head: freeing b_committed_data
[136952.123755] __journal_remove_journal_head: freeing b_committed_data
[136952.123831] __journal_remove_journal_head: freeing b_committed_data
[136952.123906] __journal_remove_journal_head: freeing b_committed_data
[136952.123975] __journal_remove_journal_head: freeing b_committed_data
[136952.123992] __journal_remove_journal_head: freeing b_committed_data
[136952.124006] __journal_remove_journal_head: freeing b_committed_data
[136952.124016] __journal_remove_journal_head: freeing b_committed_data
[136952.124030] __journal_remove_journal_head: freeing b_committed_data
[136952.124043] __journal_remove_journal_head: freeing b_committed_data
[136952.124191] __journal_remove_journal_head: freeing b_committed_data
[136952.124208] __journal_remove_journal_head: freeing b_committed_data
[136952.124280] __journal_remove_journal_head: freeing b_committed_data
[136952.124335] __journal_remove_journal_head: freeing b_committed_data
[136952.124345] __journal_remove_journal_head: freeing b_frozen_data
[136952.124350] __journal_remove_journal_head: freeing b_frozen_data
[136952.124354] __journal_remove_journal_head: freeing b_frozen_data

and this on re-mount:

mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so


Aww, dammit. What did droboview do wrong? So i googled for EXT3-fs error dmesg drobo and got an authoritative sounding page with command-line instructions. I can get down with the hands-on version so I can scope out where breaks happen. But lo and behold, it worked just fine. The rsync ran fine, and it unmounted and re-mounted correctly. We’ll see how it goes in the long run, but it’s lookin good for now! If you hear nothing, then I heartily endorse this.

Beginner Mind, Expert Mind

There is a great SciAm [article on the cognitive science of what makes experts](http://www.sciam.com/article.cfm?id=the-expert-mind). Some gems:

Dutch psychologist Adriaan de Groot, himself a chess master, confirmed this notion in 1938, when he took advantage of the staging of a great international tournament in Holland to compare average and strong players with the world’s leading grandmasters. One way he did so was to ask the players to describe their thoughts as they examined a position taken from a tournament game. He found that although experts–the class just below master–did analyze considerably more possibilities than the very weak players, there was little further increase in analysis as playing strength rose to the master and grandmaster levels. The better players did not examine more possibilities, only better ones

Chess memory was thus shown to be even more specific than it had seemed, being tuned not merely to the game itself but to typical chess positions. These experiments corroborated earlier studies that had demonstrated convincingly that ability in one area tends not to transfer to another. American psychologist Edward Thorndike first noted this lack of transference over a century ago, when he showed that the study of Latin, for instance, did not improve command of English and that geometric proofs do not teach the use of logic in daily life.

Even so, there were difficulties with chunking theory. It could not fully explain some aspects of memory, such as the ability of experts to perform their feats while being distracted (a favorite tactic in the study of memory). K. Anders Ericsson of Florida State University and Charness argued that there must be some other mechanism that enables experts to employ long-term memory as if it, too, were a scratch pad. Says Ericsson: “The mere demonstration that highly skilled players can play at almost their normal strength under blindfold conditions is almost impossible for chunking theory to explain because you have to know the position, then you have to explore it in your memory.”

Ericsson argues that what matters is not experience per se but “effortful study,” which entails continually tackling challenges that lie just beyond one’s competence. That is why it is possible for enthusiasts to spend tens of thousands of hours playing chess or golf or a musical instrument without ever advancing beyond the amateur level and why a properly trained student can overtake them in a relatively short time. It is interesting to note that time spent playing chess, even in tournaments, appears to contribute less than such study to a player’s progress; the main training value of such games is to point up weaknesses for future study.

Although nobody has yet been able to predict who will become a great expert in any field, a notable experiment has shown the possibility of deliberately creating one. L�szl� Polg�r, an educator in Hungary, homeschooled his three daughters in chess, assigning as much as six hours of work a day, producing one international master and two grandmasters–the strongest chess-playing siblings in history. The youngest Polg�r, 30-year-old Judit, is now ranked 14th in the world.

All a mess!

I was just upgrading WordPress and noticed I haven’t said much in a *while*! Not for lack of anything going on, but it’s mostly been personal stuff and work stuff. Seeing as this is already a meta post, I’ll summarize for those following my stream that care (hello, facebook).

Linux Flash – pissed that Myspace kills it? Try Flash 10 Beta! (plus 64-bit tips)

Oh yeah. So was I. The current version of Adobe Flash for Linux (9.0.124 for the intrepid) breaks as soon as you load the myspace music player. If you’re any kind of music fan, that kinda ruins 50% of new music availability. I got agitated enough to do something about it today, here’s how i installed the new flash 10 beta and fixed the issue.

On becoming a punchline

Please see exhibit A, [Episode 3 of Oddistry](http://oddistry.com/2008/05/08/finally-oddistry-episode-three-maker-faire/), a video blog produced in part by my pal Chadrick who is evidently [Valleywag's mascot](http://valleywag.com/tag/chadrick-baker/). As if I didn’t have enough ego gratification.

Ego Surfing Gratification

So I was just [ego surfing](http://en.wikipedia.org/wiki/Egosurfing) and found [a song written by Niko Donburi that has a line about my work persona, Poppy Linden](http://niko-mysecondlife.blogspot.com/2007/12/public-thank-you-to-ll.html). Woot! Here’s the song…