How to recover an ext3 volume with an unreadable journal

Posted on June 19, 2009 by Trick

Last Wednesday I came to work to find my workstation had died overnight — and upon reboot, it failed to mount or fsck the root partition.  Unfortunately it seems my disk has seen enough service and was having failed reads:

Jun 19 08:51:39 atlas kernel: [ 1324.948428] sd 3:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Jun 19 08:51:39 atlas kernel: [ 1324.948434] sd 3:0:1:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Jun 19 08:51:39 atlas kernel: [ 1324.948442] Descriptor sense data with sense descriptors (in hex):
Jun 19 08:51:39 atlas kernel: [ 1324.948446]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Jun 19 08:51:39 atlas kernel: [ 1324.948463]         00 03 fa 3f
Jun 19 08:51:39 atlas kernel: [ 1324.948470] sd 3:0:1:0: [sdb] Add. Sense: Unrecovered read error – auto reallocate failed
Jun 19 08:51:39 atlas kernel: [ 1324.948519] ata4: EH complete
Jun 19 08:51:39 atlas kernel: [ 1324.950250] sd 3:0:1:0: [sdb] 156250000 512-byte hardware sectors: (80.0 GB/74.5 GiB)
Jun 19 08:51:39 atlas kernel: [ 1324.950919] sd 3:0:1:0: [sdb] Write Protect is off
Jun 19 08:51:39 atlas kernel: [ 1324.954928] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA

After getting a new disk and the machine back operable, I’ve now plugged it in and trying to recover it.  Mounting fails, as it seems there are read errors in the journal itself:

Jun 19 08:35:39 atlas kernel: [  364.686504] EXT3-fs: INFO: recovery required on readonly filesystem.
Jun 19 08:35:39 atlas kernel: [  364.686508] EXT3-fs: write access will be enabled during recovery.
<errors like above>
Jun 19 08:36:08 atlas kernel: [  393.492868] JBD: recovery failed

It seems somewhat logical that it might be common to have physical failures in the area of the disk where the journal lives.  In my case, it seems the unreadable part(s) of the disk are all within the journal.  I ran debugfs on the volume to find that I could read all sorts of things on the disk — so I just needed to tell it to skip reading the journal and mount the disk anyway.

  1. Before you remove the journal, you need to remove the needs_recovery flag from the volume.  You’d think this is possible with tune2fs, but it doesn’t seem so.  So you do it with debugfs:

    debugfs -w -R “feature ^needs_recovery” /dev/sdb1

  2. Then remove the journal, forcibly:

    tune2fs -f -O ^has_journal /dev/sdb1

  3. Now, go ahead and mount your volume as ext2:

    mount -t ext2 -o ro /dev/sdb /mnt/disk

Voila!  You’ve now nuked your journal, and marked your volume ready for mounting.  It’s possible that it has inconsistencies and needs a fsck, but I mounted anyway and was able to recover everything without failure.

I learned this trick from an Ubuntu forums post at http://ubuntuforums.org/archive/index.php/t-953279.html

Posted in Uncategorized | Tagged | 1 Comment

Gnome terminal 2.26 annoyingly protects running child processes?

Posted on June 2, 2009 by Trick

Has anyone else been as annoyed as I am about this dialog?  Every time I close a terminal which has a running child process, rather than the terminal closing and the child process(es) getting killed, gnome-terminal tries to protect:

Dialog: Close this window?

I’m not unlike any other user: I don’t like unnecessary dialogs.

However as I’ve discovered, there is a fix!  Open up gconf-editor (Configuration Editor, not that you can find it on many menu systems these days) and browse to /apps/gnome-terminal/global and uncheck the confirm_window_close key.

GNOME Configuration Editor: /apps/gnome-terminal/global

Now you may return to killing your numerous terminals the way you’ve always enjoyed.

Posted in Uncategorized | Tagged , | 3 Comments

Firefox Three, Guinness World Records, and bananas.

Posted on June 17, 2008 by Trick

So I guess bananas has nothing to do with it, although all you monkeys will be jealous to know I had one with my dinner, and it was tasty.

However the real fruit of the day is (you guessed it) Firefox Three.  I’m pretty happy to see this release; I’ve used the betas on and off for a while and been impressed how far it’s come.  It’ll still struggle to compete with the simplicity of Epiphany, but that’s just because I’m human.

It should also be noted that the Spread Firefox group has coordinated a campaign to start a new Guinness World Record for “the most software downloaded in 24 hours.”  So far since they opened the download floodgates about 18:16 UTC, and in the past ~ten hours or so, the Spread Firefox Download Day page is reporting over three million downloads, although I’m willing to bet that this is thus far an unofficial number.

There’s been a lot of commentary on the bad choice of starting time, the lack of advertising, some issues with the map not showing all the countries accurately but please spare us.  This is supposed to be a fun thing, so have fun with it!  Three million downloads thus far implies that the masses are quite happy.  You can’t seriously expect the organizers of this to make everyone just oh-so-peachy, so just roll with it!  Go download Firefox Three and enjoy!

Well done to everyone working with Mozilla!

Posted in Uncategorized | Tagged , | 1 Comment

Evolution Crash Detection

Posted on April 28, 2008 by Trick

I’ve recently made the jump to the latest ‘n greatest desktop, GNOME 2.22. Altogether I’m quite happy with the software. Not so happy with all the Gentoo bugs. But that’s another story.

One of my favorite bugs is the crashes that Xorg keeps causing, which has made my computing experience similarly enjoyable to Windows 95 where the computer is no longer logical, and likes to crash in a multitude of ways with no warning.

However onto the subject: Evolution’s new crash detection feature. I won’t debate the merit of this feature. Just the dialog box that I had to squint at over morning tea for a whole minute before I understood what the hell it’s prompting me for:

evolution crash detection

I get what this feature exists for. They’re worried that certain types of emails are crashing Evolution upon viewing, so we have a feature to disable the preview pane in the rare case that this is the issue. I suppose. I’ve personally never once had this issue. But what the hell. I’ll play along.

I’m annoyed by this dialog. When my software makes me think, it makes me unhappy. I don’t like to think:

  • I don’t use the preview pane. Why am I seeing this dialog? There’s nothing to disable!
  • “…appears to have exited unexpectedly…” — how the hell else would Evolution’s data files be in a state indicating crash? Be assertive in your dialog messages, as your users appreciate it! This should read “Evolution exited unexpectedly the last time it was run.”
  • The text seems to indicate that “all preview panes will be hidden.” This tells me that the software will do what it says. But what options am I given? An Ignore and a Recover option. At this point in reading the dialog, I wasn’t expecting to be asked a question. I can make some assumptions about what these buttons do…but…what? I had to ponder about what button to push for a whole (admittedly sleepy) minute.

Bug 530345 filed. Hopefully not too grumpy in my bug report, I really do want this to get fixed. But it made me think so early in the morning so I’m allowed to be grumpy, right?

Posted in Uncategorized | Tagged , , | 1 Comment

Cycling in Buckinghamshire, UK

Posted on April 15, 2008 by Trick

So I got out on Sunday this weekend as well, and rode from High Wycombe to Aylesbury.  It was a good ride!  Proper English weather (rained like piss all day.)

Took a few pictures as well.

Quite nice that most of the National Rail lines here don’t care at all if you bring a bike aboard.  It makes it so easy to get out of London and enjoy!

Posted in Uncategorized | Tagged , , | 1 Comment

Hike in Surrey, UK

Posted on April 14, 2008 by Trick

This past Saturday we went on a bit of a hike around Surrey.  It was quite fun!  We had a small crew from work and a few friends from outside work.  Hop on a train from Victoria station to Holmwood, Dorking and walk to Gomshall, Surrey, and train back from there.  ’twas a mighty good time.  I’ve drawn a quick map of the area (apologies, but I’ve become completely obsessed with drawing maps of where I’ve been!)

Check out a few photos, if ya like.

This hike is out of an excellent book, Time Out Country Walks near London, which I strongly recommend for anyone living in or around London!

Posted in Uncategorized | Tagged , | Comments Off on Hike in Surrey, UK

Cycling in London: Not for the faint of heart!

Posted on April 8, 2008 by Trick

Fortunately for me I’m completely nuts!

So I’ve been in London (yes the actual City of London) for the past week and a half for work, and part of my adventure was to bring my bicycle with in order to try to ride around London and not get run over by any buses. The weather finally held (and I got my courage up) so I went for a ride. Needless to say, I didn’t die (else this bit of writing would be quite magical, no?)

It’s a bit different riding here. Shit, I’d take downtown Chicago at rush hour any day for a leisurely ride compared to here! Here I am, riding on the opposite sided of the road (which means that right turns are the difficult ones, not left turns!), none of the roads run in a straight line and always seem to be taking me away from the direction I’d like to be going, the roads are narrow and all under some sort of construction, the cabs are equally as crazy as Chicago cabs, and oh yeah — the cyclists here are completely insane as well…it makes for one heck of a ride!

However most importantly, I didn’t get eaten by a big red London bus.  Hooray!

I took a few pictures off Blackfriar’s bridge before the sun went down. Mostly just to prove that I was here. Hopefully I’ll be able to get out more now that the weather isn’t total crap.

Cycling in London == scary. However I did find some interesting resources on TFL for cyclists, and I’ve sent off for some maps. We’ll see how good they are!

If you’re a driver in London and were out today, thank you for not hitting me :)

Posted in Uncategorized | Tagged , , | 5 Comments

Hey Chicago! Voice your opinion for the Ordinance for Bicycle Safety 2008!

Posted on March 3, 2008 by Trick

I was just recently made aware that the CBF has been working hard at an amendment for some city ordinances on cycling which will help define some legal lines that drivers & cyclists can count on.  I think it’s quite reasonable from a purely safety perspective; I do not view this as a way for cyclists to get back at drivers or anything sly, but as a way to help guarantee everyone’s safety by outlining reasonable bits of understanding for cyclists and drivers.  If this is passed, this will be a good way to inform drivers of reasonable expectations so they can be better educated on how to interact with cyclists on the road.  This is a win for both drivers and cyclists in my book!  This amendment if passed, will mandate (snipped from CBF email verbatim):

  • Requiring a minimum of three feet of clearance while passing bicyclists
  • Prohibiting a motorist from opening a door into moving traffic, reducing the danger of “dooring”
  • Raising the fines for vehicles parked in bike lanes or marked shared lanes
  • Requiring motorists to yield to oncoming bicyclists when turning left, which prevents a “left hook” crash
  • Prohibiting motorists from turning right in front of a bicyclist, which prevents a “right hook” crash
  • Requiring motorists to exercise due care for bicyclists in addition to pedestrians

It also outlines some fines for doing bad things as outlined here, but I think that’s just to motivate drivers and I’m not concerned with the money values.  This amendment will help with some of the usual concerns that both myself and several others I’ve ridden with have had in Chicago; it can be quite worrysome cycling in the city and this is a good thing!

So, if you’re living in the city, please voice your two cents to your alderman!  From my understanding, not all of them sit on the Traffic Control & Safety Committee who votes on this, but it’s still good to voice your opinion.  The next meeting of the committee is quite soon – Wednesday March 5th at 9 AM.  If you support this amendment, please make yourself heard to make cycling in the city a safer place!

Feel free to check out a copy of the email going around about this.

The actual legislation is online as well.

Posted in Uncategorized | Tagged , | 1 Comment

Unfreakingbelieveable: I broke my bike (again)

Posted on February 16, 2008 by Trick

So I’m in Orange County, CA visiting my sister this weekend, and have gone to some length to get my bike out here. So what happens? I demolished my chain within 60 seconds of our first ride. In all my years, I’ve never twisted a chain. Gah!

The wreckage:

[1] [2]

The attempted repair, using nothing but pliers and a standard screwdriver:

[1][2]

’twas unsuccessful, stupid chain won’t rail around the smaller gears. D’oh!

Off to the bike shop in the morning for parts and supplies. In the mean time, we’re off to the local pub to sample the local Guinness :)

Posted in Uncategorized | Tagged , | 1 Comment

My own Upside-Down-Ternet

Posted on January 27, 2008 by Trick

So after reading about the Upside-Down-Ternet (particularly after seeing it featured on XKCD,) I couldn’t resist. After a few discussions at work, we decided that it would be a bad idea if we actually hijacked the company’s proxy and did this. The thought of 200-some people all calling IT for support help (and support in turn recommending that they all reboot their computers) would be so funny that we all might die of suffocation from laughing so hard. Then again, it might be a good way to go…

Regardless we had to do this. We needed to get someone. So I decided that I’d set up a proxy on my workstation, and have it ready for a devious day.

To prepare, I needed to set up squid and get familiar…it’s been several years since I’ve set up & run a squid proxy.  A quick setup of squid 3 on my router box here at home wasn’t much work at all, and before long I had a running proxy. Hacking in a url redirector script isn’t hard, either. Next step will be to set it running on my box at work!  Let the mischief begin(*evil grin*)

So as a variation on the original, which made me laugh (and I hope makes you laugh,) I give you Swirl-Ternet:

Swirl-Ternet

Hopefully in a week or two I’ll get a few co-workers with this and see how much trouble I can get into!

Posted in Uncategorized | Tagged | 2 Comments