Data Recovery using ddrescue

Disclaimer: although the following post describes a successful recovery process, I don't accept any responsibility for any data loss that occurs from trying to follow these instructions!

A colleague of mine came to me with an external hard disk recently, complaining about missing photos (as well as some other files). The file system in question had a "photos1" directory in the root of the drive, where all the pictures had been meticulously organised into sub-directories. However, it was no longer possible to navigate into the directory; each attempt resulted in a long wait followed by an error message hinting at I/O errors.

Immediately after connecting the drive to my laptop, I was greeted with Fedora's/Gnome's "imminent disk failure" warning (like the title image). This confirmed my suspicions that there was a physical problem with the disk, as well as the following messages in the system log:

May 18 09:43:45 alpha kernel: [233314.587708] sd 47:0:0:0: [sdd] Unhandled sense code
May 18 09:43:45 alpha kernel: [233314.587713] sd 47:0:0:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 18 09:43:45 alpha kernel: [233314.587717] sd 47:0:0:0: [sdd] Sense Key : Medium Error [current]
May 18 09:43:45 alpha kernel: [233314.587721] sd 47:0:0:0: [sdd] Add. Sense: Unrecovered read error
May 18 09:43:45 alpha kernel: [233314.587726] sd 47:0:0:0: [sdd] CDB: Read(10): 28 00 0d 02 f1 80 00 00 08 00
May 18 09:43:45 alpha kernel: [233314.587734] end_request: I/O error, dev sdd, sector 218296704
May 18 09:43:45 alpha kernel: [233314.587741] Buffer I/O error on device sdd1, logical block 27286832
May 18 09:43:47 alpha kernel: [233317.241426] sd 47:0:0:0: [sdd] Unhandled sense code
May 18 09:43:47 alpha kernel: [233317.241434] sd 47:0:0:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 18 09:43:47 alpha kernel: [233317.241438] sd 47:0:0:0: [sdd] Sense Key : Medium Error [current]
May 18 09:43:47 alpha kernel: [233317.241442] sd 47:0:0:0: [sdd] Add. Sense: Unrecovered read error
May 18 09:43:47 alpha kernel: [233317.241446] sd 47:0:0:0: [sdd] CDB: Read(10): 28 00 0d 02 f1 80 00 00 08 00
May 18 09:43:47 alpha kernel: [233317.241454] end_request: I/O error, dev sdd, sector 218296704
May 18 09:43:47 alpha kernel: [233317.241459] Buffer I/O error on device sdd1, logical block 27286832
May 18 09:43:47 alpha ntfs-3g[16779]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error
May 18 09:43:47 alpha ntfs-3g[16779]: Failed to read index block: Input/output error


I took the time to explain to my colleague that the data was most likely retrievable; just because there was a problem accessing the top-level "photos1" directory didn't necessarily mean there would be issues with the directories and files contained within. I mentioned that I myself had faced a similar problem a couple of years back and that I ended up using a professional data recovery firm, Data Recovery Direct. Time had not been on my side in that case; I ending up paying a premium for the company to expedite the recovery process, but it was totally worth it; by the next day I had access to the files I needed off the drive. Incidentally, I think Data Recovery Direct are one of the only companies that I would recommend to others requiring such services, while simultaneously hoping I never have to use them myself again!

Once I had explained the potential cost of recovery to my colleague, he enquired as to whether there was anything I would be able to do instead. I informed him that I could attempt to recover as much data from the disk as possible, but there was a chance I could render the drive completely useless! He accepted the risks and with his consent, I proceeded.

I opted to use an open source recovery tool: ddrescue. This tool attempts to make an image of the disk, partition or file in question by making multiple passes over the source data. It achieves this by writing out a log file so that you are able to perform further passes and/or continue if process is interrupted. Once I had the damaged drive connected to a machine (without mounting the file system) I followed this process:

1. Performed an initial scan across the surface of the disk, ignoring bad, unreadable sectors:
   ddrescue -f -n /dev/sdd1 /mnt/external_storage/tmp/hdd.img /mnt/external_storage/tmp/hdd.log
   
   This took a long time to complete (a day or two) and resulted in a file the size of the partition.
   
   
2. Began a scan that concentrated on the bad sectors, forcing 3 attempts at reading the data:
   ddrescue -d -f -r3 /dev/sdd1 /mnt/external_storage/tmp/hdd.img /mnt/external_storage/hdd.log
   
   Because of the existing log file, ddrescue is able to ignore sectors that it has already retrieved data from and target the damaged areas of the partition. Despite this, however, this command ended up running for several weeks. Towards the end, the "rescued data" count was only increasing by a byte or two a day(!); this was when we decided to kill the process and continue with the next steps in the recovery.
   
   
3. Created a new NTFS partition on another (larger capacity) HDD and wrote the recovered image out to it:
   dd if=/mnt/external_storage/tmp/hdd.img of=/dev/sde1
   
   
4. Attached the drive to a machine running Windows and fixed any outstanding file system errors:
   chkdsk /F G:
   
   

After completing the above steps, I was able to navigate into the previously inaccessible "photos1" directory, much to the joy of my colleague!

Resizing Partitions on a Windows 7/Vista Boot Drive

I used to find it possible to comfortably fit a Windows XP installation onto a volume as small as 20GB; the minimum specs even suggest 1.5GB would be sufficient, but I always allowed for some growth over the life of an install. With the release of Windows Vista, the minimum space required for the OS rose to 55GB; which is substantially larger! However, should you install Windows Vista (or 7, for that matter) onto such a small volume, you will quickly run into capacity issues; as I found with my first installs of the OS.

Now, my usual solution to a system suffering from this issue would be to re-partition the drive and re-install the OS. However, recently I found myself in a situation where I did not have the time to dedicate to a full system rebuild, so I decided to try and resize the existing partitions.

The system in question had two partitions on the drive; the system partition (C:), which was far too small, and the data partition (D:), which had an abundance of free space. This boiled the solution down to two steps:

1. Shrink the second volume (D:) by moving the start of the partition further towards the end of the disk.
2. Extend the first volume (C:) into the newly created space.

Despite Windows having it's own disk management tool, which supports both shrinking and growing volumes, I was unable to use this tool for the entire process for two reasons:

• It only supports shrinking by moving the end of the partition. This would mean that shrinking the 2nd partition (D:) would result in the free space being at the end of the drive.
• The Windows tool can only extend a regular partition into contiguous space; meaning that any space freed at the end of the drive would be useless, given the above limitation.

Fortunately, there are an abundance of 3rd party tools out there that are able to resize and move on partitions, as well as perform other useful operations. Partition Magic (discontinued) and Acronis Disk Director are popular commercial applications, for example. However, being an open-source aficionado, it's hardly appropriate that I use such proprietary solutions! Instead, I opted to use GParted, which is a GUI that sits on top of the parted command line utility. It's available to download as a bootable ISO image; perfect for manipulating a system's boot drive.

Although I was confident that GParted would be able to shrink the second partition, I had read several accounts online that suggest it wasn't the best tool for resizing Windows Vista/7 system partitions. While it appeared possible, it would be necessary to perform a repair of the operating system using the original installation media. This was something I wanted to avoid if possible, so I ended up using a combination of GParted and the Windows Disk Management tool.

The Resize: Step-by-Step

So, once I had downloaded the GParted ISO and burnt to a spare blank CD-ROM, I loaded it into the machine's optical drive and started my hybrid process:

1. (Re)booted from the GParted CD-ROM.
2. Used the GParted tool to configure a shrink/resize operation to be carried out on the second partition (D:) that would move the start of the partition further along the drive.
3. Checked and confirmed the operations required to achieve the desired results, before instructing GParted to carry out the resize.
4. Waited an hour or two until GParted had finished.
5. Rebooted into Windows Vista (and checked to ensure the contents of drive D: were accessible).
6. Used the Windows Disk Management tool to grow the system partition into the newly available space.

To perform a cursory check of the system, I rebooted the machine, forcing a run of chkdsk to ensure file system consistency on both partitions. I was very happy to see the checks run without any issues and the system boot successfully; I saved myself a lot of work re-installing the operating system.

HDD Upgrade

As mentioned in a previous post, my current desktop PC at home was (and still is) using storage devices from the previous system. However, the 160GB Seagate drive (a ST3160023AS) where the Windows 7 installation resided had become extremely tight for space.

I had originally intended to save enough money to purchase a large capacity mechanical hard disk alongside a smaller SSD; installing the OS and a select few apps onto the SSD and everything else on the mechanical drive. However, I had decided that the smallest capacity SSD I would be able to work with would be 128GB because of Windows 7's large install footprint, which would only increase following system updates. At the time I was investigating the upgrade, a good 128GB SSD would have cost me approximately GBP 150.00; an expense I couldn't justify, so I decided to try and hold out.

Eventually, the existing hard disk was filled to capacity, and the cost of an SSD hadn't dropped. In fact, at the time I could barely afford a regular hard disk! Fortunately, I had £40 worth of Amazon vouchers so I began searching for an upgrade and in the end I settled on a Seagate ST31000524AS, which has an unformatted capacity of 1TB. My ideal upgrade would have been a 2TB drive, but I could only spare an additional £20, which limited me to models that operated at 5400 RPM. While my primary concern was additional capacity, I was also conscious of drive performance and that a drive that spins at 7200 RPM would give me much better performance, hence the reason I opted for the 1TB Barracuda (which actually spins at 7200.12 RPM).

Before and after migrating my OS over to the new HDD, I did some tests to compare the performance of each drive. A bit unfair on the older drive as it's a SATA 1.0/1.5Gbps model, compared to the new SATA 3.0/6Gbps drive, but I was interested to see the difference. Each test was conducted 5 times and the median result used for comparison. Before performing the tests, I prepared my machine by running through a pre-flight checklist to help ensure I would achieve optimum performance (thanks to Maximum PC for originally publishing their own checklist for me to work from). The steps that I took were:

1. Turned off the screen saver.
2. Turned off power saving modes; i.e. I switched the OS to "high performance" mode.
3. Disabled the network adaptor/disconnected from the internet.
4. Disabled antivirus apps.
5. Turned off autoupdate.
6. Defragged my hard drive.
7. Disabled System Restore
8. Rebooted.
9. Logged in as "Admin"
10. Waited a few minutes until disk activity had subsided.
11. Ran ProcessIdleTasks; this involved spawning a “DOS box” with administrative privileges and typing: “Rundll32.exe advapi32.dll,ProcessIdleTasks” This forces Windows to perform all of the tasks it would normally do when the system is idle and once the system activity had died down, I was ready to benchmark.

Synthetic Tests
First, I tested the drives using the "Physical Disks" benchmarks from the SiSoftware SANDRA Lite 2011 SP1 Suite (specifically, version 2011.2.17.36). Obviously, this doesn't demonstrate real world usage, but it gives a good idea of roughly how much faster the new drive is.

The newer drive performed over twice as fast as the older one in the throughput tests, which was expected as my motherboard only supports the SATA 2.0 standard, which would limit the throughput of the newer SATA 3.0 HDD. What's particularly interesting about the results is that, despite the newer hard drive achieving faster write access times, the old drive actually posted slightly better read access time results.

Real World Tests
First, file copying; I had both drives divided into two partitions and created a 10GB file which I timed how long it took to copy from one partition to another. The results were impressive:

As you can see from the graph, the new drive copied the file more than 6 times as fast as the old; a vast improvement!

The next test was to see the difference in boot times. What I did here was time how long it took from the computer being powered on to the Windows log in screen being displayed. Not the most accurate of tests, but the difference between the old and new HDDs was much greater than a couple of milliseconds, I consider the margin of error to be acceptable.

Not as impressive as the copying results, with only around 20 second reduction in boot time. However, it's worth noting that approximately 20 seconds of the total boot time is devoted to the motherboard's BIOS POST, with the remaining time allocated to Windows itself booting. Once this is factored in, the new hard drive actually does live up to the expectations raised following the synthetic tests; that the drive should be twice as fast.

Despite being unable to afford an SSD, I am very pleased with the new hard drive. Not only is it much faster than the original drive Windows was running from, but it's so much quieter as well! Depending how the cost of SSDs changes over the next year, I may choose to upgrade another component in my machine instead, most likely the graphics card.

Cloning Windows Vista/7

Disk cloning tools can save hours of effort when upgrading a computer's hard drive, particularly when the drive in question contains an operating system. I recently purchased a new HDD (a Seagate ST31000528AS) for my home desktop to replace the aging (and full!) Seagate used for the Windows 7 installation.

There are plenty of tools available for imaging or cloning disks and partitions, including some excellent open source offerings. CloneZilla, for example, is one that I have used myself to image Windows XP installations for backup purposes. However, searching online revealed that cloning my Windows 7 installation to the new HDD wasn't going to be as easy; it would seem that since Windows Vista, Microsoft has changed the way Windows identifies hard disks and partitions. This prevents the operating system from locating the system partitions where it's installed if you migrate to a new HDD.

Reading various forum and blog posts revealed that cloning a Windows Vista or 7 install to a new drive would complete successfully, but result in a system I would be unable to boot from. Several posts detailed how it was possible to achieve a working system, but either involved editing the registry, using the Windows 7 install media's recovery mode, or a combination of the two.

As I was in a hurry to upgrade the HDD, I didn't want to have to repeat the lengthy cloning process if I was unsuccessful, or end up with an unbootable OS that would need repairing. So I dug further until I found the Seagate Disk Wizard. At first glance, it appears very similar to other cloning/imaging tools; providing a method to create bootable media you can use in the migration process. One major difference though was that you could simply install the software onto the machine in question and start the migration process from your desktop.

I was wary of using the tool at first, as I couldn't see how it could be as efficient as bootable media that carried out the cloning process outside of any installed operating system or software. Searching around I found a forum post that not only confirmed the tool worked, but detailed it's operation too. Essentially, when run from the desktop, Disk Wizard allows you to plan your migration, before rebooting into a small utility that carries out the cloning process.

Suffice to say, before I began this process, I ensured that I had backed up the data on my system to another machine - The specific steps I took were:

1. Connected the new drive to the 3rd SATA port on the motherboard in my PC.


2. Booted into Windows and installed Seagate Disk Wizard.


3. Ran Disk Wizard and started the clone utility.


4. Carefully selected my source and destination drive.


5. Opted to manually define the size I wanted the partitions to be on the new drive (even though I eventually just used the default values used by the partition tool!).


6. Double-checked my settings before finally confirming I wanted to carry out the operation.


7. My computer then restarted, booting into a very lightweight utility that managed the migration.


8. After a long wait, the utility finished, prompting me to have my operating system installation media to hand in order to perform a repair of the installed OS.


9. Powered off the machine.


10. Disconnected the source disk from the 1st SATA port on the motherboard.


11. Disconnected the destination drive from the 3rd port and connected it to the 1st.


12. Powered the machine on. During Windows booting, there was a brief flash of some text relating to Seagate Disk Wizard. I suspect this is where the software updates the registry to use the IDs of the new drive and partition.


13. Once the log in screen appeared, I was able to successfully log into my account and begin using my PC.

Since performing my upgrade in this way I have found some posts that detail successful migrations using open source tools (CloneZilla, GParted) while researching for this post. So, it appears that a seamless migration may be possible using these tools, but if you are after a quick solution for upgrading your system drive, then DiskWizard is a good option.