Admin
Recovering data from non-working flash drives is a frequent request from the friends. Let’s see what can be done with a “killed” flash drive using utilities such as TestDisk and PhotoRec, and then without them – using a Hex editor.
INFO
This article discusses flash drive recovery in Linux environment. In Windows, you can also recover data: there are various utilities and proprietary products (for example, R-Studio), but this is a topic for separate articles.
First of all, connecting the flash drive to a laptop with Linux, I made sure that the hardware part of the device is alive, and that the data is damaged.
The second thing I did, was to make the image.
Safety precautions: making the image
The most important part in data recovery is not to damage even more data with your actions. All the actions described in the article were performed exclusively with the image of a flash drive. You can take the image with the following commands (of course, you need to specify the path to your device):
$ dd if=/dev/sdc of=flash.img bs=512Alternatively, you can use the ddrescue command:
$ ddrescue /dev/sdc flash.img /tmссссссссссиPersonally, I prefer the second method, since ddrescue tries to read the data in several passes, and also (if you gave the command to write a log) to interrupt the reading and resume where it left off. Plus, the utility gives a nice report on how much data was counted and how much was not, and an estimate time until the end of the image capture.
It also makes sense to work with a copy of the image. Suddenly you spoil it, and it’s not a fact that you will be able to remove the image from the flash drive again if it dies due to hardware problems. For partial copies of the image and restoring damaged parts to their initial state, I recommend using the same dd.
$ ddrescue flash.img backup_part.img bs=10M count=1
$ ddrescue backup_part.img flash.img conv=notruncThe notrunc parameter is needed so that dd does not truncate the destination file when the data in the source file runs out.
After removing the image of the flash drive, I looked at the content. What I saw surprised me a little:
In the image, the first 4 MB of data was packed with 0xFF. Is the flash block damaged? Someone’s trying to erase data? An application crashed? Why the area is erased is irrelevant. The main thing is that we have neither a partition table nor a file system structure … Although if you look closely, you can see a pattern. Before us is a sequence of 32-bit numbers increasing by one (in the LittleEndian format ): 0x000a7601, 0x000a7602, 0x000a7603 … Therefore, we most likely had the FAT32 file system on the flash drive.
Well, let’s try to recover the data. Let’s start with the TestDisk utility.
TestDisk
TestDisk is not just a utility, but a powerful data recovery harvester.
TestDisk was developed by Christophe Grenier and is licensed under the GPL v2. This utility is designed primarily for recovering lost partitions on storage media, as well as for recovering the boot sector.
Run TestDisk with the following command:
$ testdisk flash.img
Select the menu items Procced → Intel → Analyze and get the following.
We see that TestDisk did not find the partition table. Expected, because it is overwritten. Let’s try to restore it using the “quick search” of partitions on the disk. Select the item Quick Search.
TestDisk didn’t find anything, but that’s to be expected, because the FAT32 partition is also damaged. TestDisk now offers us to register the partitions manually, but we do not know what was where. So let’s put this utility aside for now. To exit, just press q several times.
Well, then let’s take another invention of the same author – PhotoRec.
PhotoRec
PhotoRec is a program for recovering lost (deleted) files. It was originally developed to recover images from the memory of digital cameras, hence the name – PHOTO RECovery. Over time, it was overgrown with recovery functions for other types of data, but the name remained.
About PhotoRec
PhotoRec looks for known file headers. If there is no fragmentation, which is often the case, it can recover the entire file. PhotoRec recognizes numerous file formats including ZIP, Office, PDF, HTML, JPEG and other graphic file formats. The complete list of formats supported by PhotoRec contains over 390 extensions (about 225 format families).
If the data is not fragmented, the recovered file must be the same size or larger than the original file. In some cases PhotoRec can find out the original file size from the header, so the recovered file is truncated to the required size. However, if the recovered file ends earlier than its header indicates, it is discarded. Some files, such as MP3, are data streams. In this case, PhotoRec analyzes the received data and then stops recovery when the stream ends.
Let’s set this utility on our flash drive image and see what happens.
$ photorec flash.img
We see the already familiar interface, select Proceed → Search → Other, specify the folder where to save (it is better to create it in advance), press the c button. And we are waiting.
As a result, we get several folders with thousands of files in them.
A quick inspection showed that some files were restored: documents, pictures, and sources. But there are no file names, no date of their creation, no folder structure. In addition, as it turned out, there was some kind of documentation on the flash drive in the form of HTML pages with a bunch of small pictures. In this connection, the search for valuable files would take more than one hour …
And, as indicated in the sidebar, the fragmented files either did not recover, or are damaged (truncated).
Apparently, you will have to strain all your strength and manually restore the FAT32 structure.
Fixing FAT32
To restore the FAT32 structure, you need to carefully read the documentation, calculate the values of the key parameters, and then enter them into the FAT32 boot record. Briefly the essence of the FAT32 structure is shown in the figure.
This includes the boot sector, the FSInfo structure, two copies of the FAT tables, and the data area. The boot sector (aka BPB – Boot Parameter Block) contains basic data that describe the characteristics of the partition, and the bootloader code.
- Official FAT Specification (Microsoft, DOC)
- Understanding FAT32 Filesystems (GitHub, PDF)
- Design of the FAT file system (GitHub, PDF)
- Boot sector, FSINFO, FAT table and FAT (Hetman Data Recovery) directory entries
- FAT32 system data structures (“Infopedia”)
For convenient work with the image, we need a Hex editor. Personally, I really like the 010 Editor . It allows you to define structure templates in a C-like language and highlight the structure fields in the editor.
Let’s open our flash drive image in it.
Looking for displacements
To begin with, we need to calculate the addresses where the FAT32 partition and the first copy of the FAT table begin.
First, let’s understand if our first copy of FAT is damaged or both. We know from the documentation that the FAT table starts with the sequence F8 FF FF FF (number 0xFFFFFFF8 in Little Endian). Let’s look for her.
We were lucky – such a signature was found. This means that only the first copy of the FAT table is damaged and we can copy the data from the second table to the first. Of course, it is worth remembering that if the flash drive was disconnected suddenly, then the second copy may not completely coincide with the first (the changes were simply not saved in it). But still, we will be able to recover more data than using PhotoRec alone. At the very least, we will get additional file names, dates of their creation, correct chains for fragmented files, and even a directory structure.
We look at the address – 0x8AE400. This is the address of the beginning of the second copy of the table. Now we need to calculate the length of the table itself. You can, of course, flip through the dump with your hands until we notice the data in the root directory. But there is a simpler option. Since these are two copies, the record with which the chunk of the first copy of the table begins should also be in the second copy. And the difference between them will be the size!
Let’s look for the sequence 01 76 0A 00 that we saw in the beginning when we used hexdump. Options quickly begin to emerge. Stop the search by pressing ESC – we are interested in the first two occurrences.
The first occurrence (address 0x400000) is the first surviving record in the first copy of the FAT. In front of her is a blurred space.
The second occurrence (at 0xB4BC00) is the same entry in the second copy of FAT. In front of it we see the preserved data of the chains.
Let’s calculate the size of the FAT table: 0xB4BC00 – 0x400000 = 0x74BC00 bytes. Therefore, if we subtract this size from the address of the beginning of the second copy of the table, we get the address of the beginning of the first copy: 0x8AE400 – 0x74BC00 = 0x162800.
So, we have the offset of the beginning of the FAT tables. Now we need to find the address of the beginning of the section. According to the data sheets and articles in the sidebar, usually the first copy of the table starts at the 32nd sector. Let me remind you that the sectors are 512 bytes each, which means that the beginning of the section should be located at the address 0x162800 – 32 × 512 = 0x15E800.
By the way, knowing the size of the tables and the offsets of their beginning, we can find the address of the beginning of the root directory.
The root directory offset is 0x15E800 + 32 × 512 + 2 × 0x74BC00 = 0xFFA000. And it begins with a Transcend entry, which is obviously a section mark.
Let’s try to bring this plan to life.
Create a boot record
First, let’s determine the size of the partition.
$ ls -la flash.img -rw-r--r-- 1 user users 15676211200 сен 5 13:36 flash.imgThe partition size is equal to the size of the flash drive minus the partition offset. It turns out 15 676 211 200 – 0x15E800 = 15 674 775 552 bytes.
In order not to take up a lot of disk space with our empty image, we will use the ext2 file system chip – sparse files .
$ dd if=/dev/zero of=test.img bs=1 seek=15674775552 count=0
$ mkfs.vfat -f 2 -F32 -n TEST test.imgLet’s open the file in 010 Editor. And let’s use the Drive template (you may need to install it, see the Templates Repository menu). If a window appears with a warning about long work, we agree to continue the script.
Perfectly! We now have the completed boot sector structures. We ought to transfer them into our image.
Select the structure with the mouse FAT_BOOTSECTOR in the Templates Results window, the data range will be automatically selected, copy it to the clipboard (right-click in the data window and select Copy).
Assembling all together
To build the image, paste the generated boot sector and copy the second copy of the FAT over the first.
Let’s start by inserting the boot sector. We already have it in RAM. Let’s go to the calculated address 0x15E800 and insert data from the buffer.
It turns out that after the sector there are bytes 0xFF, and in the generated image there is more data after this sector.
Disorder. After the first sector comes the FSInfo structure! And at offset 0xC00 there is a copy of the boot sector (in case it is damaged). No, we’d better copy all 32 sectors (0x4000 bytes). By the way, at the same time we made sure that the image generated by mkfs at offset 0x4000 will contain the sequence F8 FF FF FF . After inserting into our image, we ended up at the address 0x162800, which we had previously calculated. So far everything is the same.
Now we need to copy the second copy of the FAT over the first. Select a segment of length 0x74BC00 from the address 0x8AE400, copy it and paste it at 0x162800. To select an area, it is convenient to use the Select Range feature (Ctrl + Shift + A) – just enter the address and size in the fields.
After insertion, we ended up at the beginning of the root directory. So far, everything fits together.
Now we would like to mount this section and see what we can read from it.
Reading data
To mount the file system, run the following commands.
$ mkdir mnt
$ sudo mount -oro,offset=1435648 flash.img mnt/
$ ls mnt/
ls: Unable to access 'mnt / 28-02- ~ 1': I / O error
ls: Unable to access 'mnt / map_n': I / O error 10_10_2016 2019.07.13 28-02- ~ 1 BOOTEX.LOG ......Awesome! The image is mounted, and there are even names of files and directories. But there are also some strange mistakes. Let’s see in more detail (some of the entries in the output are specially omitted).
$ ls -la mnt/ … d????????? ? ? ? ? ? map_n …
Very strange. And viewing the contents of the directories makes it clear that somewhere an error has crept in.
$ ls mnt/some_dir/ ls: Unable to access ‘mnt / some_dir /% PDF-1.4.’ $ ‘\ n’ ‘% ╨’: I / O error ls: Read directory ‘mnt / some_dir /’: I / O error
The file name is the same as the PDF header. It looks like the cluster pointed to by the entry for this directory contains a PDF.
Magic did not happen, you have to figure out what the jamb is. But before that, let’s make our life a little easier and create a partition table. Looking ahead, I will say that the Drive template in the 010 Editor seems to be very clever, but it cannot start parsing from a certain offset, only from the beginning of the file.
We’ll use the fdisk utility to create the partition table. Let’s create one partition starting with sector number 2804. Non-standard offset (by default – 2048), it is possible that there were two partitions on the flash drive, the first one was small and died entirely. But it doesn’t matter anymore. The sector itself is easy to calculate – just divide the beginning of the partition offset by the sector size (0x15E800 / 512 = 2804).
I draw your attention to the fact that fdisk detected the presence of a FAT32 (vfat) partition based on this offset and asked if it should be deleted. Also, do not forget to change the partition type to W95 FAT32 (LBA)(code 0c).
Work on bugs
Finding the error took me about an hour, during which I actively smoked the specifications and checked the values in the structures parsed by the Drive template in the 010 Editor. I will briefly describe my search.
At first I noticed that the root directory is located at 0xFFA800, not 0xFFA000.
I thought the cluster size was incorrectly defined. The mkfs utility created clusters of 16 sectors, and sectors of 512 bytes (see fields BytesPerSector and in the screenshot SectorsPerCluster). First, I tried to play around with the values of these parameters, remounting the image every time.
Loop
Since the image has a partition table, you don’t have to run mount every time with all the parameters ( -oro,offset=1435648 flash.img mnt/). Instead, you can plug in a loop device and ask the kernel to read the partition table from it.
$ sudo losetup -f flash.img
$ sudo partprobe /dev/loop0Or it can be even simpler (read man losetup):
$ sudo losetup -f -P flash.imgThen we can mount and remount the partition as much as we like.
$ sudo mount -oro /dev/loop0p1 mnt/
$ sudo umount mnt/After each edit, you do not need to reconnect the loop device, the offset of the partition in the image does not change.
Nothing good came of it. It only got worse.
After a while, I realized that the second copy of the table also starts in the wrong place.
And then I noticed the field SectorsPerFat32 (in the screenshot). This field describes the size of the FAT in sectors. Its value is 0x3A60, but it should be 0x74BC00 / 512 = 0x3A5E. The difference in two sectors for each copy of the FAT table just gives us 2x2x0x200 = 0x800 bytes of difference between the correct root directory offset and the erroneous one we have.
We correct it (right in the structure window, which is very convenient), save the changes and check.
$ sudo mount -oro /dev/loop0p1 mnt/
$ ls mnt/Fine! The errors are gone! All files and folders are readable. The structure looks like it’s correct.
Having set fsck on the image, we still see a bunch of errors. The first thing that catches your eye, however, is the boot record mismatch.
$ sudo fsck.vfat -n /dev/loop0p1 fsck.fat 4.1
There are differences between boot sector and its backup. This is mostly harmless. Differences: (offset:original/backup) 36:5e/60 Not automatically fixing this. FATs differ but appear to be intact. Using first FAT. ….
Please note that the -nfsck parameter says that no changes are needed.
Well, let’s fix the second copy too. Unfortunately, the 010 Editor template does not parse the second copy of the boot sector, so we will find the required byte ourselves and fix it. It is easy to find it – offset 0x24 from the beginning of the sector, in our case 0x15F424, change 0x60 to 0x5E.
Run fsck again.
Strange, but fsck throws errors and says that the copies of the FAT tables do not match (although we did copy them), and complains about the file lengths. Perhaps previous mounts or something else affected.
I restored the original image of the flash drive from the backup (did you read the safety sidebar?) And went through the steps of copying the tables, the boot sector (with editing the field SectorsPerFat32) and its copy again . Restarting fsck made me happy.
$ sudo fsck.vfat -n /dev/loop0p1 fsck.fat 4.1
Free cluster summary wrong (1911553 vs. really 899251) Auto-correcting. Leaving filesystem unchanged. /dev/loop0p1: 32183 files, 1012304/1911555 clusters
Сonclusions
In conclusion, I think it is worth noting that if the TestDisk and PhotoRec utilities do not give good results on the fly, then you should not despair and you should try to recover the data using your head and straight hands. As a result of all the work, which took about two hours in total, we recovered virtually all data from the “deceased” flash drive, including the directory structure and all metadata.
In addition, I highly recommend making backups as often as possible. Both the images you work with when recovering, and important data on your own flash drives, so that you do not have to recover them.
Comments
Loading…