How To: Repair Ubuntu on Corrupted eMMC with Fresh eMMC

We recently ran into an issue with one of our ODROID-XU4 units that handles some tasks for our shipping team. We have backups of each system, so it wasn't a huge problem. But this information may help some who have a corrupted system that they need to get back online as quickly as possible.

When booting the XU4, we ran into an issue where the system hung on a statement "Freeing unused kernel memory" about 11 seconds into the boot process. We surmised that this was due to an error in one of the files on the OS partition (not the boot partition).

In order to expedite getting the system back online, here's what we did:

1. Imaged a new eMMC with the most recent image of Ubuntu for the XU4.
2. Inserted the old eMMC (the one with the error) into a RockPro64 running Linux using a Transcend USB3.0 to microSD Adapter and an eMMC adapter. We could have used any SBC, laptop or desktop computer running Linux, but the RockPro64 was already set up and running nearby.
3. Used dmesg at the command line to find the device assigned to the eMMC (in this case, it was /dev/sda1 and /dev/sda2 )
4. Created a mountpoint and mounted it using sudo mkdir /media/sda2 && sudo mount /dev/sda2 /media/sda2
5. Inserted the new eMMC into the same RockPro64 using a USB 3.0 to eMMC Reader/Writer
6. Used dmesg at the command line to find the device assigned to the new eMMC (in this case, it was /dev/sdb1 and /dev/sdb2 )
7. Created a mountpoint and mounted it using sudo mkdir /media/sdb2 && sudo mount /dev/sdb2 /media/sdb2
8. Copied the data from the partition on the old eMMC to the partition on the new eMMC using sudo cp -av /media/sda2/* /media/sdb2/. (Note: The -v on the cp command was just so we could see what files were being copied at any instant. The -a is necessary to preserve file permissions, ownership and timestamps.)

Once the new eMMC was placed into the XU4, it booted right back up and was usable.

A couple caveats:

• If the OS version is significantly different between the old image and the new image, there may be some incompatibilities. We found issues with apt, tzdata (time zone data), and Samba.
• Attempting to fix these issues could lead to the system becoming corrupt.

However, this approach led to minimal downtime and breathing room to come up with an alternate strategy.

This method could also be used with microSD-based systems, or even systems running off hard drives or SSDs.

We are using this as an opportunity to replace the XU4 system in question with an ODROID-N2 system, which should provide a marked increase in performance.