IdeaFarm (tm) Bundle for Microsoft Windows
Key benefit: Prevent loss of archived data due to bit corruption, ransomware, and natural disasters.
Approach: Make it easy to keep up-to-date, verified exact copies, including copies that you store in other secured locations.
What you do: Connect multiple volumes (e.g. external HDD or SSD devices) to a computer, with each volume containing a copy of all of your archives in a folder with a particuar name. Install and run IdeaFarm (tm) Bundle, which includes the IdeaFarm (tm) Archive Healer feature.
What will happen: All copies of each file will be compared. If some copies differ from others, an "authoritative" copy will be selected by applying simple rules, and all copies that don't match will be replaced. This is done safely. No copy is ever deleted. Copies that are rejected are moved out of the archive, into another folder on the same device.
Limitations: The number of archives, number of files in each archive, and size of each file are limited only by available device space. (There must be enough free space to store sorted lists of file names.) The number of volumes that can be healed is limited only by the availability of device letters between 'a' and 'v'. (Device letters 'w'-'z' are reserved.) If Windows sees devices for which no media is present, you must either insert media or use Computer Management to reassign a device letter within 'w'-'z' to that device. On each volume, each archive must be a folder (with any name) within a folder with the name "X:\ideafarm.archives", where X is the Windows device letter. Each archive folder must contain "year folders" with names such as "2023". Each year folder must contain only files; any folders will be ignored. When a new file is added, or an existing file is updated, the addition or replacement must be done in at least two volumes and, for replacement, the "time modified" must be at least 15 minutes newer.
Ease of use: There is no user interface (windows) and nothing to configure. IdeaFarm (tm) Bundle installs as a Windows service that will start automatically and invisibly "in the background" whenever you start your computer. Healing progress is "displayed" by using Windows File Explorer to view the contents of a particular folder. More detailed healing progress information is "traced" within one of IdeaFarm (tm) Bundle's human-readable "trace files", which can be viewed (after copying) using Notepad or any other plain text editor.
Quality: As of this writing (March 9, 2023), this is new software that has passed "in-house" testing but has not yet been tested by any "alpha user". Use at your own risk. An email contact address is provided for support. Healing appears to work aok and produce the correct results, based upon in-house testing, using a 2 TB, 3/4 million file archive healed over seven external HDD volumes, using a laptop. For best results, be sure that your devices receive adequate power and have no bad sectors or file system errors before trying to heal them. (You can use the chkdsk command at a command prompt to verify this for each of your devices.)
Testimonial (part 1): My 2 TB, 3/4 million file archive is very valuable to me because it contains a lifetime of creative work, as well as evidence and court documents showing that criminal court in California is a farce and sham and itself a criminal organization. To ensure that I do not lose the archive, I have kept multiple copies of it on external hard disk drives stored in multiple locations. After noticing corruption in several old image files several years ago, I started healing the archive using a batch file that uses the "fc" (file compare) command. That batch file is the predecessor of the current IdeaFarm (tm) Archive Healer, which I wrote to improve performance and to handle an arbitrary number of devices. Recently, I had twelve devices that each had a copy of the archive. Before running IdeaFarm (tm) Archive Healer on those twelve devices, I was confident that I had twelve good copies of my valuable archive. While IdeaFarm (tm) Archive Healer worked, I inspected the trace file and noticed that exceptions were occurring occasionally on one or two of the devices. Whenever I saw an exception trace, I would stop Healer, remove the offending device, and restart Healer. To my surprise and dismay, this quickly reduced the number of devices to five. Healer healed those five devices with no problem; there weren't any more exceptions and Healer completed its work. So at that point I had five copies that contained exactly the same thing and that I thought were on five devices that were reliable.
Testimonial (part 2): But there was a small discrepancy involving two of those 3/4 million files. I investigated, and discovered that two of those five drives had bad sectors that were the cause of the discrepancies. Further investigation, using chkdsk, revealed that one of those five devices was so damaged that chkdsk could not even run on it. At that point, I had only four good copies out of the 12 that I thought that I had.
Testimonial (part 3): So I took those four drives and used them as sources for running Robocopy, to make seven new copies of my archive, onto seven newly purchased Toshiba canvio external hard drives. It was then that I discovered that three of the four were causing Robocopy to fail, so that I ended up with only one copy rather than seven. At that point, I saw that of the twelve good copies that I had thought that I had, only ONE was actually good enough to preserve my archive reliably.
Testimonial (part 4): I later ran chkdsk successfully on all four of the sources. Chkdsk found no errors, so I conclude that the drives are "weak" in the sense that they exhibit transient failure on read, but there is no corruption in the file system. This is consistent with my assessment that the drives have bad sectors and are "on their last legs".
Testimonial (part 5): On March 9, 2023, I used Robocopy to copy my archive from one of the new devices to the other six. Manual inspection revealed that five of the seven appear, from directory listings, to be identical, and the remaining two have some files missing. I think that this is due to Robocopy being interrupted by hibernation cycles, combined with bios allowing reboot to proceed before the drives are fully online. On March 10, 2023, I am running IdeaFarm (tm) Archive Healer on the seven brand new drives. The results will be presented here in about one week. It will take about one week to heal them, since the healing will only run for about eight hours per day.
Testimonial (part 6: conclusion): This experience taught me several things about using external hard drives to preserve valuable data. First, drives can fail in several "invisible" ways that don't come to light until you need them to work and they don't. Second, twelve copies was just barely enough to preserve my archive. Third, IdeaFarm (tm) Archive Healer is good for revealing early transient failures to identify devices that are likely to fail catastrophically soon.
Price: IdeaFarm (tm) Bundle is a proprietary product offered as "barterware". Anyone may install and use it anonymously; no registration or payment is required. Permission to use this product is conditioned on your acceptance of the User License, including "payment" in the form of computational resources of your computer that will be made available to a global system.
You can use IdeaFarm (tm) Archive Healer to "heal" any kind of digital archive, including:
Instructions for installing and using: To install IdeaFarm (tm) Bundle, click: https://github.com/ideafarm/ipdos
IdeaFarm (tm) Bundle is a product of IdeaFarm (tm) City. For more information on IdeaFarm (tm) City, click "about" above.