Data remanence

Many operating systems, file managers, and other software provide a facility where a file is not immediately deleted when the user requests that action. Instead, the file is moved to a holding area (i.e. the "trash"), making it easy for the user to undo a mistake. Similarly, many software products automatically create backup copies of files that are being edited, to allow the user to restore the original version, or to recover from a possible crash (autosave feature). Even when an explicit deleted file retention facility is not provided or when the user does not use it, operating systems do not actually remove the contents of a file when it is deleted unless they are aware that explicit erasure commands are required, like on a solid-state drive. (In such cases, the operating system will issue the Serial ATA TRIM command or the SCSI UNMAP command to let the drive know to no longer maintain the deleted data.) Instead, they simply remove the file's entry from the file system directory because this requires less work and is therefore faster, and the contents of the file—the actual data—remain on the storage medium. The data will remain there until the operating system reuses the space for new data. In some systems, enough filesystem metadata are also left behind to enable easy undeletion by commonly available utility software. Even when undelete has become impossible, the data, until it has been overwritten, can be read by software that reads disk sectors directly. Computer forensics often employs such software. Likewise, reformatting, repartitioning, or reimaging a system is unlikely to write to every area of the disk, though all will cause the disk to appear empty or, in the case of reimaging, empty except for the files present in the image, to most software. Finally, even when the storage media is overwritten, physical properties of the media may permit recovery of the previous contents. In most cases however, this recovery is not possible by just reading from the storage device in the usual way, but requires using laboratory techniques such as disassembling the device and directly accessing/reading from its components. below gives further explanations for causes of data remanence. ==Countermeasures==

There are three levels commonly recognized for eliminating remnant data: Clearing Clearing is the removal of sensitive data from storage devices in such a way that there is assurance that the data may not be reconstructed using normal system functions or software file/data recovery utilities. The data may still be recoverable, but not without special laboratory techniques. Clearing is typically an administrative protection against accidental disclosure within an organization. For example, before a hard drive is re-used within an organization, its contents may be cleared to prevent their accidental disclosure to the next user. Purging Purging or sanitizing is the physical rewrite of sensitive data from a system or storage device done with the specific intent of rendering the data unrecoverable at a later time. Purging, proportional to the sensitivity of the data, is generally done before releasing media beyond control, such as before discarding old media, or moving media to a computer with different security requirements. Destruction The storage media is made unusable for conventional equipment. Effectiveness of destroying the media varies by medium and method. Depending on recording density of the media, and/or the destruction technique, this may leave data recoverable by laboratory methods. Conversely, destruction using appropriate techniques is the most secure method of preventing retrieval. ==Specific methods==

Overwriting A common method used to counter data remanence is to overwrite the storage media with new data. This is often called wiping or shredding a disk or file, by analogy to common methods of destroying print media, although the mechanism bears no similarity to these. Because such a method can often be implemented in software alone, and may be able to selectively target only part of the media, it is a popular, low-cost option for some applications. Overwriting is generally an acceptable method of clearing, as long as the media is writable and not damaged. The simplest overwrite technique writes the same data everywhere—often just a pattern of all zeros. At a minimum, this will prevent the data from being retrieved simply by reading from the media again using standard system functions. The UEFI in modern machines may offer an ATA class disk erase function as well. The ATA-6 standard governs secure erases specifications. Bitlocker is whole disk encryption and illegible without the key. Writing a fresh GPT allows a new file system to be established. Blocks will set empty but LBA read is illegible. New data will be unaffected and work fine. In an attempt to counter more advanced data recovery techniques, specific overwrite patterns and multiple passes have often been prescribed. These may be generic patterns intended to eradicate any trace signatures; an example is the seven-pass pattern , , , , , , , sometimes erroneously attributed to US standard DOD 5220.22-M. One challenge with overwriting is that some areas of the disk may be inaccessible, due to media degradation or other errors. Software overwrite may also be problematic in high-security environments, which require stronger controls on data commingling than can be provided by the software in use. The use of advanced storage technologies may also make file-based overwrite ineffective (see the related discussion below under ). There are specialized machines and software that are capable of doing overwriting. The software can sometimes be a standalone operating system specifically designed for data destruction. There are also machines specifically designed to wipe hard drives to the department of defense specifications DOD 5220.22-M. Writing zero to each block on hard disks and SSDs has the advantage of affording the firmware to deploy spare blocks when bad blocks are identified. Bitlocker has the advantage that data is illegible without the key. Seatools and other tools can erase disks with zero which is typical to revive old consumer class disks but they can wipe server disks albeit slowly. Modern 28TB and larger disks have an enormous number of LBA48 blocks. 40TB and 60TB disks will take proportionately longer times to wipe. Feasibility of recovering overwritten data Peter Gutmann investigated data recovery from nominally overwritten media in the mid-1990s. He suggested magnetic force microscopy may be able to recover such data, and developed specific patterns, for specific drive technologies, designed to counter such. These patterns have come to be known as the Gutmann method. Gutmann's belief in the possibility of data recovery is based on many questionable assumptions and factual errors that indicate a low level of understanding of how hard drives work. Daniel Feenberg, an economist at the private National Bureau of Economic Research, claims that the chances of overwritten data being recovered from a modern hard drive amount to "urban legend". He also points to the "-minute gap" Rose Mary Woods created on a tape of Richard Nixon discussing the Watergate break-in. Erased information in the gap has not been recovered, and Feenberg claims doing so would be an easy task compared to recovery of a modern high density digital signal. As of November 2007, the United States Department of Defense considers overwriting acceptable for clearing magnetic media within the same security area/zone, but not as a sanitization method. Only degaussing or physical destruction is acceptable for the latter. On the other hand, according to the 2014 NIST Special Publication 800-88 Rev. 1 (p. 7): "For storage devices containing magnetic media, a single overwrite pass with a fixed pattern such as binary zeros typically hinders recovery of data even if state of the art laboratory techniques are applied to attempt to retrieve the data." An analysis by Wright et al. of recovery techniques, including magnetic force microscopy, also concludes that a single wipe is all that is required for modern drives. They point out that the long time required for multiple wipes "has created a situation where many organizations ignore the issue [altogether] – resulting in data leaks and loss." Encryption Encrypting data before it is stored on the media may mitigate concerns about data remanence. If the decryption key is strong and carefully controlled, it may effectively make any data on the media unrecoverable. Even if the key is stored on the media, it may prove easier or quicker to overwrite just the key, versus the entire disk. This process is called crypto-shredding. Encryption may be done on a file-by-file basis, or on the whole disk. Cold boot attacks are one of the few possible methods for subverting a full-disk encryption method, as there is no possibility of storing the plain text key in an unencrypted section of the medium. See the section Complications: Data in RAM for further discussion. Other side-channel attacks (such as keyloggers, acquisition of a written note containing the decryption key, or rubber-hose cryptanalysis) may offer a greater chance of success, but do not rely on weaknesses in the cryptographic method employed. As such, their relevance for this article is minor. Media destruction Thorough destruction of the underlying storage media is the most certain way to counter data remanence. However, the process is generally time-consuming, cumbersome, and may require extremely thorough methods, as even a small fragment of the media may contain large amounts of data. Specific destruction techniques include: • Physically breaking the media apart (e.g., by grinding or shredding) • Chemically altering the media into a non-readable, non-reverse-constructible state (e.g., through incineration or exposure to caustic/corrosive chemicals) • Phase transition (e.g., liquefaction or vaporization of a solid disk) • For magnetic media, raising its temperature above the Curie point • For many electric/electronic volatile and non-volatile storage media, exposure to electromagnetic fields greatly exceeding safe operational specifications (e.g., high-voltage electric current or high-amplitude microwave or ionizing radiation) ==Complications==

Inaccessible media areas Storage media may have areas which become inaccessible by normal means. For example, magnetic disks may develop new bad sectors after data has been written, and tapes require inter-record gaps. Modern hard disks often feature reallocation of marginal sectors or tracks, automated in a way that the operating system would not need to work with it. The problem is especially significant in solid-state drives (SSDs) that rely on relatively large relocated bad block tables. Attempts to counter data remanence by overwriting may not be successful in such situations, as data remnants may persist in such nominally inaccessible areas. Advanced storage systems Data storage systems with more sophisticated features may make overwrite ineffective, especially on a per-file basis. For example, journaling file systems increase the integrity of data by recording write operations in multiple locations, and applying transaction-like semantics; on such systems, data remnants may exist in locations "outside" the nominal file storage location. Some file systems also implement copy-on-write or built-in revision control, with the intent that writing to a file never overwrites data in-place. Furthermore, technologies such as RAID and anti-fragmentation techniques may result in file data being written to multiple locations, either by design (for fault tolerance), or as data remnants. Wear leveling can also defeat data erasure, by relocating blocks between the time when they are originally written and the time when they are overwritten. For this reason, some security protocols tailored to operating systems or other software featuring automatic wear leveling recommend conducting a free-space wipe of a given drive and then copying many small, easily identifiable "junk" files or files containing other nonsensitive data to fill as much of that drive as possible, leaving only the amount of free space necessary for satisfactory operation of system hardware and software. As storage and system demands grow, the "junk data" files can be deleted as necessary to free up space; even if the deletion of "junk data" files is not secure, their initial nonsensitivity reduces to near zero the consequences of recovery of data remanent from them. Optical media As optical media are not magnetic, they are not erased by conventional degaussing. Write-once optical media (CD-R, DVD-R, etc.) also cannot be purged by overwriting. Rewritable optical media, such as CD-RW and DVD-RW, may be receptive to overwriting. Methods for successfully sanitizing optical discs include delaminating or abrading the metallic data layer, shredding, incinerating, destructive electrical arcing (as by exposure to microwave energy), and submersion in a polycarbonate solvent (e.g., acetone). Data on solid-state drives Research from the Center for Magnetic Recording and Research, University of California, San Diego has uncovered problems inherent in erasing data stored on solid-state drives (SSDs). Researchers discovered three problems with file storage on SSDs: Solid-state drives, which are flash-based, differ from hard-disk drives in two ways: first, in the way data is stored; and second, in the way the algorithms are used to manage and access that data. These differences can be exploited to recover previously erased data. SSDs maintain a layer of indirection between the logical addresses used by computer systems to access data and the internal addresses that identify physical storage. This layer of indirection hides idiosyncratic media interfaces and enhances SSD performance, reliability, and lifespan (see wear leveling), but it can also produce copies of the data that are invisible to the user and that a sophisticated attacker could recover. For sanitizing entire disks, sanitize commands built into the SSD hardware have been found to be effective when implemented correctly, and software-only techniques for sanitizing entire disks have been found to work most, but not all, of the time. but the process can take some time, typically several minutes. Many older operating systems do not support this feature, and not all combinations of drives and operating systems work. Data in RAM Data remanence has been observed in static random-access memory (SRAM), which is typically considered volatile (i.e., the contents degrade with loss of external power). In one study, data retention was observed even at room temperature. Data remanence has also been observed in dynamic random-access memory (DRAM). Modern DRAM chips have a built-in self-refresh module, as they not only require a power supply to retain data, but must also be periodically refreshed to prevent their data contents from fading away from the capacitors in their integrated circuits. A study found data remanence in DRAM with data retention of seconds to minutes at room temperature and "a full week without refresh when cooled with liquid nitrogen." TRESOR is a kernel patch for Linux specifically intended to prevent cold boot attacks on RAM by ensuring that encryption keys are not accessible from user space and are stored in the CPU rather than system RAM whenever possible. Newer versions of the disk encryption software VeraCrypt can encrypt in-RAM keys and passwords on 64-bit Windows. ==Standards==

; Australia • ASD ISM 2014, Australian Government Information Security Manual, 2014 ; Canada • RCMP B2-002, IT Media Overwrite and Secure Erase Products, May 2009 • Communications Security Establishment Clearing and Declassifying Electronic Data Storage Devices, July 2006 ; New Zealand • GCSB NZISM 2016, New Zealand Information Security Manual v2.5, July 2016 • NZSIS PSM 2009, Protective Security Manual ; United Kingdom • Asset Disposal and Information Security Alliance (ADISA), ADISA IT Asset Disposal Security Standard ; United States • NIST Special Publication 800-88, Guidelines for Media Sanitization, September 2006 • Current editions no longer contain any references to specific sanitization methods. Standards for sanitization are left up to the Cognizant Security Authority. did contain explicit sanitization methods within the Defense Security Service (DSS) Clearing and Sanitization Matrix inserted after Section 8-306. The DSS still provides this matrix and it continues to specify methods. replaced by AR 25-2 (Army Publishing Directorate, 2009) • Air Force AFSSI 8580, Remanence Security, 17 November 2008 • Navy NAVSO P5239-26, Remanence Security, September 1993 • IEEE 2883, IEEE Standard for Sanitizing Storage, August 2022 == See also ==