Contents
Why does a job fail with no space left on device but df says otherwise?
/tmp is usually the operating system’s (OS) temporary output directory, accessed by OS users, and /tmp is typically small and on a single disk. So when Spark runs lots of jobs, long jobs, or complex jobs, /tmp can fill up quickly, forcing Spark to throw “No space left on device” exceptions.
Why does df show greater disk usage than du?
There are times when the du and df commands can show different disk usage stats. The usual reason for this is that a running process still has a file that was removed open. The solution to this issue is to reload or restart that service so that it closes that file handle. …
Why df and du show different results?
du is used to estimate file space usage—space used under a particular directory or files on a file system. So after deleting the file, the results of df and du are different until the disk space is released.
What does df in Linux do?
df command in Linux with Examples. The df command (short for disk free), is used to display information related to file systems about total space and available space. If no file name is given, it displays the space available on all currently mounted file systems.
How to check drive space on Linux from the command line
- df – reports the amount of disk space used on a file system.
- du – reports the amount of space used by specific files.
- btrfs – reports the amount of space used by a btrfs file system mount point.
Why is df used?
df (abbreviation for disk free) is a standard Unix command used to display the amount of available disk space for file systems on which the invoking user has appropriate read access. df is typically implemented using the statfs or statvfs system calls.
What’s the difference between DF and free space?
We can do a test, du says you have 10GB free space, while df says 300 MB, can you write a file (or several files) with size of, say, 2 GB? If you can, that means that df is just simply wrong (and there is actually no problem of ‘lost space’). If not, then du is wrong (which will be interesting).
Where does DF-H show used space in spark?
By default, spark uses “/tmp” to save intermediate files. When the job is running, you can tab df -h to see the used space of fs mounted at “/” growing up. When the space of the dev is runned out of, this exception is thrown.
What to do if spark says there is no space left on device?
If you actually do have space left on some device — you can alter this by creating the file SPARK_HOME/conf/spark-defaults.conf and adding the line. Here SPARK_HOME is wherever you root directory for the spark install is. You need to also monitor df -i which shows how many inodes are in use.
How to fix no space left on device in Fedora?
I’m using Fedora with GNOME, and the GNOME Disk Usage Analyzer tool shows me the following. Begin with du. Point it to the base directory on the drive that’s having the problem. Let’s assume it’s the partition with /. It’ll take some time to go through everything. Now, try with df. Add / and the filesystems mounted under it.