How much memory does grep run out of?
I have 10GB of RAM on my system and few applications running, so I am really surprised a simple grep runs out of memory. ~/Documents is about 100GB and contains all kinds of files. grep -RI might not have this problem, but I want to search in binary files too.
Is there a way to reclaim memory in grep?
If the files are not sparse and you have a version of GNU grep prior to 2.6, you can use the –mmap option. The lines will be mmapped in memory as opposed to copied there, which means the system can always reclaim the memory by paging out the pages to the file.
What to do if your grep is not sparse?
That converts sequences of NUL characters into one newline character prior to feeding the input to grep. That would cover for cases where the problem is due to sparse files. If the files are not sparse and you have a version of GNU grep prior to 2.6, you can use the –mmap option.
How to load one line at a time in grep?
Use grep -r with newer GNU grep, or use the standard syntax: (however note that the exit status won’t reflect the fact that the pattern is matched or not). grep finds the lines that match the pattern. For that, it has to load one line at a time in memory.
Is there a limit to the number of lines that grep can read?
grep finds the lines that match the pattern. For that, it has to load one line at a time in memory. GNU grep as opposed to many other grep implementations doesn’t have a limit on the size of the lines it reads and supports search in binary files.
How to grep a big file in parallel?
If you have a multicore CPU, I would really recommend GNU parallel. To grep a big file in parallel use: It’s not entirely clear from you question, but other options for grep include: Dropping the -i flag. Setting a max number of matches with the -m flag. Remove the -i option, if you can, case insensitive is quite slow. Replace the . by \\.