# This script builds bash commands that compress files in parallel gztar.py -d /tar/to/decompress #! /usr/bin/python To decompress all files from a tar into a folder named after the tar: To compress all files in a directory into a tar named after the folder: The following python script builds bash commands that recursively compress or decompress a given path. type f | parallel gzip"Īlias gzall+="find. & tar -cvf archive.tar dir/to/compressīelow are my personal terminal aliases: alias gz-="parallel gzip ::: *"Īlias gzall-="find. To compress your current directory into a. To recursively compress or decompress a directory: find. This prevents bottlenecking and improves throughput when compressing files compared to using the standard tar -zxcf command.Ĭonsider the following diagram to visualize why. In the command, I am using parallel to create a queue of gzip -d /dir/file commands that are then run asynchronously across all available cores. In the above benchmarks, we are seeing massive time reductions by leveraging all cores during the compression process.
GNU parallel can then split the input and pipe it into commands in parallel. A job can also be a command that reads from a pipe. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can be a single command or a small script that has to be run for each of the lines in the input. Parallel makes it extremely simple to multiplex terminal “jobs”. GNU Parallel is easily one of my favorite packages and a daily staple when scripting. Using GNU Parallel to Create Archives Faster:
You will see scaling in proportion to your thread count and drive speed. These results are from a near un-bottlenecked environment. How did I achieve such a massive time reduction? parallel ::: gzip & cd. gz.tar is 17x faster than normal! A reduction in compression time from nearly an hour to just 3 minutes. Not 2x faster, not 5x faster, but at its peak.
gz.tar actually more than 15x faster than. It refers to when files are first individually compressed in a directory then the whole directory is archived into a.
#Tar compress multithread archive
The third is a small set of large PCAP files containing bid prices.īelow are timed archive compression results for each scenario and archive type.Ī. The second is a relatively small number of genome sequence files in nested folders.ģ. The first is a large number of tiny CSV files containing stock data.Ģ. Compression Benchmarks: tar.gz VS gz.tar VS. tar can provide massive performance gains. While archiving then compressing a directory may seem like the intuitive sequence, we will show how compressing files before adding them to a. With the advent of multi-core and multi-socket CPU architectures, little unfortunately has been done to leverage the wider number of processors. For decades they have served as the backbone of our data archiving and transfer needs.
#Tar compress multithread zip
zip archive formats are quite ubiquitous and with good reason.
0 kommentar(er)
