There will come a time when you want to take several files and archive them together into a single file - this is where the tar (tape archive) command comes in.
tar can quickly join together multiple files into one larger file, while still preserving meta-data such as Unix permissions. By default tar does not compress files, but it does have a flag that will compress the archive using gzip. We’ll learn about this later in Section 8.7.
The source code we’ll be using for this chapter’s examples is the Rails source code, as mentioned in the Preface.
-c)Let’s start by building a very simple archive, also commonly referred to as a tarball. To do that, we use the -c flag on the tar command to tell it that we want to create a new archive, and give it a list of files we want to archive. In this case we want to archive all of the files in the tools directory.
$ tar -c tools/
The results of our command can be seen in Listing 8.1.
tools/000755 000765 000024 00000000000 12244661361 013337
5ustar00markbatesstaff000000 000000 tools/console000755 000765 000024 00000000255
12244661361 014731 0ustar00markbatesstaff000000 000000 #!/usr/bin/env ruby
require File.expand_path('../../load_paths', __FILE__)
require 'rails/all'
require 'active_support/all'
require 'irb'
require 'irb/completion'
IRB.start
tools/profile000755 000765 000024 00000003451 12244661361 014730
0ustar00markbatesstaff000000 000000 #!/usr/bin/env ruby
# Example:
# tools/profile activesupport/lib/active_support.rb
ENV['NO_RELOAD'] ||= '1'
ENV['RAILS_ENV'] ||= 'development'
require 'benchmark'
module RequireProfiler
private
def require(file, *args) RequireProfiler.profile(file) { super } end
def load(file, *args) RequireProfiler.profile(file) { super } end
@depth, @stats = 0, []
class << self
attr_accessor :depth
attr_accessor :stats
def profile(file)
stats << [file, depth]
self.depth += 1
result = nil
elapsed = Benchmark.realtime { result = yield }
self.depth -= 1
stats.pop if stats.last.first == file
stats << [file, depth, elapsed] if result
result
end
end
end
GC.start
before_rss = `ps -o rss= -p #{Process.pid}`.to_i
path = ARGV.shift
if mode = ARGV.shift
require 'ruby-prof'
RubyProf.measure_mode = RubyProf.const_get(mode.upcase)
RubyProf.start
else
Object.instance_eval { include RequireProfiler }
end
elapsed = Benchmark.realtime { require path }
results = RubyProf.stop if mode
GC.start
after_rss = `ps -o rss= -p #{Process.pid}`.to_i
if mode
if printer = ARGV.shift
RubyProf.const_get("#{printer.to_s.classify}Printer").new(results)
elsif RubyProf.const_defined?(:CallStackPrinter)
File.open("#{File.basename(path, '.rb')}.#{mode}.html", 'w') do |out|
RubyProf::CallStackPrinter.new(results).print(out)
end
else
File.open("#{File.basename(path, '.rb')}.#{mode}.callgrind", 'w') do |out|
RubyProf::CallTreePrinter.new(results).print(out)
end
end
end
RequireProfiler.stats.each do |file, depth, sec|
if sec
puts "%8.1f ms %s%s" % [sec * 1000, ' ' * depth, file]
else
puts "#{' ' * (13 + depth)}#{file}"
end
end
puts "%8.1f ms %d KB RSS" % [elapsed * 1000, after_rss - before_rss]
The first thing that should be pointed out is the apparent “gibberish” preceding each of the files that makes up the archive. This “gibberish” is, in fact, the Unix meta-data of that file; including such things as the permissions, file’s owner, group, etc - all valuable information to be retained.
The second thing that should be noted is that the archive was printed to “standard out” (STDOUT)1 and not saved to a file. We will learn how to save the archive to a file in Section 8.2.
Writing archives to the STDOUT instead of a file can be useful when you want to pipe the output of the archive into another command. We won’t be doing any of that in this chapter, but it is useful to know that tar can create an archive without having to save it to a file.
-f)Being able to write archives to STDOUT does have it uses, but a majority of the time you’ll want to create a new file representing the archive you’ve just created. The -f flag in tar lets us specify a file that we want to save the archive to.
$ tar -cf foo.tar tools/
If the creation of the archive file was successful, the tar command will produce no output.
drwxr-xr-x 28 markbates staff 952 Nov 25 11:08 .
drwxr-xr-x 8 markbates staff 272 Nov 25 09:29 ..
drwxr-xr-x 14 markbates staff 476 Nov 25 11:07 .git
-rw-r--r-- 1 markbates staff 508 Nov 25 09:47 .gitignore
-rw-r--r-- 1 markbates staff 853 Nov 25 09:47 .travis.yml
-rw-r--r-- 1 markbates staff 68 Nov 25 09:47 .yardopts
-rw-r--r-- 1 markbates staff 1203 Nov 25 09:47 CONTRIBUTING.md
-rw-r--r-- 1 markbates staff 2337 Nov 25 09:47 Gemfile
-rw-r--r-- 1 markbates staff 11 Nov 25 09:47 RAILS_VERSION
-rw-r--r-- 1 markbates staff 4118 Nov 25 09:47 README.md
-rw-r--r-- 1 markbates staff 7669 Nov 25 09:47 RELEASING_RAILS.rdoc
-rw-r--r-- 1 markbates staff 2844 Nov 25 09:47 Rakefile
drwxr-xr-x 9 markbates staff 306 Nov 25 09:47 actionmailer
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 actionpack
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 actionview
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 activemodel
drwxr-xr-x 11 markbates staff 374 Nov 25 09:47 activerecord
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 activesupport
drwxr-xr-x 3 markbates staff 102 Nov 25 09:47 ci
-rw-r--r-- 1 markbates staff 5120 Nov 25 11:08 foo.tar
drwxr-xr-x 12 markbates staff 408 Nov 25 09:47 guides
-rw-r--r-- 1 markbates staff 524 Nov 25 09:47 install.rb
-rw-r--r-- 1 markbates staff 51 Nov 25 09:47 load_paths.rb
-rw-r--r-- 1 markbates staff 1159 Nov 25 09:47 rails.gemspec
drwxr-xr-x 12 markbates staff 408 Nov 25 09:47 railties
drwxr-xr-x 3 markbates staff 102 Nov 25 09:47 tasks
drwxr-xr-x 4 markbates staff 136 Nov 25 09:47 tools
-rw-r--r-- 1 markbates staff 159 Nov 25 09:47 version.rb
Listing 8.2 shows the results of querying the current directory. In that list of files we can see the foo.tar file we created.
-v)Using the -v flag we can see which files are being added to our archive. This will output the list of files in the archive after it has been generated.
$ tar -cvf foo.tar tools/
a tools
a tools/console
a tools/profile
Listing 8.3 shows the output of the tar command using the -v flag. We will be using this flag for the rest of the examples in this chapter.
Up until now we have been creating archives by passing in a single directory, however we can pass in any number of directories or files to put in the archive. We just need to pass them as extra arguments at the end of the tar command.
$ tar -cvf foo.tar tools/ tasks/ install.rb rails.gemspec version.rb
a tools
a tools/console
a tools/profile
a tasks
a tasks/release.rb
a install.rb
a rails.gemspec
a version.rb
find (-T)In Chapter 4 we learned about the find tool and how to use it to find specific files matching a pattern. Using the -T flag and the pipe operator (|), we can use find to generate a list of files and then use tar to archive those files.
First let’s build a find query that finds all of the files ending in .gemspec.
$ find . -name \*.gemspec
./actionmailer/actionmailer.gemspec
./actionpack/actionpack.gemspec
./actionview/actionview.gemspec
./activemodel/activemodel.gemspec
./activerecord/activerecord.gemspec
./activesupport/activesupport.gemspec
./rails.gemspec
./railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
./railties/railties.gemspec
Listing 8.5 shows that our find query is working correctly. Now we can use | to pipe the output of the find query to the tar command.
The -T flag will read in a list of file names from another file. In our case we are going to use - in place of the name of a file. This tells tar that it should read in the list of files from the find query we piped in.
$ find . -name \*.gemspec | tar -cvf gemspecs.tar -T -
a ./actionmailer/actionmailer.gemspec
a ./actionpack/actionpack.gemspec
a ./actionview/actionview.gemspec
a ./activemodel/activemodel.gemspec
a ./activerecord/activerecord.gemspec
a ./activesupport/activesupport.gemspec
a ./rails.gemspec
a ./railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
a ./railties/railties.gemspec
In Listing 8.6 we can see that the files from our original find query have been added to the archive.
This demonstrates how incredibly useful these types of programs can be. They are powerful and useful individually, but when you start to combine them together you can quickly make quite complex applications from these simple building blocks.
-t)In Section 8.2.2 we created an archive that contains multiple files, and because we used the -v flag, we saw which files were included in that archive when we created it. If we were to hand this archive over to someone else, how would they know what files were in it? For that matter, how would we know what files were in it should we come back to it at a later date?
To list files in an existing archive we can use the -t flag.
$ tar -tf foo.tar
tools/
tools/console
tools/profile
tasks/
tasks/release.rb
install.rb
rails.gemspec
version.rb
Listing 8.7 lists the files in the archive that we created in Section 8.2.2.
-r)This flag is not compatible with compressed archives, Section 8.7.
In Section 8.2.2 we created an archive that contains several files. Now we’ve realized that we have forgotten a few files that need to be added. We can remedy this situation in two ways. The first way is delete the archive and start again. The second way is to use the -r flag to append the additional files to the archive.
$ tar -rvf foo.tar Gemfile README.md
a Gemfile
a README.md
tools/
tools/console
tools/profile
tasks/
tasks/release.rb
install.rb
rails.gemspec
version.rb
Gemfile
README.md
We can see in Listing 8.8 and Listing 8.9 that by using the -r flag, the files we have specified have been added to the archive.
-u)This flag is not compatible with compressed archives, Section 8.7.
As was the case in Section 8.4 where we needed to add missing files to the archive, sometimes it is necessary to update files that are already in the archive. To do this we can use the -u flag.
$ tar -uvf foo.tar version.rb
a version.rb
We have successfully updated the archive with the new version of the file, and it was much quicker than having to rebuild the entire archive again.
-x)Up until this point we have focused on getting files into archives, but what about getting them out again? To extract all of our files into the current directory we can use the -x flag.
$ tar -xvf foo.tar
x tools/
x tools/console
x tools/profile
x tasks/
x tasks/release.rb
x install.rb
x rails.gemspec
x version.rb
x Gemfile
x README.md
Listing 8.11 shows the files that have been extracted from the archive. These files will also be restored with their original meta-data, permissions, owner, etc…
Extracting the entire archive is the most common use case, however, there will be times when you’ll only want to extract one file from the archive. To extract a specific file(s), we can just list them at the end of the extraction call - much like how we listed them when creating the archive.
$ tar -xvf foo.tar rails.gemspec
x rails.gemspec
Listing 8.12 demonstrates how we are able to extract just the rails.gemspec file from the archive.
-C)Up until this point we have been extracting all of the files to the current directory we are in, but we can easily extract to a different location.
First, let’s create a new folder, named myfolder, that we will extract all of the files to.
$ mkdir myfolder
Using the -C flag we can pass a folder to the tar command as the location we want to extract the file to.
$ tar -xvf foo.tar -C ./myfolder
x tools/
x tools/console
x tools/profile
x tasks/
x tasks/release.rb
x install.rb
x rails.gemspec
x version.rb
x Gemfile
x README.md
Listing 8.13 shows that all of the files have been extracted. We can use the find tool we learned about in Chapter 4 to look at the structure of the myfolder directory.
$ find ./myfolder
./myfolder
./myfolder/Gemfile
./myfolder/install.rb
./myfolder/rails.gemspec
./myfolder/README.md
./myfolder/tasks
./myfolder/tasks/release.rb
./myfolder/tools
./myfolder/tools/console
./myfolder/tools/profile
./myfolder/version.rb
As we can see in Listing 8.14 all of the files were extracted to their appropriate locations based on how they were archived.
gzip (-z)As mentioned at the very beginning of the chapter, tar does not compress archives by default, it just generates a single large file from multiple smaller files. With that said, tar does provide three different compression algorithms to you. The first one, gzip (-z), is the most common, so that is the one we will discuss here. The others are bzip2 (-j) and Unix compress (-Z). bzip2 is considerably slower for compression and decompression, but produces smaller archives, and Unix compress is faster, but produces considerably larger archives.
Let’s start by creating a regular archive. We will archive all of the files in the activerecord directory2.
$ tar -cf ar.tar activerecord/
-rw-r--r-- 1 markbates staff 3311104 Nov 26 14:12 ar.tar
Listing 8.15 shows us that our new archive, ar.tar, is a whopping 3.3Mb. We can use the -z flag to run the archive through gzip and compress it to make it smaller. When we do this it is customary to append .gz to the end of the archive’s file name to let others know that this archive has been compressed3.
$ tar -czf ar.tar.gz activerecord/
-rw-r--r-- 1 markbates staff 3311104 Nov 26 14:12 ar.tar
-rw-r--r-- 1 markbates staff 615262 Nov 26 14:14 ar.tar.gz
In Listing 8.16 we can see that the compressed version of the archive weighs in at only 615Kb compared to the 3.3Mb of the uncompressed archive; a significant reduction in file size.
-z)Extracting files from a compressed archive is not much different than how we learned to extract files in Section 8.6. All we need to do is add the -z flag to the extract command so it first uncompresses the archive and then extracts the files.
$ tar -xvzf ar.tar.gz
x activerecord/
x activerecord/activerecord.gemspec
x activerecord/CHANGELOG.md
x activerecord/examples/
...
x activerecord/lib/active_record/associations/builder/singular_association.rb
x activerecord/examples/.gitignore
x activerecord/examples/performance.rb
x activerecord/examples/simple.rb
Listing 8.17 shows a truncated list of the files that were extracted.
In this chapter we learned how to use tar to:
The tar command is a powerful and easy to master tool, as long as you can remember the few flags and options we talked about in this chapter.
-v flag on your own to see the output. ↑-v flag on your own to see the output. ↑