Non-interactive multivolume restore

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Non-interactive multivolume restore

George Mitchell
Historically, the restore program assumed that the volumes of a
multivolume dump were on separate physical reels of tape, and that a
human would have to load each tape to complete the restore.  But now
I have multivolume dumps that are simply separate online files from
which I would like to restore without having to type each name in
one at a time.  However, you can give only one -f option on the
command line, and additional file names are assumed to specify the
files you want to restore, as opposed to additional dump volumes.

I thought perhaps 'cat dumpvols... | restore -x -f -', but that
gets confused at the beginning of the second volume.

What's the right way to do this?                          -- George


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

Warner Losh
On Sun, Feb 3, 2019, 8:23 PM George Mitchell <[hidden email] wrote:

> Historically, the restore program assumed that the volumes of a
> multivolume dump were on separate physical reels of tape, and that a
> human would have to load each tape to complete the restore.  But now
> I have multivolume dumps that are simply separate online files from
> which I would like to restore without having to type each name in
> one at a time.  However, you can give only one -f option on the
> command line, and additional file names are assumed to specify the
> files you want to restore, as opposed to additional dump volumes.
>
> I thought perhaps 'cat dumpvols... | restore -x -f -', but that
> gets confused at the beginning of the second volume.
>
> What's the right way to do this?
>

Multiple -f args? Maybe a printf format to clue restore into how to
construct the names? Maybe also some pipe command with a similar %
specification to allow restore from compressed files..

Warner

>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

O'Connor, Daniel-2


> On 4 Feb 2019, at 13:59, Warner Losh <[hidden email]> wrote:
> Multiple -f args? Maybe a printf format to clue restore into how to
> construct the names? Maybe also some pipe command with a similar %
> specification to allow restore from compressed files..

OP says you can only specify -f once.

What about using -P? That can be scripted so you could do something where a script checks a file for the current filename it should cat.

     -P pipecommand
             Use popen(3) to execute the sh(1) script string defined by
             pipecommand as the input for every volume in the backup.  This
             child pipeline's stdout (/dev/fd/1) is redirected to the restore
             input stream, and the environment variable RESTORE_VOLUME is set
             to the current volume number being read.  The pipecommand script
             is started each time a volume is loaded, as if it were a tape
             drive.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
 -- Andrew Tanenbaum


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

Rodney W. Grimes-4
In reply to this post by Warner Losh
> On Sun, Feb 3, 2019, 8:23 PM George Mitchell <[hidden email] wrote:
>
> > Historically, the restore program assumed that the volumes of a
> > multivolume dump were on separate physical reels of tape, and that a
> > human would have to load each tape to complete the restore.  But now
> > I have multivolume dumps that are simply separate online files from
> > which I would like to restore without having to type each name in
> > one at a time.  However, you can give only one -f option on the
> > command line, and additional file names are assumed to specify the
> > files you want to restore, as opposed to additional dump volumes.
> >
> > I thought perhaps 'cat dumpvols... | restore -x -f -', but that
> > gets confused at the beginning of the second volume.
> >
> > What's the right way to do this?
> >
>
> Multiple -f args? Maybe a printf format to clue restore into how to
> construct the names? Maybe also some pipe command with a similar %
> specification to allow restore from compressed files..

Doesnt restore read these file names from stdin?
#!/bin/sh
restore rf foo.first <<!
foo.second
foo.third
foo.forth
!

--
Rod Grimes                                                 [hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

Wojciech Puchar-8
In reply to this post by George Mitchell
> files you want to restore, as opposed to additional dump volumes.
>
> I thought perhaps 'cat dumpvols... | restore -x -f -', but that
> gets confused at the beginning of the second volume.
>
> What's the right way to do this?                          -- George
>
>
look at -P option.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

Diane Bruce
In reply to this post by George Mitchell
On Sun, Feb 03, 2019 at 09:50:43PM -0500, George Mitchell wrote:

> Historically, the restore program assumed that the volumes of a
> multivolume dump were on separate physical reels of tape, and that a
> human would have to load each tape to complete the restore.  But now
> I have multivolume dumps that are simply separate online files from
> which I would like to restore without having to type each name in
> one at a time.  However, you can give only one -f option on the
> command line, and additional file names are assumed to specify the
> files you want to restore, as opposed to additional dump volumes.
>
> I thought perhaps 'cat dumpvols... | restore -x -f -', but that
> gets confused at the beginning of the second volume.
>
> What's the right way to do this?                          -- George
>

Would xargs work for this?



--
- [hidden email] [hidden email] http://artemis.db.net/~db
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

Poul-Henning Kamp
In reply to this post by Wojciech Puchar-8
--------
In message <[hidden email]>, Wojciech Puchar writes:
>> files you want to restore, as opposed to additional dump volumes.
>>
>> I thought perhaps 'cat dumpvols... | restore -x -f -', but that
>> gets confused at the beginning of the second volume.
>>
>> What's the right way to do this?                          -- George
>>
>>
>look at -P option.

I was just about to say:  -P is there exactly for tape libraries etc.

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[hidden email]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

George Mitchell
On 2/4/19 8:51 AM, Poul-Henning Kamp wrote:

> --------
> In message <[hidden email]>, Wojciech Puchar writes:
>>> files you want to restore, as opposed to additional dump volumes.
>>>
>>> I thought perhaps 'cat dumpvols... | restore -x -f -', but that
>>> gets confused at the beginning of the second volume.
>>>
>>> What's the right way to do this?                          -- George
>>>
>>>
>> look at -P option.
>
> I was just about to say:  -P is there exactly for tape libraries etc.
>
Thanks to all who pointed this out to me!  I think I will propose a
documentation patch to give an example of this usage, as I had to go
poking into the code to see exactly how this works.  But first I'm
verifying that my new understanding is correct.            -- George


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

George Mitchell
On 2/4/19 8:20 PM, George Mitchell wrote:

> On 2/4/19 8:51 AM, Poul-Henning Kamp wrote:
>> --------
>> In message <[hidden email]>, Wojciech Puchar writes:
>>>> files you want to restore, as opposed to additional dump volumes.
>>>>
>>>> I thought perhaps 'cat dumpvols... | restore -x -f -', but that
>>>> gets confused at the beginning of the second volume.
>>>>
>>>> What's the right way to do this?                          -- George
>>>>
>>>>
>>> look at -P option.
>>
>> I was just about to say:  -P is there exactly for tape libraries etc.
>>
>
> Thanks to all who pointed this out to me!  I think I will propose a
> documentation patch to give an example of this usage, as I had to go
> poking into the code to see exactly how this works.  But first I'm
> verifying that my new understanding is correct.            -- George
>
This does NOT work the way I expected.  I created links to the volumes
of my dump named dump1, dump2, ... dumpN.  Then I ran this command:

restore -x -s 1 -P 'cat ../dump$RESTORE_VOLUME'

Surprise number one: despite the "-s 1" option, the program asked me:

If you are extracting just a few files, start with the last volume
and work towards the first; restore can quickly skip tapes that
have no further files to extract. Otherwise, begin with volume 1.
Specify next volume #:

So I answered "1" and got my next surprise.  Before continuing from
volume 1 to volume 2, this appeared:

Mount tape volume 2
Enter ``none'' if there are no more tapes
otherwise enter tape name (default: cat ../dump$RESTORE_VOLUME)

I pressed enter.  A similar dialog transpired at the end of each
succeeding volume of the dump.  The good news is that the restore
succeeded and I had a lot less typing to do.  But this doesn't meet
my definition of non-interactive.  What did I miss this time?


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Non-interactive multivolume restore

George Mitchell
Apparently people don't often have to restore multivolume dumps.  (I
rarely do it myself except that I'm in the middle of compiling an index
of a plethora of ancient dumps before purging them.)  Since it requires
a lot of interaction (as least as far as I have been able to determine),
I've writtien this silly python program to help out.  It relies on
misc/py-pexpect, which happily compiles in a py36 version that doesn't
run, so this program specifies python2.7:
---------------------------------------------------------------------
#!/usr/bin/env python2.7

from __future__ import print_function
import pexpect
import sys
import os

if len(sys.argv )< 2 or (len(sys.argv) == 2 and sys.argv[1] == '-h'):
    print('''
Non-interactively restore from a multivolume dump.  You can use a
glob to specify the dump volume if the glob expands to the files in
the correct order.  Creates symbolic links to your dumps in $TMPDIR
(or /tmp by default) and runs 'restore' with the -P option.  Generates
responses as needed to keep the restore program running.
''')
    print('You did not specify any dump volumes.')
    sys.exit(1)

tmpdir = os.getenv('TMPDIR') or '/tmp'
for i, volume in enumerate(sys.argv[1:]):
    try:
        os.remove(tmpdir + '/dumpvolume' + str(i + 1))
    except:
        pass
    os.symlink(sys.argv[i + 1], tmpdir + '/dumpvolume' + str(i + 1))
child = pexpect.spawn("/sbin/restore -x -P 'cat " + tmpdir +
                      "/dumpvolume$RESTORE_VOLUME'")
child.logfile = sys.stdout
while True:
    resp = child.expect(['Specify next volume #: ', 'default: ',
                         "set owner/mode for '.'. .yn. ", pexpect.EOF],
                        timeout=600)
    if resp == 0:
        child.sendline('1')
    elif resp == 1:
        child.sendline('')
    elif resp == 2:
        child.sendline('y')
    else:
        break

for i in len(sys.argv[1:]):
    os.delete(tmpdir + '/dumpvolume' + str(i + 1))


signature.asc (849 bytes) Download Attachment