RMAN Duplicate - Commvault... Second pair of eyes

[Ruan Linehan]

Hi all,

*11g Ent Edition 11.2.0.4 running on a two node RAC cluster on ASM. *

I need to clone a clients production RAC database to a point in time from
yesterday in order to retrieve some data as it appeared before
manipulation.
The duplicate DB also needs to exist on the same cluster for another reason.

We have a level 0 backup set which only exists on tape from the 1st Dec
with required ARC logs still accessible in flash recovery on disk.
The one caveat is that we are leveraging a Simpana/Commvault MML client for
the duplicate and I have not interacted with Commvault in a while.

I have configured the initial setup for performing the duplicate with the
usual suspects, single auxiliary instance, SQL Net, password files etc.
All of which I believe are configured correctly. The auxiliary instance is
initially configured with OMF/ASM "create" params rather than file name
converts
in order to handle the file placement. E.g.
*.control_files='+DATA','+FRA'
*.db_create_file_dest='+DATA'
*.db_create_online_log_dest_1='+FRA'
*.db_recovery_file_dest='+FRA'
etc.

This is one of the many variations of what I have executed...
$ rman target sys/***@PROD auxiliary sys/***@PROD_DUP
RMAN> run
{
allocate channel ch1 type 'sbt_tape'
PARAMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,BLKSIZE=1048576,ENV=(CvClientName=prodhost,CvInstanceName=Instance001,CvOraSID=PROD1)";
allocate auxiliary channel dev1 type 'sbt_tape'
PARAMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,BLKSIZE=1048576";

duplicate target database to PROD_DUP
until time "to_date('16:00 03-12-2018','hh24:mi dd-mm-yyyy')";
}

The RMAN duplicate kicks off, allocates channels successfully, alters the
auxiliary db name, sets the control file path location within ASM, db
unique name etc as normal.
Shuts down and starts up the auxiliary instance and then commences the
restoration from the backup set.

It is at this point that the restore operation completely fails, and seems
to be fetching back from tape but is unable to write back the set via the
aux channel
or create / stage the file in ASM?

Starting restore at 04-DEC-18
Channel dev1: starting datafile backup set restore
Channel dev1: restoring control file
Channel dev1: reading from backup piece c-4575686922-20181202-01
Channel dev1: ORA-19870 error while restoring backup piece
c-4575686922-20181202-01
ORA-19504: failed to create file "+DATA"
ORA-17502: ksfdcre:4 failed to create file +DATA
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12536: TNS:operation would block

1) Apologies in advance but am I doing something 'stoopid' here, which is
not apparent to me??
Or
2) Is there some configuration or env param within the Commvault config
which is necessary to facilitate the proper interaction?

I have tried multiple variations of incorporating file_name_converts or
even 'set newname' for the restore files but everything is appearing to
throw back the very same error above?

Regards,
Ruan

[Mladen Gogala]

Is your auxiliary configured in such a way that Commvault agent is a
member of ASMADMIN group? If not, use cvpkgchg to put it into that
group. The error message essentially tells you that Commvault cannot
write to +DATA. Also if the ASM disk group names are different between
target and auxiliary, you would probably need db_file_name_convert and
log_file_name_convert parameters. Do you have RMAN catalog? If you do,
you can simplify the procedure considerable, since as of 11R2 it is
possible to duplicate database without connecting to the target
database. If the problem persists, you can also open a case with
Commvault support.

Regards

Post by Ruan Linehan
Hi all,
*11g Ent Edition 11.2.0.4 running on a two node RAC cluster on ASM. *
I need to clone a clients production RAC database to a point in time
from yesterday in order to retrieve some data as it appeared before
manipulation.
The duplicate DB also needs to exist on the same cluster for another reason.
We have a level 0 backup set which only exists on tape from the 1st
Dec with required ARC logs still accessible in flash recovery on disk.
The one caveat is that we are leveraging a Simpana/Commvault MML
client for the duplicate and I have not interacted with Commvault in a
while.
I have configured the initial setup for performing the duplicate with
the usual suspects, single auxiliary instance, SQL Net, password files
etc.
All of which I believe are configured correctly. The auxiliary
instance is initially configured with OMF/ASM "create" params rather
than file name converts
in order to handle the file placement. E.g.
*.control_files='+DATA','+FRA'
*.db_create_file_dest='+DATA'
*.db_create_online_log_dest_1='+FRA'
*.db_recovery_file_dest='+FRA'
etc.
This is one of the many variations of what I have executed...
RMAN> run
{
 allocate channel ch1 type 'sbt_tape'
PARAMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,BLKSIZE=1048576,ENV=(CvClientName=prodhost,CvInstanceName=Instance001,CvOraSID=PROD1)";
 allocate auxiliary channel dev1 type 'sbt_tape'
PARAMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,BLKSIZE=1048576";
 duplicate target database to PROD_DUP
 until time "to_date('16:00 03-12-2018','hh24:mi dd-mm-yyyy')";
}
The RMAN duplicate kicks off, allocates channels successfully, alters
the auxiliary db name, sets the control file path location within ASM,
db unique name etc as normal.
Shuts down and starts up the auxiliary instance and then commences the
restoration from the backup set.
It is at this point that the restore operation completely fails, and
seems to be fetching back from tape but is unable to write back the
set via the aux channel
or create / stage the file in ASM?
Starting restore at 04-DEC-18
Channel dev1: starting datafile backup set restore
Channel dev1: restoring control file
Channel dev1: reading from backup piece c-4575686922-20181202-01
Channel dev1: ORA-19870 error while restoring backup piece
c-4575686922-20181202-01
ORA-19504: failed to create file "+DATA"
ORA-17502: ksfdcre:4 failed to create file +DATA
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12536: TNS:operation would block
1) Apologies in advance but am I doing something 'stoopid' here, which
is not apparent to me??
Or
2) Is there some configuration or env param within the Commvault
config which is necessary to facilitate the proper interaction?
I have tried multiple variations of incorporating file_name_converts
or even 'set newname' for the restore files but everything is
appearing to throw back the very same error above?
Regards,
Ruan

[Ruan Linehan]

Thank you kindly Mladen.

The ASM disk groups are one of the same since the duplicate is taking place
within the same RAC cluster and no recovery catalog is being used.

My concern was that Commvault was interpreting my pointer to '+DATA' as an
absolute filename indicator rather than a logical directory object but the
below...
"Is your auxiliary configured in such a way that Commvault agent is a
member of ASMADMIN group? If not, use cvpkgchg to put it into that group.
The error message essentially tells you that Commvault cannot write to
+DATA."
...seems more plausible in terms of where we are going wrong. I will check
this with the Commvault admins (And a support ticket has been logged).

Thanks very much again,
Ruan

Post by Mladen Gogala
Is your auxiliary configured in such a way that Commvault agent is a
member of ASMADMIN group? If not, use cvpkgchg to put it into that group.
The error message essentially tells you that Commvault cannot write to
+DATA. Also if the ASM disk group names are different between target and
auxiliary, you would probably need db_file_name_convert and
log_file_name_convert parameters. Do you have RMAN catalog? If you do, you
can simplify the procedure considerable, since as of 11R2 it is possible to
duplicate database without connecting to the target database. If the
problem persists, you can also open a case with Commvault support.
Regards
Hi all,
*11g Ent Edition 11.2.0.4 running on a two node RAC cluster on ASM. *
I need to clone a clients production RAC database to a point in time from
yesterday in order to retrieve some data as it appeared before
manipulation.
The duplicate DB also needs to exist on the same cluster for another reason.
We have a level 0 backup set which only exists on tape from the 1st Dec
with required ARC logs still accessible in flash recovery on disk.
The one caveat is that we are leveraging a Simpana/Commvault MML client
for the duplicate and I have not interacted with Commvault in a while.
I have configured the initial setup for performing the duplicate with the
usual suspects, single auxiliary instance, SQL Net, password files etc.
All of which I believe are configured correctly. The auxiliary instance is
initially configured with OMF/ASM "create" params rather than file name
converts
in order to handle the file placement. E.g.
*.control_files='+DATA','+FRA'
*.db_create_file_dest='+DATA'
*.db_create_online_log_dest_1='+FRA'
*.db_recovery_file_dest='+FRA'
etc.
This is one of the many variations of what I have executed...
RMAN> run
{
allocate channel ch1 type 'sbt_tape'
PARAMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,BLKSIZE=1048576,ENV=(CvClientName=prodhost,CvInstanceName=Instance001,CvOraSID=PROD1)";
allocate auxiliary channel dev1 type 'sbt_tape'
PARAMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,BLKSIZE=1048576";
duplicate target database to PROD_DUP
until time "to_date('16:00 03-12-2018','hh24:mi dd-mm-yyyy')";
}
The RMAN duplicate kicks off, allocates channels successfully, alters the
auxiliary db name, sets the control file path location within ASM, db
unique name etc as normal.
Shuts down and starts up the auxiliary instance and then commences the
restoration from the backup set.
It is at this point that the restore operation completely fails, and seems
to be fetching back from tape but is unable to write back the set via the
aux channel
or create / stage the file in ASM?
Starting restore at 04-DEC-18
Channel dev1: starting datafile backup set restore
Channel dev1: restoring control file
Channel dev1: reading from backup piece c-4575686922-20181202-01
Channel dev1: ORA-19870 error while restoring backup piece
c-4575686922-20181202-01
ORA-19504: failed to create file "+DATA"
ORA-17502: ksfdcre:4 failed to create file +DATA
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12536: TNS:operation would block
1) Apologies in advance but am I doing something 'stoopid' here, which is
not apparent to me??
Or
2) Is there some configuration or env param within the Commvault config
which is necessary to facilitate the proper interaction?
I have tried multiple variations of incorporating file_name_converts or
even 'set newname' for the restore files but everything is appearing to
throw back the very same error above?
Regards,
Ruan
--
Mladen Gogala
Database Consultant
Tel: [phone]