Strange behaviour with TDE wallet on Exadata

Discussion:

De DBA

2017-06-26 11:14:52 UTC

G'day.

I'm in the process of moving databases from our old ODA to a brand new Exadata, which involves RMAN duplication of TDE encrypted tablespaces. On the Exadata this initially fails with ORA-28365, even though the wallet is available and has auto-login enabled. The wallet for each encrypted database is in the default location (${ORACLE_BASE}/admin/${ORACLE_SID}).

On the ODA this has always worked without a hitch. The (standby) databases can be shutdown and restarted and the wallet is open when needed. We also run a refresh process which involves duplicating a TDE encrypted database in order to mask the data and then use that masked copy as a master for refreshing the DEV and TEST environments from. The wallet is open when the recovery phase starts and the process never errors.

Now to the Exadata. In every situation, whether the recovery phase of an RMAN duplicate or bouncing an existing and working standby database, the wallet remains closed and first access of encrypted data fails, even though the auto-login file is present and readable. In the case of RMAN, the duplication process ends in error. In the case of a standby database, enabling managed recovery fails. In both cases the reported error is

ORA-28365: wallet is not open

The strange thing is that when I query v$encryption_wallet, it reports the wallet as open. After that, recovery succeeds, but in the case of RMAN duplicate only as long as the duplicate was for standby. At no point was the wallet closed, or a password entered to open it. The .sso file exists throughout and is never changed. A normal duplicate fails to open because the database name is not updated in the control file before the duplicate process fails.

In summary:

- "RMAN duplicate .. for standby" or "alter database recover managed standby database" after restart:

RMAN-03002: failure of Duplicate Db command at ...
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-00283: recovery session canceled due to errors
...
ORA-28365: wallet is not open

SQL> select * from v$encryption_wallet;
WRL_TYPE WRL__PARAMETER STATUS
======== ================ ======
file /path/to/wallet OPEN

SQL> alter database recover...

--> recovery succeeds.

- "RMAN duplicate ... to <new_sid>

Oracle instance shut down
...
RMAN-03002: failure of Duplicate Db command at ...
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-00283: recovery session canceled due to errors
...
ORA-28365: wallet is not open

SQL> startup mount
...
ORA-01103 Database name <SOURCE_SID> in control file is not <new_sid>

The OS and database versions are the same on the ODA and the Exadata:

OUL 6.9

Oracle Database 11.2.0.4.170418

Any suggestions?

Cheers,

Tony

RajeevGM

2017-06-26 13:12:24 UTC

Permalink

Tony,

Have you :

a) Looked at bug 12927030
(or any updated ones) ?

b) Tried manually configuring
the auto_login on duplicate/
target server ?

c) Checked if MOS note 389958.1
(or an updated one) is relevant
for your case ?

Rajeev

Post by De DBA
G'day.
I'm in the process of moving databases from our old ODA to a brand new Exadata, which involves RMAN duplication of TDE encrypted tablespaces. On the Exadata this initially fails with ORA-28365, even though the wallet is available and has auto-login enabled. The wallet for each encrypted database is in the default location (${ORACLE_BASE}/admin/${ORACLE_SID}).
On the ODA this has always worked without a hitch. The (standby) databases can be shutdown and restarted and the wallet is open when needed. We also run a refresh process which involves duplicating a TDE encrypted database in order to mask the data and then use that masked copy as a master for refreshing the DEV and TEST environments from. The wallet is open when the recovery phase starts and the process never errors.
Now to the Exadata. In every situation, whether the recovery phase of an RMAN duplicate or bouncing an existing and working standby database, the wallet remains closed and first access of encrypted data fails, even though the auto-login file is present and readable. In the case of RMAN, the duplication process ends in error. In the case of a standby database, enabling managed recovery fails. In both cases the reported error is
ORA-28365: wallet is not open
The strange thing is that when I query v$encryption_wallet, it reports the wallet as open. After that, recovery succeeds, but in the case of RMAN duplicate only as long as the duplicate was for standby. At no point was the wallet closed, or a password entered to open it. The .sso file exists throughout and is never changed. A normal duplicate fails to open because the database name is not updated in the control file before the duplicate process fails.
RMAN-03002: failure of Duplicate Db command at ...
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-00283: recovery session canceled due to errors
...
ORA-28365: wallet is not open
SQL> select * from v$encryption_wallet;
WRL_TYPE WRL__PARAMETER STATUS
======== ================ ======
file /path/to/wallet OPEN
SQL> alter database recover...
--> recovery succeeds.
- "RMAN duplicate ... to <new_sid>
Oracle instance shut down
...
RMAN-03002: failure of Duplicate Db command at ...
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-00283: recovery session canceled due to errors
...
ORA-28365: wallet is not open
SQL> startup mount
...
ORA-01103 Database name <SOURCE_SID> in control file is not <new_sid>
OUL 6.9
Oracle Database 11.2.0.4.170418
Any suggestions?
Cheers,
Tony

De DBA

2017-06-27 11:11:04 UTC

Permalink

Thanks Rajeev,

Sadly none of those work in my case. I've opened an SR with Oracle, but it looks that it has them stumped as well... I wonder whether it's some OS security feature.. SELinux is disabled as far as I am aware, but perhaps something else lurks out there..

Cheers,

Tony

Post by RajeevGM
Tony,
a) Looked at bug 12927030
(or any updated ones) ?
b) Tried manually configuring
the auto_login on duplicate/
target server ?
c) Checked if MOS note 389958.1
(or an updated one) is relevant
for your case ?
Rajeev

Svetoslav Gyurov

2017-06-27 11:50:18 UTC

Permalink

Hi Tony,

Correct me if I'm wrong but if auto-login wallet was in use then the status
would say in v$encryption_wallet would be "AUTOLOGIN" and not "OPEN" ?

Did you try to open the wallet manually with a password ? Or restore the
controlfile only and see if the wallet will open when you mount the
database ?

Is the sqlnet.ora correct ? The databases on ODA could have different path
set for TNS_ADMIN, use srvctl getnev database to check that.

Regards,
Sve

Post by De DBA
Thanks Rajeev,
Sadly none of those work in my case. I've opened an SR with Oracle, but it
looks that it has them stumped as well... I wonder whether it's some OS
security feature.. SELinux is disabled as far as I am aware, but perhaps
something else lurks out there..
Cheers,
Tony
Tony,
a) Looked at bug 12927030
(or any updated ones) ?
b) Tried manually configuring
the auto_login on duplicate/
target server ?
c) Checked if MOS note 389958.1
(or an updated one) is relevant
for your case ?
Rajeev
G'day.
I'm in the process of moving databases from our old ODA to a brand new
Exadata, which involves RMAN duplication of TDE encrypted tablespaces. On
the Exadata this initially fails with ORA-28365, even though the wallet is
available and has auto-login enabled. The wallet for each encrypted
database is in the default location (${ORACLE_BASE}/admin/${ORACLE_SID}).
On the ODA this has always worked without a hitch. The (standby) databases
can be shutdown and restarted and the wallet is open when needed. We also
run a refresh process which involves duplicating a TDE encrypted database
in order to mask the data and then use that masked copy as a master for
refreshing the DEV and TEST environments from. The wallet is open when the
recovery phase starts and the process never errors.
Now to the Exadata. In every situation, whether the recovery phase of an
RMAN duplicate or bouncing an existing and working standby database, the
wallet remains closed and first access of encrypted data fails, even though
the auto-login file is present and readable. In the case of RMAN, the
duplication process ends in error. In the case of a standby database,
enabling managed recovery fails. In both cases the reported error is
ORA-28365: wallet is not open
The strange thing is that when I query v$encryption_wallet, it reports the
wallet as open. After that, recovery succeeds, but in the case of RMAN
duplicate only as long as the duplicate was for standby. At no point was
the wallet closed, or a password entered to open it. The .sso file exists
throughout and is never changed. A normal duplicate fails to open because
the database name is not updated in the control file before the duplicate
process fails.
- "RMAN duplicate .. for standby" or "alter database recover managed
RMAN-03002: failure of Duplicate Db command at ...
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-00283: recovery session canceled due to errors
...
ORA-28365: wallet is not open
SQL> select * from v$encryption_wallet;
WRL_TYPE WRL__PARAMETER STATUS
======== ================ ======
file /path/to/wallet OPEN
SQL> alter database recover...
--> recovery succeeds.
- "RMAN duplicate ... to <new_sid>
Oracle instance shut down
...
RMAN-03002: failure of Duplicate Db command at ...
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-00283: recovery session canceled due to errors
...
ORA-28365: wallet is not open
SQL> startup mount
...
ORA-01103 Database name <SOURCE_SID> in control file is not <new_sid>
OUL 6.9
Oracle Database 11.2.0.4.170418
Any suggestions?
Cheers,
Tony

De DBA

2017-06-27 11:59:53 UTC

Permalink

Hi Sve,

No I don't think so.. in all the years that I've worked with TDE, I've never seen anything else but 'OPEN' for an opened wallet, irrespective of autologin. In the mean time another support engineer has had a look at it and he points to Doc Id. 1400173.1 - the ORACLE_BASE is set wrongly when the database is restarted by RMAN/CRS. I'm checking that now.

Cheers,

Tony

Post by Svetoslav Gyurov
Hi Tony,
Correct me if I'm wrong but if auto-login wallet was in use then the status would say in v$encryption_wallet would be "AUTOLOGIN" and not "OPEN" ?
Did you try to open the wallet manually with a password ? Or restore the controlfile only and see if the wallet will open when you mount the database ?
Is the sqlnet.ora correct ? The databases on ODA could have different path set for TNS_ADMIN, use srvctl getnev database to check that.
Regards,
Sve
Thanks Rajeev,
Sadly none of those work in my case. I've opened an SR with Oracle, but it looks that it has them stumped as well... I wonder whether it's some OS security feature.. SELinux is disabled as far as I am aware, but perhaps something else lurks out there..
Cheers,
Tony