Problem with Oracle ASM/Grid 18c and 12c/11g Databases....

[Mark J. Bobak]

Hi all,

I have an SR open on this, but progress seems slow, so I thought I'd put
something here....

Environment:
AWS EC2 instance, m5d.2xlarge (8 VCPUs/32GB RAM)
Oracle 18c (18.3.0) ASM/Grid w/ ASMFD configured, single instance (SIHA),
not RAC
Three Oracle Databases: 2x 12.1.0.2 and 1x 11.2.0.4

Everything works fine, except I can't start/stop/config any DB (12c or 11g)
via srvcrl.

If I start/stop w/ SQL*Plus, everything works great.

If I try start with srvctl, I get errors:
[***@miscr2 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM
The Oracle base has been set to /u01/app
[***@miscr2 ~]$ srvctl start database -d BMPR2PRD
PRCR-1079 : Failed to start resource ora.bmpr2prd.db
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
following error:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".

ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0

CRS-2674: Start of 'ora.bmpr2prd.db' on 'miscr2' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0

If I look into /u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc
as instructed, among many other things, I see:
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
following error:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".

2018-09-21 17:41:51.511 : AGFW:34227968: {0:0:21154} sending status msg
[CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
following error:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
] for start for resource: ora.bmpr2prd.db 1 1

Which doesn't really tell me much. Oracle Support had me turn on srvm
tracing and capture more data. When I did that, I discovered this:
[main] [ 2018-09-21 10:25:56.395 EDT ]
[UnixSystem.dorunRemoteExecCmd:4172] Final unix SSH command:
/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r^M
[Thread-14] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[main] [ 2018-09-21 10:25:56.395 EDT ] [RuntimeExec.runCommand:294]
runCommand: Waiting for the process^M
[Thread-13] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>Usage: /u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -a | -d |
-o | -b | -g | -k^M

So, according to the trace, it runs the 'osdbagrp' command several times
with different command line arguments. -a, -d, -o, -b, -g, and -k all work
and (appear to) produce valid results.
When it runs 'osdbagrp -r', I get a usage error, because, in 12.1.0.2,
osdbagrp does not support the '-r' argument. The 18c version of the
osdbagrp binary works fine, but that doesn't help me, since this is a
12.1.0.2 database.

Anyone have any insight into what's going on here? If I could just get
this automated startup and shutdown working, I can proceed.

Sadly, there is very little documentation on the 'osdbagrp' command, so,
I'm not sure what all the options are. It seems to be related to returning
the various dba groups: dba,asmadmin,asmoper,oper, etc,etc....

Finally, I checked MOS Doc. ID 337737.1, and both 12.1.0.2 and 11.2.0.4
*are* supported with 18c ASM/GRID.

Help?

-Mark

[Mark J. Bobak]

Well, that was easy.

Thanks to Martin Berger for the solution.

I had the srvctl entries in the 18c Oracle home, instead of in the 12c and
11g Oracle homes, that they belonged in.

As soon as I deleted the actress from 18c, and added them to the
appropriate DB Oracle home, everything worked perfectly

Not sure how I got that turned around, but once I straightened it out, it
was fine.

-Mark

Post by Mark J. Bobak
Hi all,
I have an SR open on this, but progress seems slow, so I thought I'd put
something here....
AWS EC2 instance, m5d.2xlarge (8 VCPUs/32GB RAM)
Oracle 18c (18.3.0) ASM/Grid w/ ASMFD configured, single instance (SIHA),
not RAC
Three Oracle Databases: 2x 12.1.0.2 and 1x 11.2.0.4
Everything works fine, except I can't start/stop/config any DB (12c or
11g) via srvcrl.
If I start/stop w/ SQL*Plus, everything works great.
ORACLE_SID = [oracle] ? +ASM
The Oracle base has been set to /u01/app
PRCR-1079 : Failed to start resource ora.bmpr2prd.db
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
CRS-2674: Start of 'ora.bmpr2prd.db' on 'miscr2' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
If I look into
/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
2018-09-21 17:41:51.511 : AGFW:34227968: {0:0:21154} sending status msg
[CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
] for start for resource: ora.bmpr2prd.db 1 1
Which doesn't really tell me much. Oracle Support had me turn on srvm
[main] [ 2018-09-21 10:25:56.395 EDT ]
/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r^M
[Thread-14] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[main] [ 2018-09-21 10:25:56.395 EDT ] [RuntimeExec.runCommand:294]
runCommand: Waiting for the process^M
[Thread-13] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>Usage: /u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -a | -d |
-o | -b | -g | -k^M
So, according to the trace, it runs the 'osdbagrp' command several times
with different command line arguments. -a, -d, -o, -b, -g, and -k all work
and (appear to) produce valid results.
When it runs 'osdbagrp -r', I get a usage error, because, in 12.1.0.2,
osdbagrp does not support the '-r' argument. The 18c version of the
osdbagrp binary works fine, but that doesn't help me, since this is a
12.1.0.2 database.
Anyone have any insight into what's going on here? If I could just get
this automated startup and shutdown working, I can proceed.
Sadly, there is very little documentation on the 'osdbagrp' command, so,
I'm not sure what all the options are. It seems to be related to returning
the various dba groups: dba,asmadmin,asmoper,oper, etc,etc....
Finally, I checked MOS Doc. ID 337737.1, and both 12.1.0.2 and 11.2.0.4
*are* supported with 18c ASM/GRID.
Help?
-Mark

[Jared Still]

you deleted an actress?

which one?

Denise Crosby gets my vote.

Post by Mark J. Bobak
Well, that was easy.
Thanks to Martin Berger for the solution.
I had the srvctl entries in the 18c Oracle home, instead of in the 12c and
11g Oracle homes, that they belonged in.
As soon as I deleted the actress from 18c, and added them to the
appropriate DB Oracle home, everything worked perfectly
Not sure how I got that turned around, but once I straightened it out, it
was fine.
-Mark

Post by Mark J. Bobak
Hi all,
I have an SR open on this, but progress seems slow, so I thought I'd put
something here....
AWS EC2 instance, m5d.2xlarge (8 VCPUs/32GB RAM)
Oracle 18c (18.3.0) ASM/Grid w/ ASMFD configured, single instance (SIHA),
not RAC
Three Oracle Databases: 2x 12.1.0.2 and 1x 11.2.0.4
Everything works fine, except I can't start/stop/config any DB (12c or
11g) via srvcrl.
If I start/stop w/ SQL*Plus, everything works great.
ORACLE_SID = [oracle] ? +ASM
The Oracle base has been set to /u01/app
PRCR-1079 : Failed to start resource ora.bmpr2prd.db
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
CRS-2674: Start of 'ora.bmpr2prd.db' on 'miscr2' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
If I look into
/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
2018-09-21 17:41:51.511 : AGFW:34227968: {0:0:21154} sending status
msg [CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
] for start for resource: ora.bmpr2prd.db 1 1
Which doesn't really tell me much. Oracle Support had me turn on srvm
[main] [ 2018-09-21 10:25:56.395 EDT ]
/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r^M
[Thread-14] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[main] [ 2018-09-21 10:25:56.395 EDT ] [RuntimeExec.runCommand:294]
runCommand: Waiting for the process^M
[Thread-13] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>Usage: /u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -a | -d |
-o | -b | -g | -k^M
So, according to the trace, it runs the 'osdbagrp' command several times
with different command line arguments. -a, -d, -o, -b, -g, and -k all work
and (appear to) produce valid results.
When it runs 'osdbagrp -r', I get a usage error, because, in 12.1.0.2,
osdbagrp does not support the '-r' argument. The 18c version of the
osdbagrp binary works fine, but that doesn't help me, since this is a
12.1.0.2 database.
Anyone have any insight into what's going on here? If I could just get
this automated startup and shutdown working, I can proceed.
Sadly, there is very little documentation on the 'osdbagrp' command, so,
I'm not sure what all the options are. It seems to be related to returning
the various dba groups: dba,asmadmin,asmoper,oper, etc,etc....
Finally, I checked MOS Doc. ID 337737.1, and both 12.1.0.2 and 11.2.0.4
*are* supported with 18c ASM/GRID.
Help?
-Mark
--

Jared Still
Certifiable Oracle DBA and Part Time Perl Evangelist
Principal Consultant at Pythian
Pythian Blog http://www.pythian.com/blog/author/still/
Github: https://github.com/jkstill

[Mark J. Bobak]

Yeah, bad typo there. I deleted the entries I had created om 18c home and
added them under their respective 11g and 12c homes, and everything started
working.

It's bad enough I screwed it up, but that I had an SR open w/ Oracle for a
week, and *they* didn't catch it, that's bad!

As to Denise Crosby, apparently the ST:TNG producers agreed with you.

-Mark

Post by Jared Still
you deleted an actress?
which one?
Denise Crosby gets my vote.

Post by Mark J. Bobak
Well, that was easy.
Thanks to Martin Berger for the solution.
I had the srvctl entries in the 18c Oracle home, instead of in the 12c
and 11g Oracle homes, that they belonged in.
As soon as I deleted the actress from 18c, and added them to the
appropriate DB Oracle home, everything worked perfectly
Not sure how I got that turned around, but once I straightened it out, it
was fine.
-Mark

Post by Mark J. Bobak
Hi all,
I have an SR open on this, but progress seems slow, so I thought I'd put
something here....
AWS EC2 instance, m5d.2xlarge (8 VCPUs/32GB RAM)
Oracle 18c (18.3.0) ASM/Grid w/ ASMFD configured, single instance
(SIHA), not RAC
Three Oracle Databases: 2x 12.1.0.2 and 1x 11.2.0.4
Everything works fine, except I can't start/stop/config any DB (12c or
11g) via srvcrl.
If I start/stop w/ SQL*Plus, everything works great.
ORACLE_SID = [oracle] ? +ASM
The Oracle base has been set to /u01/app
PRCR-1079 : Failed to start resource ora.bmpr2prd.db
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
CRS-2674: Start of 'ora.bmpr2prd.db' on 'miscr2' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
If I look into
/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc
CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
2018-09-21 17:41:51.511 : AGFW:34227968: {0:0:21154} sending status
msg [CRS-5017: The resource action "ora.bmpr2prd.db start" encountered the
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 1804474595
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in
"/u01/app/diag/crs/miscr2/crs/trace/ohasd_oraagent_oracle.trc".
] for start for resource: ora.bmpr2prd.db 1 1
Which doesn't really tell me much. Oracle Support had me turn on srvm
[main] [ 2018-09-21 10:25:56.395 EDT ]
/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r^M
[Thread-14] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[main] [ 2018-09-21 10:25:56.395 EDT ] [RuntimeExec.runCommand:294]
runCommand: Waiting for the process^M
[Thread-13] [ 2018-09-21 10:25:56.395 EDT ] [StreamReader.run:62] In
StreamReader.run ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>/u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -r ^M
[Thread-14] [ 2018-09-21 10:25:56.396 EDT ] [StreamReader.run:66]
ERROR>Usage: /u01/app/oracle/product/12.1.0/dbhome_1/bin/osdbagrp -a | -d |
-o | -b | -g | -k^M
So, according to the trace, it runs the 'osdbagrp' command several times
with different command line arguments. -a, -d, -o, -b, -g, and -k all work
and (appear to) produce valid results.
When it runs 'osdbagrp -r', I get a usage error, because, in 12.1.0.2,
osdbagrp does not support the '-r' argument. The 18c version of the
osdbagrp binary works fine, but that doesn't help me, since this is a
12.1.0.2 database.
Anyone have any insight into what's going on here? If I could just get
this automated startup and shutdown working, I can proceed.
Sadly, there is very little documentation on the 'osdbagrp' command, so,
I'm not sure what all the options are. It seems to be related to returning
the various dba groups: dba,asmadmin,asmoper,oper, etc,etc....
Finally, I checked MOS Doc. ID 337737.1, and both 12.1.0.2 and 11.2.0.4
*are* supported with 18c ASM/GRID.
Help?
-Mark
--

Jared Still
Certifiable Oracle DBA and Part Time Perl Evangelist
Principal Consultant at Pythian
Pythian Blog http://www.pythian.com/blog/author/still/
Github: https://github.com/jkstill