{"id":42,"date":"2023-10-27T14:23:06","date_gmt":"2023-10-27T12:23:06","guid":{"rendered":"http:\/\/cbleile.de\/wordpress\/?p=42"},"modified":"2023-10-27T14:32:31","modified_gmt":"2023-10-27T12:32:31","slug":"oracle-database-appliance-oda-some-listeners-do-not-start-anymore-when-booting","status":"publish","type":"post","link":"http:\/\/cbleile.de\/wordpress\/oracle-database-appliance-oda-some-listeners-do-not-start-anymore-when-booting\/","title":{"rendered":"Oracle Database Appliance (ODA): Some listeners do not start anymore when booting"},"content":{"rendered":"

A colleague of mine had the issue some time ago and I recently experienced the same behaviour at a customer’s ODA: A couple of Listeners did not start when booting. Starting them as user grid with<\/p>\n\n\n

srvctl start listener -listener <listener-name><\/code><\/pre>\n\n\n\n
worked without any problem though. So what is the issue?<\/p>\n\n\n\n
Analysis<\/h2>\n\n\n\n
Looking at the listeners which do not start in the listener.ora it becomes obvious that all listeners, which had an SID_LIST_<listener-name> did not start when booting. <\/p>\n\n\n\n
My first step was to check crs:<\/p>\n\n\n\n
[root@mysrv ~]# . oraenv\nORACLE_SID = [root] ? +ASM1\nORACLE_HOME = [\/home\/oracle] ? \/u01\/app\/19.20.0.0\/grid\nThe Oracle base has been set to \/u01\/app\/grid\n[root@mysrv ~]# crsctl check crs\nCRS-4638: Oracle High Availability Services is online\nCRS-4537: Cluster Ready Services is online\nCRS-4529: Cluster Synchronization Services is online\nCRS-4533: Event Manager is online<\/code><\/pre>\n\n\n\n
That looks good. <\/p>\n\n\n\n
For testing purposes it’s of course easier to stop crs and restart it instead of rebooting the server:<\/p>\n\n\n\n
[root@mysrv ~]# crsctl stop crs -f\nCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mysrv'\nCRS-2673: Attempting to stop 'ora.crsd' on 'mysrv'\nCRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'mysrv'\n...\nCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mysrv' has completed\nCRS-4133: Oracle High Availability Services has been stopped.\n[root@mysrv ~]# crsctl start crs\nCRS-4123: Oracle High Availability Services has been started.\n[root@mysrv ~]# <\/code><\/pre>\n\n\n\n
The listeners could be shutdown without any issue, but they did not come up when starting crs. In the alert.log I did see this:<\/p>\n\n\n\n
[root@mysrv app]# view \/u01\/app\/grid\/diag\/crs\/mysrv\/crs\/trace\/alert.log\n....\n2023-10-24 17:04:57.042 [CRSD(90854)]CRS-2807: Resource 'ora.LISTENER1.lsnr' failed to start automatically.\n2023-10-24 17:04:57.044 [CRSD(90854)]CRS-2807: Resource 'ora.LISTENER2.lsnr' failed to start automatically.\n...<\/code><\/pre>\n\n\n\n
To find the reason why the cluster registry services of grid infrastructure did not start those specific listeners LISTENER1 and LISTENER2, I searched in the oraagent trace-files for SID_LIST_LISTENER1:<\/p>\n\n\n\n
[root@mysrv app]# pwd\n\/u01\/app\n[root@mysrv app]# grep -il sid_list_listener1 .\/grid\/diag\/crs\/mysrv\/crs\/trace\/crsd_oraagent_*.trc\n...\n.\/grid\/diag\/crs\/mysrv\/crs\/trace\/crsd_oraagent_grid.trc\n...\n[root@mysrv app]# view .\/grid\/diag\/crs\/mysrv\/crs\/trace\/crsd_oraagent_grid.trc\n...\n2023-10-24 20:46:58.914 :CLSDYNAM:1738524416: [ora.LISTENER1.lsnr]{1:60057:2} [start] ConfigFile::parse mmap name:sid_list_listener1 nameWithCase:SID_LIST_LISTENER1 value: comment:\n2023-10-24 20:46:58.915 :CLSDYNAM:1700701952: [ora.LISTENER1.lsnr]{1:60057:2} [start] LsnrAgent::Lsnrctl::start lsnrctl start complete lsnrname:LISTENER1 output:\nLSNRCTL for Linux: Version 19.0.0.0.0 - Production on 24-OCT-2023 20:46:58\n\nCopyright (c) 1991, 2023, Oracle.  All rights reserved.\n\nStarting \/u01\/app\/19.20.0.0\/grid\/bin\/tnslsnr: please wait...\n\nTNSLSNR for Linux: Version 19.0.0.0.0 - Production\nSystem parameter file is \/u01\/app\/19.20.0.0\/grid\/network\/admin\/grid\/listener.ora\nLog messages written to \/u01\/app\/grid\/diag\/tnslsnr\/mysrv\/listener1\/alert\/log.xml\nListening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER1)))\nTNS-01201: Listener cannot find executable \/u01\/app\/odaorahome\/oracle\/product\/19.0.0.0\/dbhome_10\/bin\/oracle for SID INST9\n\nListener failed to start. See the error message(s) above...<\/code><\/pre>\n\n\n\n
So the listener could not find the oracle executable in the ORACLE_HOME provided in the SID_LIST_LISTENER1. The executable is of course required to start a connection through a statistic listener entry. When I saw that, it became clear why the listeners did not start: Since ODA 19.11. the ORACLE_HOMEs are located on the ACFS-Filesystem<\/p>\n\n\n\n
\/u01\/app\/odaorahome<\/code><\/pre>\n\n\n\n
REMARK: For more details about that change see an earlier Blog<\/a>.<\/p>\n\n\n\n
The issue was that that filesystem has not been mounted yet when the listener is being started by the oraagent. So I needed to add a startup dependency to the listeners to wait until the ACFS-Filesystem has been mounted. <\/p>\n\n\n\n
REMARK: Starting the listener with srvctl manually worked, because in the meantime the ACFS-filesystem was mounted.<\/p>\n\n\n\n
Solution and verification of the solution<\/h2>\n\n\n\n
What is the current start dependency?<\/p>\n\n\n\n
[grid@mysrv grid]$ crsctl status resource ora.LISTENER1.lsnr -f | grep START_DEPENDENCIES=\nSTART_DEPENDENCIES=hard(type:ora.cluster_vip_net1.type) pullup(type:ora.cluster_vip_net1.type)\n[grid@mysrv grid]$<\/code><\/pre>\n\n\n\n
How to modify the start dependency? Oracle provides the command crsctl modify, but it allows modifications only with the „-unsupported“ option set. We opened a SR with Oracle and they confirmed that we can use that option. The resource we are depending on is „ora.data.orahome_sh.acfs“. So the needed adjustment is:<\/p>\n\n\n\n
[grid@mysrv grid]$ crsctl modify resource ora.LISTENER1.lsnr -attr \"START_DEPENDENCIES='hard(type:ora.cluster_vip_net1.type,ora.data.orahome_sh.acfs) pullup(type:ora.cluster_vip_net1.type)'\" -unsupported\n[grid@mysrv grid]$ <\/code><\/pre>\n\n\n\n
I tested it (crsctl stop crs -f and crsctl start crs) and it worked, the listener came up as expected. Afterwards I changed all required listeners accordingly and rebooted the server. All listeners came up during the boot process.<\/p>\n\n\n\n
Summary<\/h2>\n\n\n\n
Introducing ORACLE_HOMEs on ACFS in 19.11.<\/a> was a very welcome improvement, because it took away the space pressure we usually had on ODAs on the \/u01-filesystem with many ORACLE_HOMEs. Such new features however, may have side effects, which are not obvious initially. Doing a proper analysis usually yields to a solution or a workaround. If you have to implement the solution\/workaround then it’s a good practice to double check with Oracle Support that your solution is supported by Oracle – especially on the ODA.<\/p>\n","protected":false},"excerpt":{"rendered":"
A colleague of mine had the issue some time ago and I recently experienced the same behaviour at a customer’s ODA: A couple of Listeners did not start when booting. Starting them as user grid with worked without any problem though. So what is the issue? Analysis Looking at the listeners which do not start […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[40,27,3,41,38,39],"class_list":["post-42","post","type-post","status-publish","format-standard","hentry","category-oracle","tag-listsner","tag-oda","tag-oracle","tag-start","tag-tns-01201","tag-tns-1201"],"_links":{"self":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts\/42","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/comments?post=42"}],"version-history":[{"count":2,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts\/42\/revisions"}],"predecessor-version":[{"id":44,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts\/42\/revisions\/44"}],"wp:attachment":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/media?parent=42"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/categories?post=42"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/tags?post=42"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}