{"id":27,"date":"2023-05-04T18:24:06","date_gmt":"2023-05-04T16:24:06","guid":{"rendered":"http:\/\/cbleile.de\/wordpress\/?p=27"},"modified":"2023-05-04T18:33:06","modified_gmt":"2023-05-04T16:33:06","slug":"patching-oda-to-19-18-error-in-sshd_config","status":"publish","type":"post","link":"http:\/\/cbleile.de\/wordpress\/patching-oda-to-19-18-error-in-sshd_config\/","title":{"rendered":"Patching ODA to 19.18.: Error in sshd_config"},"content":{"rendered":"

I recently patched an Oracle Database Appliance (ODA) from 19.16. to 19.18. Detailed steps on how to patch to 19.18. are available here<\/a>. During the update-server step I got the following error:<\/p>\n\n\n

# odacli describe-job -i \"859c6b58-19e5-453e-96a9-f29bcc70894a\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  859c6b58-19e5-453e-96a9-f29bcc70894a\n            Description:  Server Patching\n                 Status:  Failure\n                Created:  April 24, 2023 4:28:10 PM CEST\n                Message:  DCS-10001:Internal error encountered: Error in sshd_config.\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nServer patching                          April 24, 2023 4:28:25 PM CEST      April 24, 2023 4:37:54 PM CEST      Failure\nValidating GI user metadata              April 24, 2023 4:28:25 PM CEST      April 24, 2023 4:28:25 PM CEST      Success\nValidate ILOM server reachable           April 24, 2023 4:28:25 PM CEST      April 24, 2023 4:28:25 PM CEST      Success\nValidate DCS Admin mTLS setup            April 24, 2023 4:28:25 PM CEST      April 24, 2023 4:28:25 PM CEST      Success\nServer patching                          April 24, 2023 4:28:26 PM CEST      April 24, 2023 4:37:54 PM CEST      Failure\nConfigure export clones resource         April 24, 2023 4:28:26 PM CEST      April 24, 2023 4:28:27 PM CEST      Success\nCreating repositories using yum          April 24, 2023 4:28:27 PM CEST      April 24, 2023 4:28:31 PM CEST      Success\nUpdating YumPluginVersionLock rpm        April 24, 2023 4:28:31 PM CEST      April 24, 2023 4:28:31 PM CEST      Success\nApplying OS Patches                      April 24, 2023 4:28:31 PM CEST      April 24, 2023 4:37:54 PM CEST      Success\nServer patching                          April 24, 2023 4:37:54 PM CEST      April 24, 2023 4:37:54 PM CEST      Failure\nModify sshd_config attribute             April 24, 2023 4:37:54 PM CEST      April 24, 2023 4:37:54 PM CEST      Failure<\/code><\/pre>\n\n\n\n
So obviously something is wrong with sshd_config. If an error happens when running a job on the ODA with odacli then I check the dcs agent logfile \/opt\/oracle\/dcs\/log\/dcs-agent.log. It contained the following:<\/p>\n\n\n\n
2023-04-24 16:37:54,771 DEBUG [Modify sshd_config attribute : JobId=859c6b58-19e5-453e-96a9-f29bcc70894a] [] c.o.d.c.u.CommonsUtils: Output :\n\/etc\/ssh\/sshd_config line 147: Directive 'Protocol' is not allowed within a Match block<\/code><\/pre>\n\n\n\n
At the end of the \/etc\/ssh\/sshd_config I saw those lines:<\/p>\n\n\n\n
# Example of overriding settings on a per-user basis\n#Match User anoncvs\n#       X11Forwarding no\n#       AllowTcpForwarding no\n#       PermitTTY no\n#       ForceCommand cvs server\n# BEGIN ANSIBLE MANAGED BLOCK\nMatch User user41\n  PasswordAuthentication no\nMatch User user46\n  PasswordAuthentication no\n# END ANSIBLE MANAGED BLOCK<\/code><\/pre>\n\n\n\n
So obviously the sshd_config file was adjusted by the customer using ansible. This update was done already a year ago and the ODA has been patched in between successfully. But why did I get an error <\/p>\n\n\n\n
Directive 'Protocol' is not allowed within a Match block<\/code><\/pre>\n\n\n\n
The line <\/p>\n\n\n\n
Protocol 2<\/code><\/pre>\n\n\n\n
was already in the file sshd_config. Unfortunately Oracle rolls back all changes in case of failure and I could not see what happened with sshd_config. In the dcs-agent.log I saw that 2 files were temporarily created to modify the sshd_config-file:<\/p>\n\n\n\n
\/opt\/oracle\/dcs\/bin\/modifySshdConfScript.sh\n\/tmp\/dcsfiles\/temp.txt<\/code><\/pre>\n\n\n\n
Both files were deleted as part of the cleanup of the failure. So I created a simple loop to check every second if one of above files has been created:<\/p>\n\n\n\n
# while true\n> do\n> cat \/opt\/oracle\/dcs\/bin\/modifySshdConfScript.sh \/tmp\/dcsfiles\/temp.txt\n> sleep 1\n> done\ncat: \/opt\/oracle\/dcs\/bin\/modifySshdConfScript.sh: No such file or directory\ncat: \/tmp\/dcsfiles\/temp.txt: No such file or directory\n...<\/code><\/pre>\n\n\n\n
While this was running I started my update-server again:<\/p>\n\n\n\n
# \/opt\/oracle\/dcs\/bin\/odacli update-server -v 19.18.0.0.0<\/code><\/pre>\n\n\n\n
REMARK: After the failure the clusterware was no longer running. I.e. above update-server resulted in an immediate message<\/p>\n\n\n\n
DCS-10059:Oracle Clusterware is not running on all nodes.<\/code><\/pre>\n\n\n\n
To start the clusterware again I just did a reboot.<\/p>\n\n\n\n
After the reboot and starting my loop again (followed by the update-server in a separate session) I got this output:<\/p>\n\n\n\n
# while true\n> do\n> cat \/opt\/oracle\/dcs\/bin\/modifySshdConfScript.sh \/tmp\/dcsfiles\/temp.txt\n> sleep 1\n> done\ncat: \/opt\/oracle\/dcs\/bin\/modifySshdConfScript.sh: No such file or directory\ncat: \/tmp\/dcsfiles\/temp.txt: No such file or directory\n...\ncat: \/opt\/oracle\/dcs\/bin\/modifySshdConfScript.sh: No such file or directory\ncat: \/tmp\/dcsfiles\/temp.txt: No such file or directory\n#!\/bin\/sh\nSSHD_CONF_FILE=\/etc\/ssh\/sshd_config\nif [ -f \"$SSHD_CONF_FILE\" ]; then\n...\n  sed -i 's\/^[^#]*Protocol\/#&\/' $SSHD_CONF_FILE\n...\n  echo 'AllowTcpForwarding no' >> $SSHD_CONF_FILE\n  echo 'Protocol 2' >> $SSHD_CONF_FILE\n...\nelse\n  echo \"Unable to locate sshd config file SSHD_CONF_FILE\"\n  exit 1\nfi\n...\ncat: \/opt\/oracle\/dcs\/bin\/modifySshdConfScript.sh: No such file or directory\ncat: \/tmp\/dcsfiles\/temp.txt: No such file or directory\n...<\/code><\/pre>\n\n\n\n
So the issue became clear now. The existing line<\/p>\n\n\n\n
Protocol 2<\/code><\/pre>\n\n\n\n
was put in comments and new lines (including the line „Protocol 2“) were added at the end of sshd_config. The line<\/p>\n\n\n\n
Protocol 2<\/code><\/pre>\n\n\n\n
is not allowed in a Match Block, which has been included „open ended“ with the ansible update.<\/p>\n\n\n\n
My workaround was to temporarily comment the Match-commands in sshd_config and then my update-server did run through:<\/p>\n\n\n\n
# BEGIN ANSIBLE MANAGED BLOCK\n#Match User user41\n#  PasswordAuthentication no\n#Match User user46\n#  PasswordAuthentication no\n# END ANSIBLE MANAGED BLOCK<\/code><\/pre>\n\n\n\n
Afterwards I moved above lines to the end of the sshd_config again and un-commented the lines as they were before. Then restart sshd or reboot the server.<\/p>\n\n\n\n
I’ve found several blogs and discussions on how to end a Match-block in sshd_config to avoid the errors I’ve seen. You may check here<\/a> or here<\/a>. Basically the customer did modify the sshd_config correctly, because Match-Blocks should be added at the end of the file. This can also been seen by the example, which was there in the default sshd_config of the ODA at the end of the file:<\/p>\n\n\n\n
# Example of overriding settings on a per-user basis\n#Match User anoncvs\n#       X11Forwarding no\n#       AllowTcpForwarding no\n#       PermitTTY no\n#       ForceCommand cvs server<\/code><\/pre>\n\n\n\n
Workarounds to end a Match-Block with the line „Match“ should not be used anymore. Better would be to end the block with „Match All“, but even that may not be correct. So is it an Oracle bug of the ODA patching software here? Yes and No. Yes, because modifying files should be done in a way, which won’t break the syntax. No, because changes (customizations) in sshd_config should be undone before patching an ODA. So it goes back to the statement that you should not modify\/customize an Oracle Database Appliance. But if you do then undo those modifications before patching and redo them after the patch has been successfully installed.<\/p>\n","protected":false},"excerpt":{"rendered":"
I recently patched an Oracle Database Appliance (ODA) from 19.16. to 19.18. Detailed steps on how to patch to 19.18. are available here. During the update-server step I got the following error: So obviously something is wrong with sshd_config. If an error happens when running a job on the ODA with odacli then I check […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[30,27,3,28,29],"class_list":["post-27","post","type-post","status-publish","format-standard","hentry","category-oracle","tag-failure","tag-oda","tag-oracle","tag-sshd_config","tag-update-server"],"_links":{"self":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts\/27","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/comments?post=27"}],"version-history":[{"count":2,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts\/27\/revisions"}],"predecessor-version":[{"id":29,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/posts\/27\/revisions\/29"}],"wp:attachment":[{"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/media?parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/categories?post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/cbleile.de\/wordpress\/wp-json\/wp\/v2\/tags?post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}