Yesterday I came across an issue when I was installing Oracle Agent 12cR3 on my TEST environment, the installation ended with a new error message for me this time. It was for the first time in 3 years while working on Oracle Enterprise Manager I faced this Issue.
This was the error message:- ” ERROR: Agent Configuration Failed SEVERE: emctl start agent command has failed with status1“
I have added the complete installation logs to make picture more clear.
[oracle@testdb1 linux_x64]$ ./agentDeploy.sh OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_BASE_DIR=/app/oracle/product/agent12c AGENT_REGISTRATION_PASSWORD=formula123 Validating the OMS_HOST & EM_UPLOAD_PORT Executing command : /app/oracle/product/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /app/oracle/product/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12c/core/12.1.0.4.0 /app/oracle/product/agent_sw/linux_x64 /app/oracle/product/agent12c -prereq Validating oms host & port with url: http://testdb1.oii.test.com:4903/empbs/genwallet Validating oms host & port with url: https://testdb1.oii.test.com:4903/empbs/genwallet Return status:3-oms https port is passed Unzipping the agentcoreimage.zip to /app/oracle/product/agent12c .... 12.1.0.4.0_PluginsOneoffs_226.zip Executing command : /app/oracle/product/agent_sw/linux_x64/unzip -o /app/oracle/product/agent_sw/linux_x64/12.1.0.4.0_PluginsOneoffs_226.zip -d /app/oracle/product/agent12c Executing command : /app/oracle/product/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/xmlparserv2.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/srvm.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/emCfg.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12c/core/12.1.0.4.0 /app/oracle/product/agent_sw/linux_x64 /app/oracle/product/agent12c /app/oracle/product/agent12c/agent_inst AGENT_BASE_DIR=/app/oracle/product/agent12c Executing agent install prereqs... Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion -prereqlogloc /app/oracle/product/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/app/oracle/product/agent12c/core/12.1.0.4.0/prereqs -J-DAGENT_BASE_DIR=/app/oracle/product/agent12c Agent install prereqs completed successfully Cloning the agent home... Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0 AGENT_BASE_DIR=/app/oracle/product/agent12c OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_BASE_DIR=/app/oracle/product/agent12c AGENT_REGISTRATION_PASSWORD=formula123 -noconfig ORACLE_HOME_NAME=agent12c3 -force b_noUpgrade=true AGENT_PORT=-1 Cloning of agent home completed successfully Attaching sbin home... Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -attachHome -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12c/sbin ORACLE_HOME_NAME=sbin12c3 -force Attach home for sbin home completed successfully. Updating home dependencies... Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/app/oracle/product/agent12c/sbin:/app/oracle/product/agent12c/core/12.1.0.4.0} -invPtrLoc /app/oracle/product/agent12c/core/12.1.0.4.0/oraInst.loc -force Update home dependency completed successfully. Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/app/oracle/product/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true ERROR: Agent Configuration Failed SEVERE: emctl start agent command has failed with status1 Agent Deploy Log Location:/app/oracle/product/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy/agentDeploy_2016-08-04_22-40-59-PM.log [oracle@testdb1 linux_x64]$
I checked the log file as mentioned in the last line and this I what I came across.
——————————————————————————————————————————————
INFO: Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/app/oracle/product/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true
INFO: Setting the invPtrLoc to /app/oracle/product/agent12c/core/12.1.0.4.0/oraInst.loc
INFO:
INFO: perform - mode is starting for action: configure
INFO:
INFO: ** Agent Port Check completed successfully.**
INFO: SEVERE: emctl start agent command has failed with status1
INFO: Follow the below steps to manually start the agent and add the host and agent targets:
INFO: 1. Start the agent by executing the command: /app/oracle/product/agent12c/agent_inst/bin/emctl start agent
INFO: 2. Add the targets by executing the command:/app/oracle/product/agent12c/agent_inst/bin/emctl config agent addinternaltargets.
INFO:
INFO: perform - mode finished for action: configure
INFO:
INFO: You can see the log file: /app/oracle/product/agent12c/core/12.1.0.4.0/cfgtoollogs/oui/configActions2016-08-04_10-41-54-PM.log
INFO: Plugin homes:
INFO: Plugin homes:
INFO: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/app/oracle/product/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true completed with status=3
SEVERE:
ERROR: Agent Configuration Failed
SEVERE:
ERROR: Agent Deployment Failed
I followed the steps as mentioned in the log file and tried to start the agent. It failed again, to be honest I was expecting that. 😉
My Concern:-
Same setup is working fine for my production OEM why it is failing again and again for Test OEM environment. I used the same setup to install and configure agent on multiple servers which were being monitored by my PRODUCTION OEM12c and they were installed and configured successfully without any errors.
So why now it was failing for OMS in my TEST OEM12c environment.
Then last option left for me was to explore the trace files and see what exactly is causing issue. So I started digging deeper in the log and trace files until I reached common error message in all of them.
My findings.
— EMState agent
---------------------------------------- ----- Thu Aug 4 23:01:15 2016::27430::Auto tuning the agent at time Thu Aug 4 23:01:15 2016 ----- ----- Thu Aug 4 23:01:16 2016::27430::Finished auto tuning the agent at time Thu Aug 4 23:01:16 2016 ----- ----- Thu Aug 4 23:01:16 2016::27430::Launching the JVM with following options: -Xmx128M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops ----- ----- Thu Aug 4 23:01:16 2016::27430::Agent Launched with PID 27484 at time Thu Aug 4 23:01:16 2016 ----- ----- Thu Aug 4 23:01:16 2016::27484::Time elapsed between Launch of Watchdog process and execing EMAgent is 1 secs ----- ----- Thu Aug 4 23:01:16 2016::27430::Previous Thrash State(-1,-1) ----- 2016-08-04 23:01:16,750 [1:main] WARN - Missing filename for log handler 'wsm' 2016-08-04 23:01:16,757 [1:main] WARN - Missing filename for log handler 'opss' 2016-08-04 23:01:16,759 [1:main] WARN - Missing filename for log handler 'opsscfg' OMS decided to shutdown the agent because of the following reason sent from OMS: EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED ----- Thu Aug 4 23:01:29 2016::27430::Checking status of EMAgent : 27484 ----- ----- Thu Aug 4 23:01:29 2016::27430::EMAgent exited at Thu Aug 4 23:01:29 2016 with return value 0. ----- ----- Thu Aug 4 23:01:29 2016::27430::writeAbnormalExitTimestampToAgntStmp: exitCause=NORMAL : restartRequired=0 ----- ----- Thu Aug 4 23:01:29 2016::27430::EMAgent was shutdown normally. ----- ----- Thu Aug 4 23:01:29 2016::27430::Exiting watchdog loop -----
gcagent.log
---------------------- 2016-08-04 23:01:19,441 [49:905DEBE9] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: SHUTDOWN EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED) 2016-08-04 23:01:19,443 [57:CB968A7D:GC.SysExecutor.2 (ShutdownFromPing)] INFO - OMS decided to shutdown the agent because of the following reason sent from OMS: EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED 2016-08-04 23:01:19,443 [1:3305B9] INFO - Agent is shutting down 2016-08-04 23:01:19,444 [49:905DEBE9] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: SHUTDOWN EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED) 2016-08-04 23:01:19,444 [1:3305B9] INFO - Invoking SHUTDOWN (1) on Internal Metrics 2016-08-04 23:01:19,444 [1:3305B9] INFO - Done: SHUTDOWN (1) on Internal Metrics 2016-08-04 23:01:19,444 [1:3305B9] INFO - Invoking SHUTDOWN on Miscellaneous 2016-08-04 23:01:19,444 [1:3305B9] INFO - Done: SHUTDOWN on Miscellaneous
I gave one try more but this time with proper procedure. So I followed my own document (blog) for “Silent Installation of Agent” and this time it worked and agent was installed and configured successfully.
What was the issue ? as OMS mentioned in the Error logs. “ OMS decided to shutdown the agent because of the following reason sent from OMS: EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED“
SOLUTION:-
The setup which I used earlier was downloaded from the production environment this was the compatible with EM Plugin in the production not with test environment and this was the reason why it was giving error again and again.
When I downloaded agent from the test environment and deployed it got configured successfully.
Procedure is mentioned below:-
————————————————
[oracle@testdb1 bin]$ pwd /app/oracle/product/middleware/OMS12CR4/oms/bin [oracle@testdb1 bin]$ ./emcli login -username=sysman -password=formula1 Login successful [oracle@testdb1 bin]$ ./emcli sync Synchronized successfully [oracle@testdb1 bin]$ ./emcli get_supported_platforms ----------------------------------------------- Version = 12.1.0.3.0 Platform = Linux x86-64 ----------------------------------------------- Version = 12.1.0.4.0 Platform = Linux x86-64 ----------------------------------------------- Platforms list displayed successfully. [oracle@testdb1 bin]$ ./emcli get_agentimage -destination=/tmp/deepak -platform="Linux x86-64" -version=12.1.0.3.0 === Partition Detail === Space free : 3 GB Space required : 1 GB Check the logs at /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/.emcli/get_agentimage_2016-08-05_00-57-38-AM.log Downloading /tmp/deepak/12.1.0.3.0_AgentCore_226.zip File saved as /tmp/deepak/12.1.0.3.0_AgentCore_226.zip Downloading /tmp/deepak/12.1.0.3.0_PluginsOneoffs_226.zip File saved as /tmp/deepak/12.1.0.3.0_PluginsOneoffs_226.zip Downloading /tmp/deepak/unzip File saved as /tmp/deepak/unzip Agent Image Download completed successfully. [oracle@testdb1 bin]$ pwd [oracle@testdb1 agent_sw]$ ./agentDeploy.sh AGENT_BASE_DIR=/app/oracle/product/agent12cr3 OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=formula123 Validating the OMS_HOST & EM_UPLOAD_PORT Executing command : /app/oracle/product/agent12cr3/core/12.1.0.3.0/jdk/bin/java -classpath /app/oracle/product/agent12cr3/core/12.1.0.3.0/jlib/agentInstaller.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12cr3/core/12.1.0.3.0 /app/oracle/product/agent_sw /app/oracle/product/agent12cr3 -prereq Validating oms host & port with url: http://testdb1.oii.test.com:4903/empbs/genwallet Validating oms host & port with url: https://testdb1.oii.test.com:4903/empbs/genwallet Return status:3-oms https port is passed Unzipping the agentcoreimage.zip to /app/oracle/product/agent12cr3 .... 12.1.0.3.0_PluginsOneoffs_226.zip Executing command : /app/oracle/product/agent_sw/unzip -o /app/oracle/product/agent_sw/12.1.0.3.0_PluginsOneoffs_226.zip -d /app/oracle/product/agent12cr3 Executing command : /app/oracle/product/agent12cr3/core/12.1.0.3.0/jdk/bin/java -classpath /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/OraInstaller.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/xmlparserv2.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/srvm.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/emCfg.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/jlib/agentInstaller.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12cr3/core/12.1.0.3.0 /app/oracle/product/agent_sw /app/oracle/product/agent12cr3 /app/oracle/product/agent12cr3/agent_inst AGENT_BASE_DIR=/app/oracle/product/agent12cr3 Executing agent install prereqs... Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion -prereqlogloc /app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/app/oracle/product/agent12cr3/core/12.1.0.3.0/prereqs -J-DAGENT_BASE_DIR=/app/oracle/product/agent12cr3 Prereq Logs Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/prereq<timestamp>.log Agent install prereqs completed successfully Cloning the agent home... Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12cr3/core/12.1.0.3.0 AGENT_BASE_DIR=/app/oracle/product/agent12cr3 AGENT_BASE_DIR=/app/oracle/product/agent12cr3 OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=formula123 -noconfig ORACLE_HOME_NAME=agent12c4 -force b_noUpgrade=true AGENT_PORT=-1 Clone Action Logs Location:/app/oracle/oraInventory//logs/cloneActions<timestamp>.log Cloning of agent home completed successfully Attaching sbin home... Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -attachHome -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12cr3/sbin ORACLE_HOME_NAME=sbin12c4 -force Attach Home Logs Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/AttachHome<timestamp>.log Attach home for sbin home completed successfully. Updating home dependencies... Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug -ignoreSysPrereqs -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/app/oracle/product/agent12cr3/sbin:/app/oracle/product/agent12cr3/core/12.1.0.3.0} -invPtrLoc /app/oracle/product/agent12cr3/core/12.1.0.3.0/oraInst.loc -force Update Home Dependencies Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/UpdateHomeDeps<timestamp>.log Update home dependency completed successfully. Performing the agent configuration... Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12cr3/core/12.1.0.3.0 RESPONSE_FILE=/app/oracle/product/agent12cr3/core/12.1.0.3.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true Configuration Log Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/cfgfw/CfmLogger<timestamp>.log Agent Configuration completed successfully The following configuration scripts need to be executed as the "root" user. #!/bin/sh #Root script to run /app/oracle/product/agent12cr3/core/12.1.0.3.0/root.sh To execute the configuration scripts:
-
Open a terminal window
-
Log in as "root"
-
Run the scripts
Agent Deployment Successful. Agent deployment log location: /app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/agentDeploy_2016-08-05_01-06-08-AM.log Agent deployment completed successfully. [oracle@testdb1 agent_sw]$ cd /app/oracle/product/agent12cr3/agent_inst/bin/ [oracle@testdb1 bin]$ ./emctl secure agent Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Agent successfully stopped... Done. Securing agent... Started. Enter Agent Registration Password : Agent successfully restarted... Done. EMD gensudoprops completed successfully Securing agent... Successful. [oracle@testdb1 bin]$ ./emctl config agent addinternaltargets Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. [oracle@testdb1 bin]$ ./emctl upload Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD upload completed successfully [oracle@testdb1 bin]$
This time as well I learnt something new and I thought of sharing it with you so I did. Hope this will help you some time in future. Thanks ADDKS
Oracle EM DBA 
Thanks Deepak. Nice Info.Was facing the same issue.
LikeLike