Event 1000, Application Error {opmn.exe} ” Oracle-agent12cR1ProcessManager”

“Event 1000, Application Error {opmn.exe}” One of the Common Issues in Agent/ OMS 12c. Very Often you will find under “Application Error” in Event Viewer of Windows some thing like this:

Faulting application name: opmn.exe, version:, time stamp: 0x488507f7
Faulting module name: unknown, version:, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00000000
Faulting process id: 0x163c
Faulting application start time: 0x01d050194c4e6030
Faulting application path: C:\app\oracle\product\agent12c\core\\opmn\bin\opmn.exe
Faulting module path: unknown
Report Id: 89fd4e64-bc0c-11e4-987a-0050569857ec

This Issue occur if you install 32-bit agent on 64-bit Windows Server. In Oracle 12c when you install Oracle Agent 12c, there is one extra Service ” Oracle-agent12cR1ProcessManager” which gets install.

This is known Bug in Oracle with BUG ID 14610919. You might not be able to see the contents of this BUG as this is not a PUBLIC Bug.

This Application Error can be easily stopped by stopping this Service and changing its property “Startup Type” to Manual. Stopping this service will not impact functionality of Oracle Agent but this does not means that this is useless or extra service.

We can use this Service to start and stop Management Agent, OMS and other related components including “HTTP Server, OC4J_EM, WebCache etc” These Components not only run as Child Processes of this service but also inherit its security settings.  The reason why this service gets installed by default is that in 12C we have got the OPMN component installed with Agent by default.

To avoid this Application Error from occurring every second, just STOP  “Oracle-agent12cR1ProcessManager” service.



OMS failed to start “Connection to Repository Failed”


Recently one issue blew my mind off, when all of a sudden my OEM test environment failed to startup. Since it was running fine earlier but this time when I start OMS using the command “>emctl start oms” it failed.

When I explored the logs I came across that some has changed the repository credentials for SYSMAN, which was done at the backend and was not synchronized with the OMS. This led the OMS not to connect with the repository. Something like this popped up in the logs.

“failureReason: DB Connection service is invalid or temporarily unavailable” and “ERROR commands.BaseCommand logAndPrint.623 – Oracle Management Server is not functioning because of the following reason:
Connection to the repository failed. Verify that the repository connection information provided is correct.”

Key Point to remember is that “SYSMAN” password should not be changed directly in repository database. The ONLY way to change SYSMAN password is using emctl command. So next thing I tried was reset SYSMAN password using “emctl” command.

[oracle@houoemap1 bin]$ ./emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd formula123 -new_pwd formula123
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Error occurred. Check the log /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/secure.log

Which Failed again 😦

I checked the logs and find that my “Admin Server” was not up and running but even after bringing Admin Server up I was not allowed to change password.

2015-02-10 07:03:32,915 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.177 – Getting credentials from wallet
2015-02-10 07:03:33,218 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.192 – Read the credentials from wallet
2015-02-10 07:03:33,221 [main] INFO util.EmctlUtil logp.251 – Connecting over t3s to: houoemap1.xxxxxx.com/7102 using id: weblogic
2015-02-10 07:03:33,851 [main] INFO util.EmctlUtil logp.251 – Unable to get mbean conn over t3s :null
2015-02-10 07:03:33,854 [main] ERROR oms.ChangeReposPwd logp.251 –

at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:178)
… 7 more
Caused by: java.net.ConnectException: t3s://houoemap1.xxxxxx.com:7102: Destination unreachable; nested exception is:
java.net.ConnectException: Connection refused; No available router to destination

I started Admin server using ” >emctl start oms -admin_only”

Followed by this I again tried to reset the sysman password using “EMCTL” but again no luck 😦

[oracle@houoemap1 bin]$ ./emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd formula123 -new_pwd formula1
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Error occurred. Check the log /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/secure.log
[oracle@houoemap1 bin]$

Now I had to open case with MOS and they guided me to reset the password of all four dependent users “sysman, sysman_apm, sysman_mds, sysman_opss” to common password and then try to start oms, so I did.

Reset Password—————

[oracle@houoemap1 db_1]$ ss
SQL*Plus: Release Production on Fri Feb 13 02:41:33 2015
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> conn sysman/formula1
SQL> conn sysman_apm/formula1
SQL> conn sysman_mds/formula1
SQL> conn sysman_opss/formula1

Start OMS —————-

[oracle@houoemap1 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting Oracle Management Server…
Starting WebTier…
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.
Check the following log files:
EM log files: /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log, emoms.trc, emoms_pbs.trc
[oracle@houoemap1 bin]$

Providing logs to Oracle Support we came to know that there is mismatch of the information in JDBC xml files.

Used “/app/oracle/product/middleware/gc_inst/user_projects/domains/GCDomain/config/jdbc/*” to find details.

Finally below action plan worked.


1. Take a backup of the directory /app/oracle/product/middleware/gc_inst/user_projects/domains/GCDomain/config/jdbc
2. Access weblogic admin console. https://houoemap1.xxxxxx.com/7102/console
Enter ‘weblogic’ as the username and its password.
If the weblogic admin server is not up, then start it using:
OMS_HOME/bin>./emctl start oms -admin_only
3. From the admin console, Leftpanel > Services > Datasources
Here you will have list of all datasources. Click on “Lock and Edit” on top corner left side. Click on Datasources one by one.
For example: Click on emgc-sysman-pool > Connection Pool tab > Enter Password and Confirm Password as formula1
And Save each change.
Please repeat this for all the datasources.
4. Click on “Release Configuration” (Under Lock and Edit) after the changes are done.
5. Now restart the OMS with
$emctl stop oms -all -force
$emctl start oms

If you face this Issue ever, this Action plan will help you to get the Issue resolved.



How to add Additional target properties in Oracle Enterprise Manager12c

In Oracle Enterprise Manager we have the option set properties of targets which in-turn we can later use to categorize them.

The Administration Group option in Oracle Enterprise Manager 12c is one of the good features introduced. This feature help us categorize our targets in different hierarchy sub groups. The hierarchy was decided on the number of target properties having values set.

Up to recently I was under the assumption that this was a fixed list of target properties like “Department, LifeCycle Status, Contact etc”. However I was caught in a scenario
where I have to increase the hierarchy in my Administration Group to meet my business need. This was the point when I thought what If I can add something more to this property set. This gave me reason to explore some more feature of EMCLI.

I’ll take example of my test case in this Blog.

I wanted to add a property “ENVIRONMENT” using which I can differentiate all targets for my PeopleSoft Environments “Financials, HR, Brazil and Portal”
This came as a solution of the enhancement request that was already in place. MOS provided the solution to this problem. We can add additional target property fields by using EMCLI.

The parameter to add additional target properties for a target in Oracle Enterprise Manager is “add_target_property”.
By using this parameter with EMCLI command we can extend the number of fields per target type.

For example, if you wants to add new target property “ENVIRONMENT” for all targets it can be easily done using below mentioned command.

>emcli add_target_property -target_type=”*” -property=”ENVIRONMENT”

In case you wanted to add the ENVIRONMENT property to only oracle_database targets you could do so by executing the below command:

>emcli add_target_property -target_type=”oracle_database” – property=”ENVIRONMENT”

This option provides you a versatile Oracle Enterprise Manager with extended capability to store more information on a target level.

Hope you have find this post interesting and useful in your business.