JMX monitoring OC4J JVM

To monitor the JVM with jconsole or an external profiler the opmn.xml settings of the container should be changed:

This properties must be added:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9900 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Restart the container and now you can connect on the url:

service:jmx:rmi:///jndi/rmi://[HOSTNAME]:9900/jmxrmi






Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Multiple java processes when using webgate

If you use an webgate version which needs the parameter you will see 50 processes or more on linux for the JVM. When you change the webgate to a version which uses NTPD this problem is solved.

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Invalid Access SDK Path. -Djava.library.path must point to {SDK_HOME}/oblix/lib.

There are a couple of reasons for this error. You can try the following things to solve it.

- configureAccessGate (check the file ../AccessServerSDK/oblix/lib/ObAccessClient.xml)

- cp AccessServerSDK/oblix/lib/jobaccess.jar to $ORACLE_HOME/j2ee/home/lib/ext

- in the LD_LIBRARY_PATH in opmn.xml file, please remove the trailing slash "/" from the patch "/AccessServerSDK/oblix/lib/

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Determine OAS (10.1.3) version

$ORACLE_HOME/OPatch/opatch lsinventory -detail

Get version Oracle Containers for J2EE

cd $ORACLE_HOME/j2ee/home
java -jar oc4j.jar -version

Output:
Oracle Containers for J2EE 10g (10.1.3.5.0) (build 090727.2000.36696)

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

SOA BPEL (10.1.3.x) Identity service: java.security.PrivilegedActionException: javax.xml.soap.SOAPException: Message send failed: Connection timed out

The reason for this error:
"java.security.PrivilegedActionException: javax.xml.soap.SOAPException: Message send failed: Connection timed out:"
was that an proxy server was configured on the oc4j_soa container in opmn.xml

BPEL Console gave this error:
The following exception occured while processing this request:
Io exception: The Network Adapter could not establish the connection

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Delete of bpel domain on SOA 10.1.3.4 is not complete

When a BPEL domain is delete from the BPELAdmin there are still records left in de dehydration database. These entries can be deleted:

Logon on the dehydration store database as ORABPEL

Check which domains are active:
select domain_ref from domain;

Delete all other domains:

delete suitcase_bin where domain_ref not in (select domain_ref from domain);
delete audit_details where domain_ref not in (select domain_ref from domain);
delete audit_trail where domain_ref not in (select domain_ref from domain);
delete cube_instance where domain_ref not in (select domain_ref from domain);
delete cube_scope where domain_ref not in (select domain_ref from domain);
delete dlv_message where domain_ref not in (select domain_ref from domain);
delete dlv_subscription where domain_ref not in (select domain_ref from domain);
delete document_ci_ref where domain_ref not in (select domain_ref from domain);
delete document_dlv_msg_ref where domain_ref not in (select domain_ref from domain);
delete domain_properties where domain_ref not in (select domain_ref from domain);
delete invoke_message where domain_ref not in (select domain_ref from domain);
delete process where domain_ref not in (select domain_ref from domain);
delete process_default where domain_ref not in (select domain_ref from domain);
delete process_descriptor where domain_ref not in (select domain_ref from domain);
delete process_log where domain_ref not in (select domain_ref from domain);
delete wi_fault where domain_ref not in (select domain_ref from domain);
delete work_item where domain_ref not in (select domain_ref from domain);
delete xml_document where domain_ref not in (select domain_ref from domain);
delete wftask where domainid='[NOT EXISTING DOMAINS]';
delete wftaskhistory where domainid='[NOT EXISTING DOMAINS]';
delete wftaskmetadata where domainid='[NOT EXISTING DOMAINS]';
delete wftask where domainid='[NOT EXISTING DOMAINS]';

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Oracle OMS agent: Rediscover targets

In the AGENT_HOME execute agentca -d

$ORACLE_HOME/bin/agentca -d

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

A problem occured while connecting to server "hostname" using port "80": java.security.AccessControlException: access denied (com.collaxa.security.Dom

In our environment the appications hw_services/orabpel and deploy_service in the oc4j_soa container are integrated with the OID. After change the password of oc4jadmin we got problems with the automatic deployment of BPEL Processes. See the stack trace below.

We found out that the following line: was missing in the $ORACLE_HOME/j2ee/oc4j_soa/application_deployments/deploy_service/orion-application.xml.

BUILD FAILED
/oracle/opleveringen/DDSAddressAdapter/1.2/1.1/DeploymentMijnLoket/build.xml:170: The following error occurred while executing this line:
/oracle/opleveringen/DDSAddressAdapter/1.2/1.1/DeploymentMijnLoket/build.xml:127: The following error occurred while executing this line:
/oracle/opleveringen/DDSAddressAdapter/1.2/1.1/DeploymentMijnLoket/build.xml:102: The following error occurred while executing this line:
/oracle/opleveringen/DDSAddressAdapter/1.2/1.1/JDeveloper/DDSAdapter/DDSAddressService/bpel/build.xml:89: A problem occured while connecting to server "hostname" using port "80": java.security.AccessControlException: access denied (com.collaxa.security.DomainPermission XXXX read)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:264)
at java.security.AccessController.checkPermission(AccessController.java:427)
at com.collaxa.security.OC4JSecurityService.checkAccess(OC4JSecurityService.java:16)
at com.collaxa.security.SecurityService.checkDomainAccess(SecurityService.java:26)
at com.collaxa.cube.fe.util.ServletUtils.getLocatorWithoutUrlRewrite(ServletUtils.java:77)
at _deployHttpClientProcess._jspService(_deployHttpClientProcess.java:332)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:473)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at oracle.security.jazn.oc4j.JAZNFilter$1.run(JAZNFilter.java:400)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:414)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:644)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:908)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Error during deployment war: Incompatible argument to function

After we changed the JDK of the OC4J container from 1.5 to 1.6 we had problems deploying applications.

BUILD FAILED
build.xml:115: Deploy error: Operation failed with error:
(class: xx/xx/xxx , method: YyyyyY signature: (Ljava/util/Map;)RRR/rrr/RRR;) Incompatible argument to function

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Apache Address already in use: make_sock: could not bind to port xxxx

Error in the Apache error.log

Apache Address already in use: make_sock: could not bind to port xxxx


Find the process which is using the port:
netstat -tulpn| grep :xxxxx

Kill the process:

kill -9 httpd

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

ESB Console: no systems. Error in log.xml: WSDL parsing failed : WSDL Parsing Failed: Connection timed out

In the metalink note: 467440.1 a possible solution is provided but in our case this was not working. We had a big application which was using Oracle BPEL en Oracle ESB. In the new release the Oracle ESB was replaced by another ESB (mule) but the old ESB systems were still deployed in the Oracle ESB. The ESB Console was not able to retrieve the ESB systems anymore, so everything was empty in the screen. The only error in log.xml in $ORACLE_HOME/j2ee/oc4j_soa/log/oc4j/.../log.xml was

WSDL parsing failed :WSDL Parsing Failed: Connection timed out

We did an undeploy of the esb system with ant and this solved the problem.

This is how we did the undeployment:

build.xml:

<?xml version="1.0" encoding="windows-1252" ?>
<project name="deploy" default="undeploy-esb">

  <property file="build.properties"/>

  <import file="../etc/ESBMetadataMigrationTaskdefs.xml"/>
  <target name="undeploy-esb">

      <undeployESBEntities
            esbMetadataServerHostname="[HOSTNAME]"
            esbMetadataServerPort="[PORT]"
            userName="oc4jadmin"
            password="[PASSWORD]">
         <system guid="[ESBSYSTEM GUID]"/>
      </undeployESBEntities>

  </target>

</project>

[ESBSYSTEM GUID] can be found in de dehydration store of the SOA suite:

select name, guid from oraesb.wf_systems;

You need some extra files to make ant work, these files can be found in $ORACLE_HOME/integration/esb/deployment there is an zip file: documentation.zip. In this zip file the files you need are there.

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Oracle Application Server 10.1.2.3: oidldapd process, high load on server

On the production environment of the customer there was a problem that the servers sometimes did not respond on anything anymore. The oidldapd process was consuming 100% CPU and the load on the servers was very high (100+). The only thing we could do was an hard reboot of the server.

We solved this issue bij tuning the OID. We replaced the default index by a bitmap index:

To recreate the ei_attrstore index perform the following steps: 

a.  Use sqlplus to connect to the OID database as user "ods" or connect as system and set the schema to ods, e.g. 

$ sqlplus system/ 
SQL> alter session set current_schema=ods; 

b.  Drop the existing ei_attrstore index: 

SQL> drop index ei_attrstore; 

c.  Recreate the ei_attrstore index: 

SQL> create bitmap index ei_attrstore on ds_attrstore(entryid, attrname) tablespace olts_attrstore nologging compute statistics; 

SQL> alter index ei_attrstore noparallel;

And we changed the configset0

$ORACLE_HOME/bin/ldapmodify -h [SERVERNAME] -p [PORT] -D cn=orcladmin -w [PASSWORD] -v <<EOF
dn: cn=configset0,cn=osdldapd,cn=subconfigsubentry
changetype: modify
replace: orclmaxcc
orclmaxcc: 10
-
replace: orclserverproces
orclserverprocs: 2
EOF

After these changes the oidldapd did not have the strange behavior anymore.

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Oracle Application Server 10.1.3 / JDBC connections based on tnsnames.ora

The customer wants to use DataGuard as backend for the applications that are hosted on the Oracle Application Server 10.1.3. To implement this solution it is necessary to configure the JDBC connections with the tnsnames.ora, because with this solution it is possible to put more databases (on different servers) in one JDBC connection.

Oracle has an java program to check if this connection is working:

// -- begin : cut here : ThinTnsnames.java -- cut there
import java.sql.*;
import oracle.jdbc.pool.*;

public class ThinTnsnames {
  // private static OracleConnection connection = null;
  static String sql = null;

  public static void main(String[] args) {
    String entry_name = args [0];
    String userId = args [1];
    String password = args [2];
    test (entry_name, userId, password);
  }

  public static void test (String entry_name, String userId, String password) {
    Connection pconnection = null;
    try {
      String l_url = "jdbc:oracle:thin:@" + entry_name;
      System.out.println( " Connection string = " + l_url );

      OracleDataSource ods = new OracleDataSource();
      ods.setUser(userId);
      ods.setPassword(password);
      ods.setURL(l_url);
      pconnection = ods.getConnection ();
      version (pconnection );
    }
    catch(SQLException e) {
      e.printStackTrace();
    }
    finally {
      try {
        if (pconnection != null )pconnection .close();
      }
      catch(Exception e) {
        e.printStackTrace();
      }
    }
  }

  public static void version (Connection pconnection) {
    try {
      DatabaseMetaData dmd = pconnection.getMetaData();
      System.out.println("DriverVersion: ["+dmd.getDriverVersion()+"]");
      System.out.println("DriverMajorVersion: ["+dmd.getDriverMajorVersion()+"]");
      System.out.println("DriverMinorVersion: ["+dmd.getDriverMinorVersion()+"]");
      System.out.println("DriverName: ["+dmd.getDriverName()+"]");
      System.out.println("URL: ["+dmd.getURL()+"]");
      System.out.println("UserName: ["+dmd.getUserName()+"]");
      System.out.println(dmd.getDatabaseProductName() );
      System.out.println(dmd.getDatabaseProductVersion() );

      String ver;
      ver = System.getProperty("java.version");
      System.out.println("The JDK version is " + ver);
    }
    catch(SQLException e) {
      e.printStackTrace();
    }

  }
}

// --end : cut here : ThinTnsnames.java -- cut there

compile the program:
javac -classpath ojdbc14.jar:. ThinTnsnames.java

Execute the program:
java -Doracle.net.tns_admin=$ORACLE_HOME/network/admin -classpath .:ojdbc14.jar ThinTnsnames [DATABASE] [USER] [PASSWORD]

Output:

 Connection string = jdbc:oracle:thin:@[DATABASE]
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 221
        at oracle.net.nl.NVTokens.parseTokens(Unknown Source)
        at oracle.net.nl.NVFactory.createNVPair(Unknown Source)
        at oracle.net.nl.NLParamParser.addNLPListElement(Unknown Source)
        at oracle.net.nl.NLParamParser.initializeNlpa(Unknown Source)
        at oracle.net.nl.NLParamParser.(Unknown Source)
        at oracle.net.resolver.TNSNamesNamingAdapter.loadFile(Unknown Source)
        at oracle.net.resolver.TNSNamesNamingAdapter.checkAndReload(Unknown Source)
        at oracle.net.resolver.TNSNamesNamingAdapter.resolve(Unknown Source)
        at oracle.net.resolver.NameResolver.resolveName(Unknown Source)
        at oracle.net.resolver.AddrResolution.resolveAndExecute(Unknown Source)
        at oracle.net.ns.NSProtocol.establishConnection(Unknown Source)
        at oracle.net.ns.NSProtocol.connect(Unknown Source)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)
        at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:441)
        at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:165)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
        at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:297)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:221)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:165)
        at ThinTnsnames.test(ThinTnsnames.java:26)
        at ThinTnsnames.main(ThinTnsnames.java:13)

I could not find the problem for this issue. But after logging an SR on metalink the problem was the tnsnames.ora file.

We had one database entry in the tnsnames.ora:

[DATABASE] =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = [SERVERNAME] l)(PORT = 1521)))
(CONNECT_DATA = (SERVICE_NAME = [DATABASE] )(SERVER = DEDICATED)))

But this entry was not correct.

We changed the tnsnames.ora:

[DATABASE] =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = [SERVERNAME] )(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = [DATABASE])
)
)

And now execute the program:

java -Doracle.net.tns_admin=$ORACLE_HOME/network/admin -classpath .:ojdbc14.jar ThinTnsnames [DATABASE] [USER] [PASSWORD]

Connection string = jdbc:oracle:thin:@[DATABASE]
DriverVersion: [10.2.0.4.0]
DriverMajorVersion: [10]
DriverMinorVersion: [2]
DriverName: [Oracle JDBC driver]
URL: [jdbc:oracle:thin:@[database]]
UserName: [USER]
Oracle
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
The JDK version is 1.5.0_06

If this program is working it is also possible to make a JDBC connection in enterprise managers like this:
jdbc:oracle:thin:@[DATABASE]

In the start parameters of the OC4J container there should be an extra option:
-Doracle.net.tns_admin=[TNSNAMES LOCATION]

Note:
It is also possible to configure the JDBC connection like an tnsnames.ora entry in the enterprise manager:

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(failover=on)(ADDRESS=(PROTOCOL=TCP)(HOST=[SERVERNAME1])(PORT=1521))(address=(protocol=tcp)(host=[SERVERNAME2])(port=1521)))(CONNECT_DATA=(SERVICE_NAME=[DATABASE])))

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Upgrade Default JDK to new version Oracle Application Server 10.1.3

Metalink note: 396096.1]

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

upgrade 10.1.2.x to 10.1.2.3: Oracle Internet Directory Patch Configuration Assistant fails

When I upgraded the infra 10.1.2.x to 10.1.2.3 I got an failed on Oracle Internet Directory Patch Configuration Assistant. I found out that in the $ORACLE_HOME/ldap/log/patchca.log there was an error:

Wed Mar 10 09:51:41 CET 2010OID PatchCA started..
Check if OID is configured.
oracle.sysman.assistants.util.NetAPIException: TNS-04404: no error
  caused by: oracle.net.config.ConfigException: TNS-04414: File error
  caused by: TNS-04612: Null RHS for "oaiinfra"
        at oracle.sysman.assistants.util.NetworkUtils.getNetServiceNameValue(NetworkUtils.java:2241)
        at oracle.ldap.oidinstall.OIDPatchCA.run(OIDPatchCA.java:255)
        at oracle.ldap.oidinstall.OIDPatchCA.main(OIDPatchCA.java:443)

After some investigation I found out that the tnsnames.ora was not correct. We changed this an now everything is working.

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Make Access Server debug log (oblog.log) more readable

tail -f oblog.log | sed 's/\%25253d/=/g' | sed 's/\%253d/=/g' | sed 's/\%20/ /g' | sed 's/\%3d/=/g' | sed 's/\%2520/ /g' | sed 's/\%2525253/=/g' | tr ' ' '\n'


Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

After CPU patch OC4J containers will not start anymore

We applied a CPU patch an Oracle Infrastructure 10.1.2.3 and after that the OC4J containers did not start anymore.

In de $ORACLE_HOME/ldap/log/oidldapd01.log the following error:

Bind failed on communication endpoint (13)  

With an opmnctl status the OID seems to be up.

When this problem occurs you need to execute root.sh.

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

The requested resource could not be mapped to a policy domain in the Policy database. Check if the corresponding directory service is up

In the browser we got an 401 Unauthorized error when we tried to start an application behind Oracle Access manager.

In the log.xml in $ORACLE_HOME/j2ee/[OC4J_CONTAINER]/log/[OC4J_CONTAINER/oc4j/ the error:

[CoreIDLoginModule]: Could not authentication user using ObSSOCookie token. Exception: [CoreIDLoginModule]: Could not fetch groups from Access Server. Exception: The requested resource could not be mapped to a policy domain in the Policy database. Check if the corresponding directory service is up.

In the SDK log directory $SDK_HOME/oblix/logs

2010/01/29@07:24:43.738335      7278    7278    ACCESS_SDK      ERROR   0x00001830      /usr/abuild/Oblix/10142hf/palantir/access_api/src/obresource_request.cpp:715  "The requested resource could not be mapped to a policy domain in the Policy database. Check if the corresponding directory service is up."   raw_code^305

I forgot to create the policy for /myresourceurl in the policy manager.



Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

java.lang.OutOfMemoryError: PermGen space

Nice blog post about Perm Gen

http://oraclebpelindepth.blogspot.com/2009/04/javalangoutofmemoryerror-permgen-space.html
http://www.freshblurbs.com/explaining-java-lang-outofmemoryerror-permgen-space

Install WebGate 10.1.4.2.3 on Apache2 RHEL 5.x 64bit

Download the WebGate software from:
http://download.oracle.com/otn/linux/ias/101401/oam_int_linux_v8_cd1.zip

Login with putty on the server:
su - root

mkdir -p $WEBGATE_HOME/gcc

cp /lib64/libgcc_s.so.1 /appl/oracle/access_10.1.4/gcc/
cp /usr/lib64/libstdc++.so.5 /appl/oracle/access_10.1.4/gcc/

unzip oam_int_linux_v8_cd1.zip

./Oracle_Access_Manager10_1_4_2_3_linux64_APACHE22_WebGate

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Identity Service Authentication failure.

We had an problem in de $ORACLE_HOME/opmn/logs/soa_group....log

<2010-01-13 10:19:05,694> <ERROR> <ari.collaxa.cube.services> <PCException::<init>> Identity Service Authentication failure.
<2010-01-13 10:19:05,695> <ERROR> <ari.collaxa.cube.services> <PCException::<init>> Identity Service Authentication failure.
<2010-01-13 10:19:05,695> <ERROR> <ari.collaxa.cube.services> <PCException::<init>> Check the error stack and fix the cause of the error. Contact oracle support if error is not fixable.

Our SOA is integrated with the OID.

The problem was that the account of the oc4jadmin was locked out in the OID.

This can be tested by starting oidadmin and try to login with cn=oc4jadmin, cn=users, dc=oracle, dc=com.

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.

Tuning Oracle Access Manager

Metalink:

https://support.oracle.com/CSP/main/article?cmd=show&id=404902.1&type=NOT

Did this post help you in any way can you please leave a comment? This will motivate me writing more posts.