oracle_oud_troubleshooting [My wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
=====Overview=====
In my humble opinion, weblogic is full of bugs :) 
So let's see some common of them.


=====Cannot start due to missing identity file=====
if you have missing identity file when you try to start the NodeManager as follows:

<Code:bash|Trying to start Nodemanager>
[oracle@oud bin]$ ./startNodeManager.sh
CLASSPATH=/home/oracle/jdk1.8.0_301/lib/tools.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/server/lib/weblogic.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/modules/thirdparty/ant-contrib-1.0b3.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/modules/features/oracle.wls.common.nodemanager.jar::/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/..:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/modules/features/oracle.wls.common.grizzly.jar
+ /home/oracle/jdk1.8.0_301/bin/java -server -Xms32m -Xmx200m -Djdk.tls.ephemeralDHKeySize=2048 -Dweblogic.RootDirectory=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/common/nodemanager -Dcoherence.home=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../coherence -Dbea.home=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/.. -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.security.policy=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/home/oracle/jdk1.8.0_301 weblogic.NodeManager -v
<Nov 7, 2021 2:50:50 PM UTC> <INFO> <Loading domains file: /opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.domains>
<Nov 7, 2021 2:50:50 PM UTC> <WARNING> <Domains file not found: /opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.domains>
<Nov 7, 2021 2:50:51 PM UTC> <INFO> <Loading identity key store: FileName=/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/security/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Nov 7, 2021 2:50:51 PM UTC> <SEVERE> <Fatal error in NodeManager server>
weblogic.nodemanager.common.ConfigException: Identity key store file not found: /opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/security/DemoIdentity.jks
        at weblogic.nodemanager.server.SSLConfig.loadKeyStoreConfig(SSLConfig.java:225)
        at weblogic.nodemanager.server.SSLConfig.access$000(SSLConfig.java:33)
        at weblogic.nodemanager.server.SSLConfig$1.run(SSLConfig.java:118)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLConfig.<init>(SSLConfig.java:115)
        at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:169)
        at weblogic.nodemanager.server.NMServer.getInstance(NMServer.java:134)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:589)
        at weblogic.NodeManager.main(NodeManager.java:31)
</Code>

Then we have to generate again the SSL certificate and the identity file as follows:

1. Export correct variables:

<Code:bash|Export Variables: >
[oracle@oud bin]$ . setWLSEnv.sh
CLASSPATH=/home/oracle/jdk1.8.0_301/lib/tools.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/modules/features/cieCfg_cam_lib.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/modules/features/wlst.wls.classpath.jar:

PATH=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/server/bin:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/modules/thirdparty/org.apache.ant/1.10.5.0.0/apache-ant-1.10.5/bin:/home/oracle/jdk1.8.0_301/jre/bin:/home/oracle/jdk1.8.0_301/bin:/home/oracle/jdk1.8.0_301/bin:/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/modules/org.apache.maven_3.2.5/bin

Your environment has been set.
[oracle@oud bin]$
</Code>


2. We can generate the SSL and the identity file as follows:

<Code:bash|Re-generate the files>
[oracle@oud security]$ pwd
/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/security
[oracle@oud security]$ java utils.CertGen -keyfilepass DemoIdentityPassPhrase -certfile democert -keyfile demokey -strength 2048 -noskid
Generating a certificate with common name oud and key strength 2048
issued by CA with certificate from /opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/server/lib/CertGenCA.der file and key from /opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/server/lib/CertGenCAKey.der file
[oracle@oud security]$ 

[oracle@oud security]$ java utils.ImportPrivateKey -keystore DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase -keyfile demokey.pem -keyfilepass DemoIdentityPassPhrase -certfile democert.pem -alias demoidentity
No password was specified for the key entry
Key file password will be used
<Nov 7, 2021 3:38:33 PM UTC> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>
<Nov 7, 2021 3:38:33 PM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>

Imported private key demokey.pem and certificate democert.pem
into a new keystore DemoIdentity.jks of type jks under alias demoidentity
[oracle@oud security]$
[oracle@oud security]$
</Code>


3. We can check if everything is created:

<Code:bash|Verify all has been created>
[oracle@oud security]$ ls -lart
total 24
-rw-r-----. 1 oracle oinstall   80 Nov  7 14:25 SerializedSystemIni.dat
drwxr-x---. 3 oracle oinstall   45 Nov  7 15:37 ..
-rw-r--r--. 1 oracle oinstall 1260 Nov  7 15:38 demokey.der
-rw-r--r--. 1 oracle oinstall 1781 Nov  7 15:38 demokey.pem
-rw-r--r--. 1 oracle oinstall  923 Nov  7 15:38 democert.der
-rw-r--r--. 1 oracle oinstall 1306 Nov  7 15:38 democert.pem
-rw-r--r--. 1 oracle oinstall 2282 Nov  7 15:38 DemoIdentity.jks
drwxr-x---. 2 oracle oinstall  139 Nov  7 15:38 .
[oracle@oud security]$
</Code>


4. After that, we can start the node manager normally as follows:

<Code:bash|Starting Node manager>
[oracle@oud bin]$ nohup ./startNodeManager.sh > nm.out&
[1] 6328
[oracle@oud bin]$ nohup: ignoring input and redirecting stderr to stdout

[oracle@oud bin]$
[oracle@oud bin]$
[oracle@oud bin]$ tail -100f nm.out
CLASSPATH=/home/oracle/jdk1.8.0_301/lib/tools.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/server/lib/weblogic.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/modules/thirdparty/ant-contrib-1.0b3.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/modules/features/oracle.wls.common.nodemanager.jar:/home/oracle/jdk1.8.0_301/lib/tools.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/modules/features/cieCfg_cam_lib.jar:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/modules/features/wlst.wls.classpath.jar::/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/..:/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/modules/features/oracle.wls.common.grizzly.jar
+ /home/oracle/jdk1.8.0_301/bin/java -server -Xms32m -Xmx200m -Djdk.tls.ephemeralDHKeySize=2048 -Dweblogic.RootDirectory=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../oracle_common/common/nodemanager -Dcoherence.home=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/../coherence -Dbea.home=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/.. -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.security.policy=/opt/oracle/Oracle/Middleware/Oracle_Home/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/home/oracle/jdk1.8.0_301 weblogic.NodeManager -v
<Nov 7, 2021 3:43:42 PM UTC> <INFO> <Loading domains file: /opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.domains>
<Nov 7, 2021 3:43:42 PM UTC> <WARNING> <Domains file not found: /opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.domains>
<Nov 7, 2021 3:43:43 PM UTC> <INFO> <Loading identity key store: FileName=/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/security/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Nov 7, 2021 3:43:43 PM UTC> <WARNING> <NodeManager configuration properties file '/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.properties' not found. Using default settings.>
<Nov 7, 2021 3:43:43 PM UTC> <INFO> <Saving NodeManager configuration properties to '/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.properties'>
Node manager v12.2.1.4.0

Configuration settings:

DomainsFile=/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.domains
LogLimit=0
DomainsDirRemoteSharingEnabled=false
AuthenticationEnabled=true
LogLevel=INFO
DomainsFileEnabled=true
ListenAddress=
NativeVersionEnabled=true
ProcessDestroyTimeout=20000
ListenPort=5556
LogToStderr=true
weblogic.StartScriptName=startWebLogic.sh
SecureListener=true
LogCount=1
LogAppend=true
weblogic.StopScriptEnabled=false
StateCheckInterval=500
CrashRecoveryEnabled=false
weblogic.StartScriptEnabled=true
LogFile=/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
coherence.StartScriptEnabled=false
ListenBacklog=50
NodeManagerHome=/opt/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/nodemanager
RestEnabled=false
isLogLockingEnabled=true
weblogic.startup.JavaHome=/home/oracle/jdk1.8.0_301
weblogic.startup.MW_Home=
coherence.startup.JavaHome=/home/oracle/jdk1.8.0_301
coherence.startup.MW_Home=

Domain name mappings:


<Nov 7, 2021 3:43:43 PM UTC> <INFO> <12.2.1.4.0>
<Nov 7, 2021 3:43:43 PM UTC> <INFO> <Server Implementation Class: weblogic.nodemanager.server.NMServer$ClassicServer.>
<Nov 7, 2021 3:43:44 PM UTC> <INFO> <Secure socket listener started on port 5556>
</Code>