Tuesday, June 23, 2015

Accessing Weblogic Console gives 404 and Log4JLogger cannot be found

Environment / assumptions in this post:

  • Weblogic 10.3.6
  • Enterprise Manager 11.1.1.6
  • Fusion Middleware domain with 11.1.1.6 ADF libraries

I recently came across this problem on one of our 11.1.1.6 development environments, Weblogic version 10.3.6.  The Enterprise Manager was working fine, but going to the WL console gave a 404. For sure EM is used more often, but we would occasionally use the Console to configure Providers in the Security Realm and a few other things. Very strange. A bounce of the Admin Server sometimes would fix unexplained problems, but not this time.

Checking the Admin Server logs revealed this error:

Caused By: org.apache.commons.logging.LogConfigurationException: User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is not useable.
        at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)

So there would appear to be a conflict with logging classes of some kind, or something was missing. In the end, the fix was pretty simple.  The weblogic-application.xml of the console application had to be modified to include the packages that were missing. How and why they went missing is a mystery so far, could be the result of security patching or some other system updates.   If it happens again, it will be interesting to see if the problem and solution is exactly the same or not.

What we did to fix it - we navigated to the console's weblogic-application.xml file located here:

<weblogic_home>/server/lib/consoleapp/META-INF/

and added the following to the beginning of the  <prefer-application-packages> section:

    <package-name>org.apache.log4j.*</package-name>
    <package-name>org.apache.commons.*</package-name>

And then restart the Admin Server. The Console should be available again.

Saturday, May 9, 2015

Connecting JConsole to a remote server two ways

Update: JVisualVM is also very nice, I talk about it here on a post Using JVisualVM to connect to a remote Weblogic 12c Managed Server.

Environment / assumptions for this post:
  • Weblogic 12c Managed Server(s)
  • jdk7 / 8
  • Cygwin on Windows 7

This post is about connecting JConsole to a remote JVM. This can be useful for gathering some basic memory and thread details, information about the JVM itself such as  version and classpath, and even control any MBeans defined.

JConsole memory details
JConsole MBean control

The following setup is using a local Windows 7 setup with cygwin, and remote linux server. It was using jdk7, although jdk6 or jdk8 should also work.

There are two general ways to run JConsole to connect to a remote JVM:

  1. Run local JConsole, and connect to JVM via port forwarding through ssh. In this case local means running JConsole off your local JDK. This has the advantage of being the faster option (faster setup and also faster application response), but has the disadvantage of only being able to connect to JVMs  you explicitly set up i.e. you need to add extra JVM options, which means a bounce. Not usually feasible for Production systems.
  2. Run remote JConsole, but display on local display i.e. forward X11. This has the advantage that you can pick any JVM running on the server by PID, and there is no extra setup or JVM parameters needed. This has the disadvantage of being harder to setup and also being very slow.

Option 1 - Local JConsole

Summary of general steps:
  1. Setup JVM with proper params.
  2. Connect with ssh and forward the appropriate port.
  3. Start JConsole locally, and connect.
Step 1 details - setup JVM with proper params

There are a few needed JVM params that will make this work. With them, we can specify which port to connect to (pick an unused one), which security files to use (if any), and to use SSL or not.

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=50500
-Dcom.sun.management.jmxremote.ssl=false 

If you want to setup some security, then that is also possible. The standard roles are "controlRole" and "monitorRole", you can use these or make up your own. The relevant files are found under JRE_HOME/lib/management, and are:

jmxremote.access - defines roles and their access e.g. 'controlRole' = read/write
jmxremote.password - password for the roles defined in the access file
jmxremote.password.template - copy this file somewhere and protect it
management.properties - specify lots of options in here, including those above

The idea is that you copy the template file to you home directory or somewhere else, then set the permissions so only you can read it. You can also modify the management.properties mentioned above and point to it.  Then, you point to them with two additional options:

-Dcom.sun.management.config.file=your_management.properties_file
-Dcom.sun.management.jmxremote.password.file=your_password_file

Step 2 details - connect with ssh and forward the specified port

Open a cygwin window and ssh to your server with the following options:

ssh oracle@my_server -L 50500:localhost:50500

Or you can, if already connected, hit a quick ~C (that is tilde, then capital letter C), then:

ssh> -L 50500:localhost:50500

Step 3 details - Start JConsole locally and connect

If you have the env variable JAVA_HOME setup, plus JAVA_HOME\bin on your PATH, then you should be able to just type jconsole from a cmd window, or hit cmd+r and type jconsole.

Once it starts, enter the JMX connect string in the Remote Process box. Use localhost, and the port that you chose for forwarding.
service:jmx:rmi:///jndi/rmi://localhost:50500/jmxrmi

JConsole - Remote Process connection string with no security

If you added security, then add the username (e.g. controlRole) and password as well.

For more information, you can check this Oracle page - Monitoring and Management using JMX.

Option 2 - Remote JConsole

Step 1

From a cygwin window, start your X server. This is usually with startxwin, or startx. The X11 system is a topic for many books, so I'll say no more.  Check this Cygwin page for some documentation. An xterm window should open.

Update Note: If you are so lucky :)  to be running a Linux desktop, your job is even easier, and you can skip this step. If you want to try running a Linux desktop (especiall Oracle Enterprise Linux), check my post Coexisting at work with Oracle Enterprise Linux (OEL 7.2) and Microsoft Windows.

Step 2

Connect to the server with ssh. You have to make sure that X11 forwarding is enabled for this to work. See the last half of my post about configuring ssh. To check if it is working properly, you can try running xclock, it should display a small analog clock on your monitor.
Note: below, 'dm' is an alias for our server, it stands for  'dev middleware'. You set this up in the ssh config file, which is where you also configure the X11 forwarding.

$ ssh dm
Enter passphrase for key '/home/jjames/.ssh/id_rsa': 
Last login: Tue Apr 28 11:07:41 2015 from mybox.rcpsc.edu
[Tue Apr 28 11:10:15 oracle@devbox:~ ] $ xclock &
[1] 18987
[Tue Apr 28 11:11:01 oracle@devbox:~ ] $ 

Step 3

Now, you want to ensure that the environment variable JAVA_HOME is set, and that PATH contains JAVA_HOME/bin. This is where JConsole lives. You can try running JConsole at this point, although we need to find the process id first in the next step.

[Tue Apr 28 11:24:30 oracle@devbox:~ ] $ export JAVA_HOME=/u01/app/oracle/product/jdk
[Tue Apr 28 11:24:40 oracle@devbox:~ ] $ export PATH=$JAVA_HOME/bin:$PATH
[Tue Apr 28 11:25:01 oracle@devbox:~ ] $ jconsole &
[2] 22183
[Tue Apr 28 11:25:05 oracle@devbox:~ ] $ 

Step 4

Using ps command, find the PID of the JVM you are interested in.

[Tue Apr 28 11:41:46 oracle@devbox:~ ] $ ps -ef | grep oam_server1
oracle   13050     1  0 Apr21 ?        00:00:00 /bin/sh ./startManagedWebLogic.sh oam_server1
oracle   13196 13051  0 Apr21 ?        01:05:33 /u01/app/oracle/product/jdk/bin/java -client -Xms512m

Step 5

Run JConsole, select the PID.

JConsole with a list of remote PIDs

...and clickety-boo, you should have your connection! It may take a minute, and you may get an SSL error, but if so it will ask you if you want to retry without SSL.






Friday, April 10, 2015

The Story Behind the Name

Ham sandwiches and coffee, at least together, are not my favourite thing. Individually, great.  But neither of those have anything to do with the name of this blog. The java of course is Java. The Ham part? Well there used to be a manager (at my current place of employment) that used to say "this place could complicate a ham sandwich".  So it is very much a Dilbertesque sort of thing.

I suppose "java sandwich" could also be an allegory to layers of complexity in the world of software development. This would also be true, but it is the tendency to unnecessarily complicate things that don't need to be complicated that really inspires the name. Plus it is sort of silly, which appeals to the Monty Python fan in me. Also, every other domain name was taken. Almost literally.