JDeveloper 12.2.1.0 Integrated Weblogic - logging level doesn't change when value deleted

Environment / assumptions for this post:

• JDeveloper 12.2.1
• Integrated Weblogic server
• Windows 7

In a nutshell: blanking out logging level in the Diagnostic Logging Congifuration doesn't work.

When working with the Integrated Weblogic server in JDeveloper, it is common to increase logging for a particular class or package when troubleshooting. Typically I would set a level of FINEST for something like oracle.jbo package (very useful to discover the SQL being generated by ViewObjects and their bind parameters).

After I finished my troubleshooting, I would decrease the logging level again. However, the logging continued even though I "blanked" it out. This used to work on 11g versions of JDev / Weblogic. With 12c, you must actually set it to something. So in my case I just set it to "INFO" and that did the trick.

Useful Linux commands

It's amazing what you can do on the command line in a Linux environment - a combination of find, grep, sed, awk and others will do a lot to help with troubleshooting. There's not a day that goes by where I don't have to do something "interesting". There isn't anything you cannot do with a combination of the right commands it seems.  Here's some of the ones I use every so often.

• ⇩ Move or copy files in place
• ⇩ Find files larger than XX size
• ⇩ Extract lines from the middle of a file
• ⇩ Adding a column of numbers
• ⇩ Add the total size of a particular file type when you don't have the file ending
• ⇩ Counting docs by mime type
• ⇩ Counting docs by mime type - human readable version
• ⇩ Finding a class inside hierarchy of jar files
• ⇩ Find the files open for writing by a process e.g. log files
• ⇩ Find something in a file, but showing the previous line

NOTE: in my case these all function for Oracle Enterprise Linux v7. Your *nix version (or shell) may require slightly different syntax.

Move or Copy files in place
This will expand to: mv myfilename myfilename.old

mv myfilename{,.old}

Find file larger than 500Mb
You may have to adjust the field numbers. This will print out the size (field #5) and filename (field #9).

find . -type f -size +500000k -exec ls -lh {} \; | awk '{ print $5 ": " $9 }' 

Extract lines from the middle of a file

tail -n +startinglinenum filename | head -n numlines

Add a column of numbers
Here, we are adding the total size of deleted files whose handle is still active. In my case, field #8 was the file size.

/usr/sbin/lsof | grep "deleted" | awk '{sum += $8} END {print sum/1024/1024,"Mb"}'

Add total size of all files of a particular type, when you don't have file ending
For example, all PDF files. Our Alfresco doc repository has all binaries named with ".bin", so we cannot simply filter by filename ending.

find . -type f -exec file -in {} \; | grep zip | cut -d ':' -f 1 | xargs ls -l | awk '{ sum += $5 } END { print sum/1024/1024,"Mb" }'

Count docs by mime type

find ./foldername -type f -exec file -inb {} \;| sort |uniq -c|sort -nr

You'll get output something like this:
     20 application/pdf
      6 application/x-zip
      3 text/plain; charset=us-ascii
      3 image/png
      2 application/octet-stream
      1 application/msword

Count docs by file type - same as previous, more human readable

find . -type f | xargs file | cut -d ':' -f 2 | sed 's/^ *//' | sort | uniq -c | sort -nr

Now the output is like this:
     13 PDF document, version 1.5
      6 Zip archive data, at least v2.0 to extract
      5 PDF document, version 1.6
      3 PNG image data, 77 x 100, 16-bit/color RGBA, non-interlaced
      2 PDF document, version 1.7
      2 data
      1 Microsoft Office Document
      1 ASCII text, with no line terminators
      1 ASCII English text, with very long lines
      1 ASCII English text, with no line terminators

Finding a class file inside jar files, then ignoring 1 or more folders

for x in $(find . -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

for x in $(find . -path ./blah -prune -o -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

for x in $(find . \( -path blah -o -path woof \) -prune -o -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

Finding the files open for writing by a process
e.g. what log files does a process write to? The 0-9 is the file descriptor, the uw is either updatable or writeable, the REG is regular file. Look for handles 1 and 2, they are stdout and stderr.
Note: you may have to be a privileged user to see some of this information.

lsof -p PID | egrep ' [0-9]+[uw] ' | grep 'REG'

Grepping for something in a file, but extracting the previous line for each
I like to specify to show only interesting columns. In this case, the columns for this particular type of log showed date, user, command, etc. Otherwise there can be a lot of noise that gets in the way.

grep -B 1 -E 'NoClassDefFoundError' diagnostic.log | cut -d ' ' -f 1,2,5,13,14,20

Setting up JDeveloper and Weblogic Integrated Server as 64 bit on Windows 7

Update: this applies to Windows 8/10 just as well, as well as JDeveloper 12c. As well, with JDK7 there is no permgen memory anymore.

I never really bothered with 64bit JDev and WLS, even though my work station is 64bit Windows 7, until we started deploying taskflows to WebCenter Portal. Just those extra few libraries and classes were causing memory shortages. I would boost the Heap, then run out of PermGen. I would then boost PermGen a bit and reduce Heap a little bit, still problems. The thing is, I was running into the 3Gb limit of 32 bit architecture. So, I decided to muss around and get the 64 bit JDev and WLS running. It was a perfect opportunity, since I had just gotten an upgraded machine with SSD.

First, do the following:

• If you don't already have it, download and intall the 64 bit jdk appropriate for your work.
• Download and unpack the 64 bit JDev. I like getting the "generic" version, since it doesn't come with a JDK so you can use the one you want. If you already have one or two JDev installs, that's ok you can always just stick it in a new Middleware Home. When you first run this, it will prompt you to supply the location of the java.exe you want to use. 

JDeveloper Settings

I installed in c:\Oracle\Middleware3.  So for heap, I changed this file:

C:\Oracle\Middleware3\jdeveloper\ide\bin\ide.conf

with these settings (or something larger if desired!):

 AddVMOption  -Xmx1408M
 AddVMOption  -Xms1408M

And for PermGen, I changed (see note about Java 7 and above):

 C:\Oracle\Middleware3\jdeveloper\jdev\bin\jdev.conf

with these settings:

 AddVMOption  -XX:PermSize=384m
 AddVMOption  -XX:MaxPermSize=384m

Integrated Weblogic Settings

The WLS settings are stored under AppData. the subfolder starting with "system" will look like below, but the numbers will be different. The startWebLogic.cmd / startWebLogic.sh file calls the setDomainEnv.cmd / setDomainEnv.sh file, which is the one you want here:

C:\Users\jjames\AppData\Roaming\JDeveloper\system11.1.1.7.40.64.93\DefaultDomain\bin

or on Linux type systems:

~/.jdeveloper/system12.2.1.3.12345bunchofnumbers/DefaultDomain/bin

Note: the DefaultDomain folder won't even exist until you go through the "Create default domain" procedure in JDev.

And then change the heap and permgen (again, no permgen > Java 6) as you need:

set JAVA_VM=-client
set MEM_ARGS=-Xint -Xms512m -Xmx1536m -XX:PermSize=512m -XX:MaxPermSize=512m
echo JAVA Memory arguments: %MEM_ARGS%

SQL Developer - viewing BLOBs or CLOBs with the internal viewer

Environment / assumptions for this post:

• SQL Developer 4.x (likely goes back quite a ways)

I've been using SQL Developer for an embarassingly long time before I discovered this one. I did a quick poll of a few people around the office - two others knew about it and two others did not. I guess I should read the New Features of a version when it comes out. Anyway, the tip could be summarized as - Use the little pencil! If you are looking at some results of a query, click a cell and a pencil will appear.

SQL Developer query results - cliking the pencil allows you to view BLOB types

You have a few options:

• External editor;
• Download;
• Internal viewer as Image or Text

SQL Developer - viewing a BLOB as Image or Text, internally or externally

If it is a photo, you can look at it inline by clicking Image:

SQL Developer Internal viewer showing an image from a BLOB

If it is a CLOB, or even, say, a Microsoft Word document, you can click Text:

SQL Developer Internal viewer showing a Microsoft Word docx file from a CLOB

Occasional dropped JDBC connections with Integrated WLS in JDeveloper 11.1.1.6

Environment / assumptions for this post:

• JDeveloper 11.1.1.6 and Integrated Weblogic Server
• WLS Console application
• JDBC Datasource configuration

I was getting an inconsistent error relating to configuration of a datasource on my local JDeveloper's Integrated Weblogic Server.

java.sql.SQLException: Cannot obtain connection: driverURL = jdbc:weblogic:pool:dsname, props = {EmulateTwoPhaseCommit=false, connectionPoolID=dsname, jdbcTxDataSource=true, LoggingLastResource=false, dataSourceName=dsname}.
Nested Exception: java.lang.RuntimeException: Failed to setAutoCommit to true for pool connection.
at weblogic.jdbc.wrapper.PoolConnection.init(PoolConnection.java:47)
at weblogic.jdbc.pool.Driver.allocateConnection(Driver.java:252)
at weblogic.jdbc.pool.Driver.connect(Driver.java:163)

In the end, to make sure WLS new about the dead connections we configured two things in the Weblogic Datasource, under Advanced tab: the Test Connections on Reserve and also Seconds to Trust an Idle Pool Connection.

Weblogic Datasource configuration - Test Connections on Reserve, and
Seconds to Trust an Idle Pool Connection

Accessing Weblogic Console gives 404 and Log4JLogger cannot be found

Environment / assumptions in this post:

• Weblogic 10.3.6
• Enterprise Manager 11.1.1.6
• Fusion Middleware domain with 11.1.1.6 ADF libraries

I recently came across this problem on one of our 11.1.1.6 development environments, Weblogic version 10.3.6.  The Enterprise Manager was working fine, but going to the WL console gave a 404. For sure EM is used more often, but we would occasionally use the Console to configure Providers in the Security Realm and a few other things. Very strange. A bounce of the Admin Server sometimes would fix unexplained problems, but not this time.

Checking the Admin Server logs revealed this error:

Caused By: org.apache.commons.logging.LogConfigurationException: User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is not useable.
        at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)

So there would appear to be a conflict with logging classes of some kind, or something was missing. In the end, the fix was pretty simple.  The weblogic-application.xml of the console application had to be modified to include the packages that were missing. How and why they went missing is a mystery so far, could be the result of security patching or some other system updates.   If it happens again, it will be interesting to see if the problem and solution is exactly the same or not.

What we did to fix it - we navigated to the console's weblogic-application.xml file located here:

<weblogic_home>/server/lib/consoleapp/META-INF/

and added the following to the beginning of the  <prefer-application-packages> section:

    <package-name>org.apache.log4j.*</package-name>
    <package-name>org.apache.commons.*</package-name>

And then restart the Admin Server. The Console should be available again.

Connecting JConsole to a remote server two ways

Update: JVisualVM is also very nice, I talk about it here on a post Using JVisualVM to connect to a remote Weblogic 12c Managed Server.

Environment / assumptions for this post:

• Weblogic 12c Managed Server(s)
• jdk7 / 8
• Cygwin on Windows 7

This post is about connecting JConsole to a remote JVM. This can be useful for gathering some basic memory and thread details, information about the JVM itself such as  version and classpath, and even control any MBeans defined.

JConsole memory details

JConsole MBean control

The following setup is using a local Windows 7 setup with cygwin, and remote linux server. It was using jdk7, although jdk6 or jdk8 should also work.

There are two general ways to run JConsole to connect to a remote JVM:

1. Run local JConsole, and connect to JVM via port forwarding through ssh. In this case local means running JConsole off your local JDK. This has the advantage of being the faster option (faster setup and also faster application response), but has the disadvantage of only being able to connect to JVMs  you explicitly set up i.e. you need to add extra JVM options, which means a bounce. Not usually feasible for Production systems.
2. Run remote JConsole, but display on local display i.e. forward X11. This has the advantage that you can pick any JVM running on the server by PID, and there is no extra setup or JVM parameters needed. This has the disadvantage of being harder to setup and also being very slow.

Option 1 - Local JConsole

Summary of general steps:

1. Setup JVM with proper params.
2. Connect with ssh and forward the appropriate port.
3. Start JConsole locally, and connect.

Step 1 details - setup JVM with proper params

There are a few needed JVM params that will make this work. With them, we can specify which port to connect to (pick an unused one), which security files to use (if any), and to use SSL or not.

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=50500
-Dcom.sun.management.jmxremote.ssl=false 

If you want to setup some security, then that is also possible. The standard roles are "controlRole" and "monitorRole", you can use these or make up your own. The relevant files are found under JRE_HOME/lib/management, and are:

jmxremote.access - defines roles and their access e.g. 'controlRole' = read/write
jmxremote.password - password for the roles defined in the access file
jmxremote.password.template - copy this file somewhere and protect it
management.properties - specify lots of options in here, including those above

The idea is that you copy the template file to you home directory or somewhere else, then set the permissions so only you can read it. You can also modify the management.properties mentioned above and point to it.  Then, you point to them with two additional options:

-Dcom.sun.management.config.file=your_management.properties_file
-Dcom.sun.management.jmxremote.password.file=your_password_file

Step 2 details - connect with ssh and forward the specified port

Open a cygwin window and ssh to your server with the following options:

ssh oracle@my_server -L 50500:localhost:50500

Or you can, if already connected, hit a quick ~C (that is tilde, then capital letter C), then:

ssh> -L 50500:localhost:50500

Step 3 details - Start JConsole locally and connect

If you have the env variable JAVA_HOME setup, plus JAVA_HOME\bin on your PATH, then you should be able to just type jconsole from a cmd window, or hit cmd+r and type jconsole.

Once it starts, enter the JMX connect string in the Remote Process box. Use localhost, and the port that you chose for forwarding.

service:jmx:rmi:///jndi/rmi://localhost:50500/jmxrmi

JConsole - Remote Process connection string with no security

If you added security, then add the username (e.g. controlRole) and password as well.

For more information, you can check this Oracle page - Monitoring and Management using JMX.

Option 2 - Remote JConsole

Step 1

From a cygwin window, start your X server. This is usually with startxwin, or startx. The X11 system is a topic for many books, so I'll say no more.  Check this Cygwin page for some documentation. An xterm window should open.

Update Note: If you are so lucky :)  to be running a Linux desktop, your job is even easier, and you can skip this step. If you want to try running a Linux desktop (especiall Oracle Enterprise Linux), check my post Coexisting at work with Oracle Enterprise Linux (OEL 7.2) and Microsoft Windows.

Step 2

Connect to the server with ssh. You have to make sure that X11 forwarding is enabled for this to work. See the last half of my post about configuring ssh. To check if it is working properly, you can try running xclock, it should display a small analog clock on your monitor.
Note: below, 'dm' is an alias for our server, it stands for  'dev middleware'. You set this up in the ssh config file, which is where you also configure the X11 forwarding.

$ ssh dm
Enter passphrase for key '/home/jjames/.ssh/id_rsa': 
Last login: Tue Apr 28 11:07:41 2015 from mybox.rcpsc.edu
[Tue Apr 28 11:10:15 oracle@devbox:~ ] $ xclock &
[1] 18987
[Tue Apr 28 11:11:01 oracle@devbox:~ ] $ 

Step 3

Now, you want to ensure that the environment variable JAVA_HOME is set, and that PATH contains JAVA_HOME/bin. This is where JConsole lives. You can try running JConsole at this point, although we need to find the process id first in the next step.

[Tue Apr 28 11:24:30 oracle@devbox:~ ] $ export JAVA_HOME=/u01/app/oracle/product/jdk
[Tue Apr 28 11:24:40 oracle@devbox:~ ] $ export PATH=$JAVA_HOME/bin:$PATH
[Tue Apr 28 11:25:01 oracle@devbox:~ ] $ jconsole &
[2] 22183
[Tue Apr 28 11:25:05 oracle@devbox:~ ] $ 

Step 4

Using ps command, find the PID of the JVM you are interested in.

[Tue Apr 28 11:41:46 oracle@devbox:~ ] $ ps -ef | grep oam_server1
oracle   13050     1  0 Apr21 ?        00:00:00 /bin/sh ./startManagedWebLogic.sh oam_server1
oracle   13196 13051  0 Apr21 ?        01:05:33 /u01/app/oracle/product/jdk/bin/java -client -Xms512m

Step 5

Run JConsole, select the PID.

JConsole with a list of remote PIDs

...and clickety-boo, you should have your connection! It may take a minute, and you may get an SSL error, but if so it will ask you if you want to retry without SSL.

The Story Behind the Name

Ham sandwiches and coffee, at least together, are not my favourite thing. Individually, great.  But neither of those have anything to do with the name of this blog. The java of course is Java. The Ham part? Well there used to be a manager (at my current place of employment) that used to say "this place could complicate a ham sandwich".  So it is very much a Dilbertesque sort of thing.

I suppose "java sandwich" could also be an allegory to layers of complexity in the world of software development. This would also be true, but it is the tendency to unnecessarily complicate things that don't need to be complicated that really inspires the name. Plus it is sort of silly, which appeals to the Monty Python fan in me. Also, every other domain name was taken. Almost literally.

Weblogic OAM Identity Assertion problem - JXDocumentBuilderFactory not found

Environment / assumptions in this post:

• Weblogic 10.3.1
• Alfresco 3.4
• OAM / OID

There is a valuable lesson here, which could be summarized as:

don't configure anything between one middleware home and another middleware home

Or even more generally:

don't point to a file or folder that could be changed unexpectedly by something else

Duh.  But these things can come about innocently enough.

Problem Description

We have Alfresco (ECM) deployed to WLS 10.3.1 (yes, very old now!), with OAM Asserter and OID Authenticator in place. At one point, users started experiencing intermittent problems logging in. After authentication, the Alfresco Dashboard would sometimes not come up and an error page would be displayed.

Log files showed this each time:

javax.xml.parsers.FactoryConfigurationError: Provider oracle.xml.jaxp.JXDocumentBuilderFactory not found
   at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:174)
   at oracle.security.wls.oam.util.OAMIdentityAssertion.(OAMIdentityAssertion.java:170)
   at oracle.security.wls.oam.providers.asserter.OAMIdentityAssertionProviderImpl.updateCallbackHandler(OAMIdentityAssertionProviderImpl.java:791)
   at oracle.security.wls.oam.providers.asserter.OAMIdentityAssertionProviderImpl.headerBasedAssertion(OAMIdentityAssertionProviderImpl.java:991)
   at oracle.security.wls.oam.providers.asserter.OAMIdentityAssertionProviderImpl.assertIdentity(OAMIdentityAssertionProviderImpl.java:658)
Truncated. see log file for complete stacktrace

So a couple questions came up right away:

1. How is it that JXDocumentBuilderFactory can be "not found" intermittently? 

2. What environment change caused this?

Very hard questions to answer. After any patching or upgrading, thorough testing is done on all applications and systems and no problems like this every came up.

I had a hunch that the JXDocumentBuilderFactory was a bit of a red herring, so focused on the next class mentioned in the stack trace: OAMIdentityAssertion. After searching through all jar files with something like this:

for x in $(find . -name "*.jar"); do echo $x; jar tvf $x | grep OAMIdentityAssertion; done

...I came across several references, but only in a different middleware home than where Alfresco was running.  Huh?  That's when I found the managed server JVM startup switch pointing to a different place. 

Problem Solution

We ended up changing the following WLS startup switch:

-Dweblogic.alternateTypesDirectory=.......

...to point to OAM libraries under a WLS 10.3.5 install rather than the current WLS 10.3.6. The 10.3.6 was upgraded in place at some point, which is likely the reason of eventual failure.

Actually, the real root cause is a bit complicated. We installed Alfresco on WLS 10.3.1, originally using OSSO. When we eventually migrated to OAM, installed in another middleware home, we needed to use the OAM Asserter, but the code didn't exist in 10.3.1 so we had to point to where it did. We should have properly upgraded the Alfresco WLS instance, but we didn't want to "kick the sleeping dog". Alfresco was running fine, but we had had some issues and the feeling among management was to not disturb it - we would leave the upgrade for another day. 

So a couple lessons learned:

• if the sleeping dog needs kicking, then kick it. :) In other words, don't put off upgrading too long
• don't point to something outside of your own "sandbox", even if it is unlikely to change

Weblogic Admin Server upload too big

Environment / assumptions in this post:

• Weblogic 10.3.6
• Enterprise Manager 11.1.1.6.0
• Custom ADF Fusion Web Application (EAR file)

In a nutshell, the default upload limit size for Enterprise Manager 11.1.1.x is 40Mb, but it can be changed.

So I was trying to deploy one of our larger custom ADF applications to our Test environment the other day. I was minding my own business when all of a sudden, from behind the bushes, just kidding. No scary animals or nefarious characters here, just an uncooperative Enterprise Manager. In this case, it is Weblogic 10.3.6 and EM version 11.1.1.6.0.

Error when trying to upload the application archive -
"The file could not be uploaded because it is too large"

Turns out there is a rational explanation. There is a baked-in hard limit of 40Mb in the Enterprise Manager ear file controlled by a Trinidad configuration parameter in the web.xml file called:

 org.apache.myfaces.trinidad.UPLOAD_MAX_DISK_SPACE 

You only need to extract the war file from the em.ear file, then modify the web.xml and assemble it all back together.

Here are the steps that worked for our particular install (linux):

1. Go to where the em.ear file is located, under .../user_projects/applications/<domain>
   
   cd /u01/app/oracle/product/fmw/user_projects/applications/MyDomain
    
2. Backup the em.ear file somewhere, just in case you mess up.
3. Copy the em.ear file to a temp folder:
   
   mkdir ~/em_tmp
   cp em.ear ~/em_tmp
    
4. Extract the war file only:
   
   cd ~/em_tmp
   jar -xvf em.ear em.war
    
5. Extract the web.xml from the war file:
   
   jar -xvf em.war WEB-INF/web.xml
    
6. Modify parameter in WEB-INF/web.xml as follows:



old:
  <context-param>
    <param-name>org.apache.myfaces.trinidad.UPLOAD_MAX_DISK_SPACE</param-name>
    <param-value>40960000</param-value>
  </context-param>

new: set some larger value, e.g. 75mb = 1024 x 1024 x 75 = 78643200
<context-param>
   <param-name>org.apache.myfaces.trinidad.UPLOAD_MAX_DISK_SPACE</param-name>
   <param-value>78643200</param-value>
</context-param>




7. Update the war file
   
   jar -uvf em.war WEB-INF/web.xml
    
8. Update the ear file with the new war file
   
   jar -uvf em.ear em.war
    
9. Copy the ear file back again, and bounce the admin server

Some relevant links

Trinidad Configuration:
https://myfaces.apache.org/trinidad/devguide/configuration.html

Upload File Portlet Does NOT Warn File Size Limit (Doc ID 1530144.1)
https://support.oracle.com/epmos/faces/DocContentDisplay?id=1530144.1

Bug 16079700 : [WCP11164]UPLOAD FILE PORTLET DOES NOT WARN FILE SIZE LIMIT
https://support.oracle.com/epmos/faces/BugDisplay?id=16079700