Troubleshooting Fusion Middleware applications - configuring SSH on Windows

Environment / assumptions for this post:

• Windows 7 desktop
• Oracle Enterprise Linux 7.2
• Cygwin with SSH packages

My work machine was and still is on the venerable Windows 7, but this applies to Windows 8/10 just as well. All of our Dev / Test / Production environments are Oracle Enterprise Linux based, and we have many applications deployed to Weblogic 12c.  Although we can sometimes rely on Nodemanager and Enterprise Manager for restarts and things like checking log files, it is sometimes easier to do things manually from the command line. Occasionally some creative use of 'grep' is invaluable in finding specific issues.

In any case, there are many ways to configure ssh for this, but I'll just focus on two: the easy way and the long way.

Easy Way - PuTTY

A simple install and go client like PuTTY is very quick and easy. Download here. Also available as part of the download are scp and sftp clients. There's also MobaXterm and many others.

Longer Way - Cygwin

Again, not so much hard but longer - although worth it in the end! Cygwin is a bunch of Open Source tools that give Linux functionality to Windows. You download an installer, then selectively choose the packages you like. There are tons of websites and blogs that describe in graphic detail how to install and configure Cygwin. Apart from basic ssh sessions, you can also do things like start an X server and launch programs from a remote server i.e. using your own monitor as the display. This can be very helpful - see my other post on this.

First, download and install Cygwin. Search for and select the openssh package and finish the install.  Any required dependencies will automatically be chosen for you.  Lots of default base install stuff will come along as well - things like bash, less, grep, etc etc.
Note: you can go back to the cygwin install directory at any time and re-run the installer to install additional packages.

Cygwin initial setup - search for ssh and select the openssh package.

Under your cygwin folder, you'll have a home directory. Mine looks like:

c:\cygwin64\home\jjames

To start a session, just find the Cygwin Terminal under the Windows menu if you chose to create it there. The first time you run it, the usual login scripts will be created such as .profile and .bash_rc. You can then ssh to one of your servers. The very first time you do so, you will get a message about authenticity - just type yes and the file "known_hosts" under a new folder .ssh will be created.

$ ssh oracle@hostname
The authenticity of host 'hostname (192.168.1.1)' can't be established.
RSA key fingerprint is SHA256:ABCDEFT9OlFlKy9YIv4Fsj0RXCldRbv4MKMxab8P5iw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostname,192.168.1.1' (RSA) to the list of known hosts.
oracle@hostname's password:

At this point, you can stop - no more is required. 

There are other cool things you can do though, such as generating a public/private key pair, and other misc config that will help with things like X forwarding.  See my post about running a remote JConsole on your local X server.

Public/private key pair

Next, you will want to generate a public/private key pair. This will allow you to embed the key in the authorized_hosts file in your home directory on a remote server without having to enter a password, or by entering the same password i.e. you could use the same password for many servers without changing your actual password on each machine. So you do this by running ssh-keygen as follows:

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jjames/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jjames/.ssh/id_rsa.
Your public key has been saved in /home/jjames/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:us/XYZABCDEFPW12345ZAeW+aUBKpBunYxLix+d58KI jjames@myhost
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|  x  s      .    |
|       o   o     |
|  . . o o o o .  |
| . e . *So = . . |
|  . + B.t . = .+ |
|   . j.=  . .=+o+|
|      -oo.o.Boo+o|
|     ..+o..o.oo..|
+----[SHA256]-----+

jjames@myhost ~
$

Two files are generated: id_rsa and id_rsa.pub which are the private and public keys respectively. Add the value in the public key file to each remote server in ~/.ssh/authorized_hosts.

Misc Config

Finally, you can optionally put other configuration in the config file. My config file contains many entries that look like this:

host dev
 hostname development.company.com
 user oracle
 ForwardX11Trusted yes
 ForwardX11 yes
 ServerAliveInterval 300 
 ForwardAgent yes

An explanation is as follows:

host dev - in my case, 'dev' is just an alias, it can be whatever you want.
hostname development.company.com - the server
user oracle - the server login
ForwardX11Trusted yes - set this for X11 forwarding
ForwardX11 yes - set this for X11 forwarding
ServerAliveInterval 300 - seconds before sending a null packet to keep the connection alive
ForwardAgent yes - this is about forwarding authentication when you log in from one machine to another. There is a good guide here.

JDeveloper 12.2.1.0 Integrated Weblogic - logging level doesn't change when value deleted

Environment / assumptions for this post:

• JDeveloper 12.2.1
• Integrated Weblogic server
• Windows 7

In a nutshell: blanking out logging level in the Diagnostic Logging Congifuration doesn't work.

When working with the Integrated Weblogic server in JDeveloper, it is common to increase logging for a particular class or package when troubleshooting. Typically I would set a level of FINEST for something like oracle.jbo package (very useful to discover the SQL being generated by ViewObjects and their bind parameters).

After I finished my troubleshooting, I would decrease the logging level again. However, the logging continued even though I "blanked" it out. This used to work on 11g versions of JDev / Weblogic. With 12c, you must actually set it to something. So in my case I just set it to "INFO" and that did the trick.

Useful Linux commands

It's amazing what you can do on the command line in a Linux environment - a combination of find, grep, sed, awk and others will do a lot to help with troubleshooting. There's not a day that goes by where I don't have to do something "interesting". There isn't anything you cannot do with a combination of the right commands it seems.  Here's some of the ones I use every so often.

• ⇩ Move or copy files in place
• ⇩ Find files larger than XX size
• ⇩ Extract lines from the middle of a file
• ⇩ Adding a column of numbers
• ⇩ Add the total size of a particular file type when you don't have the file ending
• ⇩ Counting docs by mime type
• ⇩ Counting docs by mime type - human readable version
• ⇩ Finding a class inside hierarchy of jar files
• ⇩ Find the files open for writing by a process e.g. log files
• ⇩ Find something in a file, but showing the previous line

NOTE: in my case these all function for Oracle Enterprise Linux v7. Your *nix version (or shell) may require slightly different syntax.

Move or Copy files in place
This will expand to: mv myfilename myfilename.old

mv myfilename{,.old}

Find file larger than 500Mb
You may have to adjust the field numbers. This will print out the size (field #5) and filename (field #9).

find . -type f -size +500000k -exec ls -lh {} \; | awk '{ print $5 ": " $9 }' 

Extract lines from the middle of a file

tail -n +startinglinenum filename | head -n numlines

Add a column of numbers
Here, we are adding the total size of deleted files whose handle is still active. In my case, field #8 was the file size.

/usr/sbin/lsof | grep "deleted" | awk '{sum += $8} END {print sum/1024/1024,"Mb"}'

Add total size of all files of a particular type, when you don't have file ending
For example, all PDF files. Our Alfresco doc repository has all binaries named with ".bin", so we cannot simply filter by filename ending.

find . -type f -exec file -in {} \; | grep zip | cut -d ':' -f 1 | xargs ls -l | awk '{ sum += $5 } END { print sum/1024/1024,"Mb" }'

Count docs by mime type

find ./foldername -type f -exec file -inb {} \;| sort |uniq -c|sort -nr

You'll get output something like this:
     20 application/pdf
      6 application/x-zip
      3 text/plain; charset=us-ascii
      3 image/png
      2 application/octet-stream
      1 application/msword

Count docs by file type - same as previous, more human readable

find . -type f | xargs file | cut -d ':' -f 2 | sed 's/^ *//' | sort | uniq -c | sort -nr

Now the output is like this:
     13 PDF document, version 1.5
      6 Zip archive data, at least v2.0 to extract
      5 PDF document, version 1.6
      3 PNG image data, 77 x 100, 16-bit/color RGBA, non-interlaced
      2 PDF document, version 1.7
      2 data
      1 Microsoft Office Document
      1 ASCII text, with no line terminators
      1 ASCII English text, with very long lines
      1 ASCII English text, with no line terminators

Finding a class file inside jar files, then ignoring 1 or more folders

for x in $(find . -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

for x in $(find . -path ./blah -prune -o -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

for x in $(find . \( -path blah -o -path woof \) -prune -o -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

Finding the files open for writing by a process
e.g. what log files does a process write to? The 0-9 is the file descriptor, the uw is either updatable or writeable, the REG is regular file. Look for handles 1 and 2, they are stdout and stderr.
Note: you may have to be a privileged user to see some of this information.

lsof -p PID | egrep ' [0-9]+[uw] ' | grep 'REG'

Grepping for something in a file, but extracting the previous line for each
I like to specify to show only interesting columns. In this case, the columns for this particular type of log showed date, user, command, etc. Otherwise there can be a lot of noise that gets in the way.

grep -B 1 -E 'NoClassDefFoundError' diagnostic.log | cut -d ' ' -f 1,2,5,13,14,20