Useful Linux commands

It's amazing what you can do on the command line in a Linux environment - a combination of find, grep, sed, awk and others will do a lot to help with troubleshooting. There's not a day that goes by where I don't have to do something "interesting". There isn't anything you cannot do with a combination of the right commands it seems.  Here's some of the ones I use every so often.

• ⇩ Move or copy files in place
• ⇩ Find files larger than XX size
• ⇩ Extract lines from the middle of a file
• ⇩ Adding a column of numbers
• ⇩ Add the total size of a particular file type when you don't have the file ending
• ⇩ Counting docs by mime type
• ⇩ Counting docs by mime type - human readable version
• ⇩ Finding a class inside hierarchy of jar files
• ⇩ Find the files open for writing by a process e.g. log files
• ⇩ Find something in a file, but showing the previous line

NOTE: in my case these all function for Oracle Enterprise Linux v7. Your *nix version (or shell) may require slightly different syntax.

Move or Copy files in place
This will expand to: mv myfilename myfilename.old

mv myfilename{,.old}

Find file larger than 500Mb
You may have to adjust the field numbers. This will print out the size (field #5) and filename (field #9).

find . -type f -size +500000k -exec ls -lh {} \; | awk '{ print $5 ": " $9 }' 

Extract lines from the middle of a file

tail -n +startinglinenum filename | head -n numlines

Add a column of numbers
Here, we are adding the total size of deleted files whose handle is still active. In my case, field #8 was the file size.

/usr/sbin/lsof | grep "deleted" | awk '{sum += $8} END {print sum/1024/1024,"Mb"}'

Add total size of all files of a particular type, when you don't have file ending
For example, all PDF files. Our Alfresco doc repository has all binaries named with ".bin", so we cannot simply filter by filename ending.

find . -type f -exec file -in {} \; | grep zip | cut -d ':' -f 1 | xargs ls -l | awk '{ sum += $5 } END { print sum/1024/1024,"Mb" }'

Count docs by mime type

find ./foldername -type f -exec file -inb {} \;| sort |uniq -c|sort -nr

You'll get output something like this:
     20 application/pdf
      6 application/x-zip
      3 text/plain; charset=us-ascii
      3 image/png
      2 application/octet-stream
      1 application/msword

Count docs by file type - same as previous, more human readable

find . -type f | xargs file | cut -d ':' -f 2 | sed 's/^ *//' | sort | uniq -c | sort -nr

Now the output is like this:
     13 PDF document, version 1.5
      6 Zip archive data, at least v2.0 to extract
      5 PDF document, version 1.6
      3 PNG image data, 77 x 100, 16-bit/color RGBA, non-interlaced
      2 PDF document, version 1.7
      2 data
      1 Microsoft Office Document
      1 ASCII text, with no line terminators
      1 ASCII English text, with very long lines
      1 ASCII English text, with no line terminators

Finding a class file inside jar files, then ignoring 1 or more folders

for x in $(find . -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

for x in $(find . -path ./blah -prune -o -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

for x in $(find . \( -path blah -o -path woof \) -prune -o -name "*.jar"); do echo $x; jar tvf $x | grep myclassname; done

Finding the files open for writing by a process
e.g. what log files does a process write to? The 0-9 is the file descriptor, the uw is either updatable or writeable, the REG is regular file. Look for handles 1 and 2, they are stdout and stderr.
Note: you may have to be a privileged user to see some of this information.

lsof -p PID | egrep ' [0-9]+[uw] ' | grep 'REG'

Grepping for something in a file, but extracting the previous line for each
I like to specify to show only interesting columns. In this case, the columns for this particular type of log showed date, user, command, etc. Otherwise there can be a lot of noise that gets in the way.

grep -B 1 -E 'NoClassDefFoundError' diagnostic.log | cut -d ' ' -f 1,2,5,13,14,20

Setting up JDeveloper and Weblogic Integrated Server as 64 bit on Windows 7

Update: this applies to Windows 8/10 just as well, as well as JDeveloper 12c. As well, with JDK7 there is no permgen memory anymore.

I never really bothered with 64bit JDev and WLS, even though my work station is 64bit Windows 7, until we started deploying taskflows to WebCenter Portal. Just those extra few libraries and classes were causing memory shortages. I would boost the Heap, then run out of PermGen. I would then boost PermGen a bit and reduce Heap a little bit, still problems. The thing is, I was running into the 3Gb limit of 32 bit architecture. So, I decided to muss around and get the 64 bit JDev and WLS running. It was a perfect opportunity, since I had just gotten an upgraded machine with SSD.

First, do the following:

• If you don't already have it, download and intall the 64 bit jdk appropriate for your work.
• Download and unpack the 64 bit JDev. I like getting the "generic" version, since it doesn't come with a JDK so you can use the one you want. If you already have one or two JDev installs, that's ok you can always just stick it in a new Middleware Home. When you first run this, it will prompt you to supply the location of the java.exe you want to use. 

JDeveloper Settings

I installed in c:\Oracle\Middleware3.  So for heap, I changed this file:

C:\Oracle\Middleware3\jdeveloper\ide\bin\ide.conf

with these settings (or something larger if desired!):

 AddVMOption  -Xmx1408M
 AddVMOption  -Xms1408M

And for PermGen, I changed (see note about Java 7 and above):

 C:\Oracle\Middleware3\jdeveloper\jdev\bin\jdev.conf

with these settings:

 AddVMOption  -XX:PermSize=384m
 AddVMOption  -XX:MaxPermSize=384m

Integrated Weblogic Settings

The WLS settings are stored under AppData. the subfolder starting with "system" will look like below, but the numbers will be different. The startWebLogic.cmd / startWebLogic.sh file calls the setDomainEnv.cmd / setDomainEnv.sh file, which is the one you want here:

C:\Users\jjames\AppData\Roaming\JDeveloper\system11.1.1.7.40.64.93\DefaultDomain\bin

or on Linux type systems:

~/.jdeveloper/system12.2.1.3.12345bunchofnumbers/DefaultDomain/bin

Note: the DefaultDomain folder won't even exist until you go through the "Create default domain" procedure in JDev.

And then change the heap and permgen (again, no permgen > Java 6) as you need:

set JAVA_VM=-client
set MEM_ARGS=-Xint -Xms512m -Xmx1536m -XX:PermSize=512m -XX:MaxPermSize=512m
echo JAVA Memory arguments: %MEM_ARGS%

SQL Developer - viewing BLOBs or CLOBs with the internal viewer

Environment / assumptions for this post:

• SQL Developer 4.x (likely goes back quite a ways)

I've been using SQL Developer for an embarassingly long time before I discovered this one. I did a quick poll of a few people around the office - two others knew about it and two others did not. I guess I should read the New Features of a version when it comes out. Anyway, the tip could be summarized as - Use the little pencil! If you are looking at some results of a query, click a cell and a pencil will appear.

SQL Developer query results - cliking the pencil allows you to view BLOB types

You have a few options:

• External editor;
• Download;
• Internal viewer as Image or Text

SQL Developer - viewing a BLOB as Image or Text, internally or externally

If it is a photo, you can look at it inline by clicking Image:

SQL Developer Internal viewer showing an image from a BLOB

If it is a CLOB, or even, say, a Microsoft Word document, you can click Text:

SQL Developer Internal viewer showing a Microsoft Word docx file from a CLOB