Dollop of Desi

Java 7 monitoring files/folders

2011-12-21T15:37:00.000-08:00

Java 7 has this new watchservice which is a welcome development and you can read about it in many blogs as well as the oracle site. I tried playing with it using the code sample from some of the blogs. But unfortunately it seemed to work only the very first time and not after that.
Here is the code

public FileWatcher(String directoryPath) throws IOException {
        this.directoryPath  = directoryPath;
        FileSystem fileSystem = FileSystems.getDefault();
        watcher = fileSystem.newWatchService();
        
        Path myDir = fileSystem.getPath(this.directoryPath);
        myDir.register(watcher, StandardWatchEventKinds.ENTRY_CREATE, 
                  StandardWatchEventKinds.ENTRY_DELETE, StandardWatchEventKinds.ENTRY_MODIFY);
        monitorFolder();
} 
 
private void monitorFolder() {
  executor = Executors.newSingleThreadScheduledExecutor();
  executor.submit(this);
} 
 
public void run() {
  WatchKey watckKey = null;
  try {
   while(!shutdown ) {
    watckKey = watcher.take();
    events = watckKey.pollEvents();
    listener.processEvent(events);       
   }
  } catch (InterruptedException e) {
   logger.error("Error monitoring directory", e);
  }
}

Some more beating around the bush and googling I discovered that the WatchKey needs to be reset for subsequent events to flow through to the listener.
Wonder why this strange behaviour. Anyhow here is the fix and after that everything was fine and dandy!

public void run() {
  WatchKey watchKey = null;
  try {
   while(!shutdown ) {
    watchKey = watcher.take();
    events = watchKey.pollEvents();
    listener.processEvent(events);
          
    watchKey.reset();   }
  } catch (InterruptedException e) {
   logger.error("Error monitoring directory", e);
  }
}

Found out later by paying more attention(!!) to the article on the oracle site that resetting the watchkey is key
to getting future events.

Interleaving audio files to different channels

2011-09-23T14:19:00.000-07:00

Audio files have inherent characteristics like number of channels, sample size, frame size, sample rate, file type, number of samples, etc. to quote a few.

Here is a nice description of samples and channels from the java sound faq.

Each second of sound has so many (on a CD, 44,100)		  digital samples of sound pressure per second. The number of		  samples per second is called sample rate or sample		  frequency. In PCM (pulse code modulation) coding, each		  sample is usually a linear representation of amplitude as a		  signed integer (sometimes unsigned for 8 bit).  

There is one		  such sample for each channel, one channel for mono, two		  channels for stereo, four channels for quad, more for		  surround sound. One sample frame consists of one sample for		  each of the channels in turn, by convention running from		  left to right.

Each sample can be one byte (8 bits), two bytes (16		  bits), three bytes (24 bits), or maybe even 20 bits or a		  floating-point number. Sometimes, for more than 16 bits per		  sample, the sample is padded to 32 bits (4 bytes) The order		  of the bytes in a sample is different on different		  platforms. In a Windows WAV soundfile, the less significant		  bytes come first from left to right ("little endian" byte		  order). In an AIFF soundfile, it is the other way round, as		  is standard in Java ("big endian" byte		  order). 

Some more audio file fundamentals from the javadocs...

frameSize is the number of bytes in each frame of a sound that has this format.

sampleRate is the number of samples played or recorded per second, for sounds that have this format.

More definitions can be found here.
Audio files are made up of samples and samples are made up of bytes.
Audio files come in two types, mono and stereo. Mono files have only one channel (perhaps was recorded with one receiver). Stereo audio files can have from two to as many channels.

Mono audio files when played, samples from the single channel are automatically duplicated and sent to all the channels. For example if speakers are attached to a computer, that would be two channels.In the case of a stereo audio file, it is pre-recorded for multiple channels.

Now what if you want to interleave two different audio files into one audio file such a way that each audio file is played on separate channels. There are many possible use cases for doing that. One could be to get the remix effect. For example you can take a song and mix it with some background music or add some percussion effect. Of course there are many sophisticated audio software out there which can do this and much more. But the goal here is to demonstrate a simple java program to achieve interleaving of audio files.

Here I am going to take two mono audio files of the same format (wave file), encoding (PCM) and sample rate. Interleave them to produce another wave file with different audio playing on each channel.

I am using Java Sound API for this exercise. Java supports only AU, AIFF and WAV formats. Extensions/plug-ins for MP3 and other audio formats are available through 3rd party vendors.
The audio files used in this example are leftChannelAudio.wav and rightChannelAudio.wav

The format details for the audio files are

rightChannelAudio.wav

nbChannel = 1

frameRate = 44100.0

frameSize = 2

sampleSize(bits)= 16

nbSamples = 1105408

encoding = PCM_SIGNED

sample rate = 44100.0

leftChannelAudio.wav

nbChannel = 1
frameRate = 44100.0
frameSize = 2
sampleSize(bits) = 16
nbSamples = 1069056
encoding = PCM_SIGNED
sample rate = 44100.0

Here is the main method of the class.

public static void main(String[] args) {
        try {
            String soundFileLeft = "leftChannelAudio.wav";
            File fileLeft = new File(soundFileLeft);   // This is the file we'll be playing on the left channel.
            String soundFileRight = "rightChannelAudio.wav";
            File fileRight = new File(soundFileRight);
            
            float sampleRate = 44100.0f;
            int sampleSizeInBits = 16;
            int channels = 2;
            boolean signed = true;
            boolean bigEndian = false;
            AudioFormat targetFormat = new AudioFormat(sampleRate, sampleSizeInBits, channels, signed, bigEndian);
            AudioMixer mixAudio = new AudioMixer(fileLeft, fileRight, targetFormat);
            
            File outFile = new File("outSingleSingleMixer.wav");
            mixAudio.mixnWrite(AudioFileFormat.Type.WAVE, outFile);
        } catch(Exception e) {
            e.printStackTrace();
        }
    }

The interleaving of audio bytes is done in mixIntoStereoAudio method as shown below.

 private AudioInputStream mixIntoStereoAudio(AudioInputStream leftAudioInputStream,
                                                AudioInputStream rightAudioInputStream) throws IOException{
        ArrayList byteArrays = new ArrayList();
        int nbChannels = 1;
        byte[] compiledStream = null;
        int leftAudioBytes = -1;
        int rightAudioBytes = -1;
        
        byteArrays.add(convertStream(leftAudioInputStream));
        byteArrays.add(convertStream(rightAudioInputStream));
        
        long maxSamples;
        if (leftAudioInputStream.getFrameLength() > rightAudioInputStream.getFrameLength()) {
            maxSamples = leftAudioInputStream.getFrameLength();
            nbChannels = leftAudioInputStream.getFormat().getChannels();
        } else {
            maxSamples = rightAudioInputStream.getFrameLength();
            nbChannels = rightAudioInputStream.getFormat().getChannels();
        }
        long maxOutputSizeinBytes = maxSamples * sampleSizeinBytes;
        if (nbChannels == 1)
            maxOutputSizeinBytes = maxOutputSizeinBytes * 2;
        
        compiledStream = new byte[(int) maxOutputSizeinBytes]; //max size of number of bytes
            
        log.info("Output bytes size: " + compiledStream.length);
        for(int i = 0; i < compiledStream.length; i += sampleSizeinBytes){
            leftAudioBytes = writeSamplestoChannel(byteArrays, 0,
                    sampleSizeinBytes, compiledStream, leftAudioBytes, i);
            
            i += sampleSizeinBytes;
            
            rightAudioBytes = writeSamplestoChannel(byteArrays, 1, 
                    sampleSizeinBytes, compiledStream, rightAudioBytes, i);
        }

        AudioInputStream newaudioStream = generateNewAudioStream(compiledStream);
        return newaudioStream;
    }

In this case the sample has two bytes. In a stereo audio file samples are written out in the following fashion. First sample for left channel and second for right and so on.
This example uses a simple technique of filling a sample from each audio file for each channel respectively.Filling in silence value whenever we run out of samples from an audio file. This is because the two audio files being interleaved might not have the same number of samples.

This technique can be extended easily to handle stereo audio files or list of audio files for each channel.
Go ahead and run this program with the sample audio files provided or you can just listen to how the interleaved output file plays out.

The complete program and sample audio files are available here.

Some good resources on using Java Sound API are listed below.

http://www.jsresources.org/faq_audio.html
http://www.builogic.com/java/javasound-read-write.html

Routing Serial data to a socket

2011-07-08T12:22:00.000-07:00

You don't need a hardware device (like Lantronix adapter) to route data from a Serial port to a socket. Thanks to socat, you can relay data from a serial port to another or serial port to a socket or socket to socket and plenty more such combos, all in one line! socat is exhaustive in what it can take as arguments. Check it out.

Here is an example of making available data from serial port /dev/ttyS0 on a linux environment (usually this is COM1 on windows boxes) through a socket (port = 8023).

socat TCP-LISTEN:8023,fork /dev/ttyS0,raw,b9600,echo=0

The baud rate specified here is 9600. fork option lets you make multiple connections to the source of data, which in this case is the serial port.

You can verify the data from the socket by using a telnet or similar application.

telnet localhost 8023

Monitoring data from a Serial port

2011-06-02T17:10:00.000-07:00

Serialports have gone out of vogue! You no longer get serial ports on a computer except on Desktop PCs, unless you custom order it. Notebooks/Laptops, no such luck. But there there are many people and companies which deal with serial ports on a daily basis.

In the windows world we had the good old Hyperterminal program which was shipped with all windows distributions until recently. Now Hyperterminal is a payware utility and can be purchased separately. No free utility available out there for windows! But then in the linux world, things are friendlier. There is minicomm and many more free utilities that come with the operating system.

Well the point of this blog post is not to berate windows or anything. Just wanted to share a way you could look at the data coming in from a serial port without downloading/purchasing any software. This is very handy for troubleshooting, say if you are looking for any control characters or any noise in the serial line, etc.All this while an app is running using the serialport. This can be done outside the app space to monitor activity on a serial port.
To see the data from a serial port on command line as well as save it to a file here is what you can do.

The script command writes data from a terminal to the specified output file.

script -af serialOut.txt

hexdump reads from a serial port and prints out the bytes in hex and ASCII

hexdump -vC /dev/ttyS0

Here is what the whole thing looks like after executing the commands above...

The top window shows the output from the terminal and the bottom one shows the contents of the output file in real time.

You can do this and much more on a windows platform too! By installing cygwin and running linux commands within the cygwin environment.

Advanced Installer - Changing the package Install Drive

2011-03-18T10:42:00.000-07:00

The default drive for installing an application on a windows system is C. In fact on launching the .msi or .exe installer file it does show you the complete path where the package will be installed. but if you preset the drive to say D on the .aip file, the msi installer chokes on boxes where there is no D drive.

The fix here is to create some custom action and execute the custom action as part of the install/uninstall sequence. See the custom action screen snapshot below.

Here we set the value for APPDIR property pointing to D drive if the condition D_DRIVE holds true. The INSTALL_DIR property is set in the Install parameters tab as shown below.

Last but not the least is defining the D_DRIVE condition.

Following the above steps should enable you to install on D drive or for that matter any drive (if it exists) by default.

Unable to login to Ubuntu. Screensaver unresponsive!

2011-01-19T10:30:00.000-08:00

Generally I don't power off my machine at the end of the day or for that matter anytime unless there are updates that prompt me to restart. Also I leave firefox ON with 10-15 tabs! So the system is ON for weeks sometimes even a month or two! Ofcourse some of the processes start getting slower and slower. especially Firefox. But then all I need to fix that is to restart Firefox.
But in the last 6 months or so I have been encountering this weird problem.
when I come to work the next day...many times I couldn't get to the login dialog screen, no matter how hard I tried with the mouse or keyboard. Seemed like the gnome screen saver had gone into the weeds! The machine was ON and everything was running. The solution to this is to login through a command line session CTRL-ALT-F6 and kill the gnome-screen saver process. I used top utility to find the process id (look for gnome-screensaver in the command column) and also kill it. and then try ALT-F7 and everything is fine.

SOAP vs REST in Java land

2010-07-14T11:34:00.000-07:00

SOAP and REST are the two most popular web service technologies in use today. REST has become the more preferred one, since it deals directly with URIs and can handle requests in plain text directly over HTTP. Testing it with a browser or tools like curl is super easy.

Let's go through some examples here. Since our focus is on the Java landscape.All the examples are in the Java environment.

The development environment I used is eclipse 3.5, Ubuntu 10.04 LTS the Lucid Lynx, JDK 1.16.20

With Java annotations kicking off in big style with Java 5. Now the coolest way to develop a web service is by using POJOs and some sprinkling of annotations here and there.

Let's start with a simple Java class (POJO) and make a web service out of it!

Let's take the same Java class (CalcWSImpl.java and Calculator.java are pretty much identical except for the annotations) and setup a web service using SOAP as well as REST.

Developing a SOAP based web service

There are many nice tools available today bundled with the JRE like wsimport, wsgen to generate web service client source code and wrapper classes to build a web service.

We start with a simple class Calculator.java. see the listing below.

This class already has the following JAX-WS annotations...

@WebService - to identify itself as an endpoint class.

@SOAPBinding(style=SOAPBinding.Style.RPC) – style for messages used in a webservice.

@WebMethod – to expose the method as a webservice operation.

For a complete list of all the JAX-WS annotations refer to JAX-WS annotations

Just with these annotations , we are ready to expose this class through a web service.

Take a look at TestCalc.java, which launches a web service.

package org.webservice.server;

import javax.xml.ws.Endpoint;

public class TestCalc {

        public static void main(String[] args) {
            Calculator calcWS = new Calculator();
            
            Endpoint.publish("http://localhost:8085/calc", calcWS);
        }

}

Run it and you have a running webservice at http://localhost:8085/calc.

The wsdl is available at http://localhost:8085/calc?wsdl

This is what the generated .wsdl looks like.

<?xml version="1.0" encoding="UTF-8"?><!-- Published by JAX-WS RI at http://jax-ws.dev.java.net. RI's version is JAX-WS RI 2.1.6 in JDK 6. --><!-- Generated by JAX-WS RI at http://jax-ws.dev.java.net. RI's version is JAX-WS RI 2.1.6 in JDK 6. --><definitions xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://server.webservice.org/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.xmlsoap.org/wsdl/" targetNamespace="http://server.webservice.org/" name="CalcWS">
<types></types>
<message name="div">
<part name="arg0" type="xsd:int"></part>
<part name="arg1" type="xsd:int"></part>
</message>
<message name="divResponse">
<part name="return" type="xsd:float"></part>
</message>
<message name="add">
<part name="arg0" type="xsd:int"></part>
<part name="arg1" type="xsd:int"></part>
</message>
<message name="addResponse">
<part name="return" type="xsd:int"></part>
</message>
<message name="list"></message>
<message name="listResponse">
<part name="return" type="xsd:int"></part>
</message>
<portType name="Calc">
<operation name="div" parameterOrder="arg0 arg1">
<input message="tns:div"></input>
<output message="tns:divResponse"></output>
</operation>
<operation name="add" parameterOrder="arg0 arg1">
<input message="tns:add"></input>
<output message="tns:addResponse"></output>
</operation>
<operation name="list">
<input message="tns:list"></input>
<output message="tns:listResponse"></output>
</operation>
</portType>
<binding name="CalcWSPortBinding" type="tns:Calc">
<soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="rpc"></soap:binding>
<operation name="div">
<soap:operation soapAction=""></soap:operation>
<input>
<soap:body use="literal" namespace="http://server.webservice.org/"></soap:body>
</input>
<output>
<soap:body use="literal" namespace="http://server.webservice.org/"></soap:body>
</output>
</operation>
<operation name="add">
<soap:operation soapAction=""></soap:operation>
<input>
<soap:body use="literal" namespace="http://server.webservice.org/"></soap:body>
</input>
<output>
<soap:body use="literal" namespace="http://server.webservice.org/"></soap:body>
</output>
</operation>
<operation name="list">
<soap:operation soapAction=""></soap:operation>
<input>
<soap:body use="literal" namespace="http://server.webservice.org/"></soap:body>
</input>
<output>
<soap:body use="literal" namespace="http://server.webservice.org/"></soap:body>
</output>
</operation>
</binding>
<service name="CalcWS">
<port name="CalcWSPort" binding="tns:CalcWSPortBinding">
<soap:address location="http://localhost:8085/calc"></soap:address>
</port>
</service>
</definitions>

Now we are ready to test the newly created web service. Either you can go use the free SOAP clients available on the Internet like soap client or tools like wsimport (that comes with JDK) to generate wrapper classes to test a web service endpoint.

Now using the live web service and wsimport tool, let's generate some client classes. Run the following command from the root folder of the project.

wsimport -d bin -s test http://localhost:8085/calc?wsdl

The last step creates an interface Calc.....


package org.webservice.server;

import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.jws.WebService;
import javax.jws.soap.SOAPBinding;


/**
 * This class was generated by the JAX-WS RI.
 * JAX-WS RI 2.1.6 in JDK 6
 * Generated source version: 2.1
 * 
 */
@WebService(name = "Calc", targetNamespace = "http://server.webservice.org/")
@SOAPBinding(style = SOAPBinding.Style.RPC)
public interface Calc {


    /**
     * 
     * @param arg1
     * @param arg0
     * @return
     *     returns float
     */
    @WebMethod
    @WebResult(partName = "return")
    public float div(
        @WebParam(name = "arg0", partName = "arg0")
        int arg0,
        @WebParam(name = "arg1", partName = "arg1")
        int arg1);

    /**
     * 
     * @param arg1
     * @param arg0
     * @return
     *     returns int
     */
    @WebMethod
    @WebResult(partName = "return")
    public int add(
        @WebParam(name = "arg0", partName = "arg0")
        int arg0,
        @WebParam(name = "arg1", partName = "arg1")
        int arg1);

    /**
     * 
     * @return
     *     returns int
     */
    @WebMethod
    @WebResult(partName = "return")
    public int list();

}

.....and a service class CalcWS


package org.webservice.server;

import java.net.MalformedURLException;
import java.net.URL;
import java.util.logging.Logger;
import javax.xml.namespace.QName;
import javax.xml.ws.Service;
import javax.xml.ws.WebEndpoint;
import javax.xml.ws.WebServiceClient;
import javax.xml.ws.WebServiceFeature;


/**
 * This class was generated by the JAX-WS RI.
 * JAX-WS RI 2.1.6 in JDK 6
 * Generated source version: 2.1
 * 
 */
@WebServiceClient(name = "CalcWS", targetNamespace = "http://server.webservice.org/", wsdlLocation = "http://localhost:8085/calc?wsdl")
public class CalcWS
    extends Service
{

    private final static URL CALCWS_WSDL_LOCATION;
    private final static Logger logger = Logger.getLogger(org.webservice.server.CalcWS.class.getName());

    static {
        URL url = null;
        try {
            URL baseUrl;
            baseUrl = org.webservice.server.CalcWS.class.getResource(".");
            url = new URL(baseUrl, "http://localhost:8085/calc?wsdl");
        } catch (MalformedURLException e) {
            logger.warning("Failed to create URL for the wsdl Location: 'http://localhost:8085/calc?wsdl', retrying as a local file");
            logger.warning(e.getMessage());
        }
        CALCWS_WSDL_LOCATION = url;
    }

    public CalcWS(URL wsdlLocation, QName serviceName) {
        super(wsdlLocation, serviceName);
    }

    public CalcWS() {
        super(CALCWS_WSDL_LOCATION, new QName("http://server.webservice.org/", "CalcWS"));
    }

    /**
     * 
     * @return
     *     returns Calc
     */
    @WebEndpoint(name = "CalcWSPort")
    public Calc getCalcWSPort() {
        return super.getPort(new QName("http://server.webservice.org/", "CalcWSPort"), Calc.class);
    }

    /**
     * 
     * @param features
     *     A list of {@link javax.xml.ws.WebServiceFeature} to configure on the proxy.  Supported features not in the features parameter will have their default values.
     * @return
     *     returns Calc
     */
    @WebEndpoint(name = "CalcWSPort")
    public Calc getCalcWSPort(WebServiceFeature... features) {
        return super.getPort(new QName("http://server.webservice.org/", "CalcWSPort"), Calc.class, features);
    }

}

Now let's run the test client CalcWSClient

package org.webservice.server;

import javax.xml.ws.WebServiceRef;


public class CalcWSClient {
      @WebServiceRef(wsdlLocation="http://localhost:8085/calc?wsdl")
      static CalcWS service = new CalcWS();

      public static void main(String[] args) {
        try {
            CalcWSClient client = new CalcWSClient();
          client.doTest(args);
        } catch(Exception e) {
          e.printStackTrace();
        }
      }

      public void doTest(String[] args) {
        try {
          System.out.println("Retrieving the port from the following service: " + service);
          Calc port = service.getCalcWSPort();
          System.out.println("Invoking the add operation on the port.");

          int response = port.add(1, 3);
          System.out.println(response);
        } catch(Exception e) {
          e.printStackTrace();
        }
      }
}

The picture below shows the results on eclipse console.

Developing a RESTful web service

Using JAX-RS the API for REST based web service. There are a couple frameworks out there. The popular ones are Jersey and Apache CXF. This example was tested using Jersey 1.2 (you can download from jersey), the Sun's open source implementation.

Just with few annotations and some associated classes, Jersey lets you expose POJOs as web services. It also provides support for JSON.

You can find all the literature and code at jersey docs.

All the annotations are defined in jsr311-api.jar.

Here is a nice article about implementing REST in java. So in short RESTful architecture is about exposing resources and it's operations.

A resource class is a Java class with JAX-RS annotations to identify itself as a web resource.

The Root resource class used here is CalcWSImpl.java. It has three resource methods, add, div and list.

The code listing is shown below.

package org.rest.examples;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;

import org.json.JSONException;
import org.json.JSONObject;

//The Java class will be hosted at the URI path "/calc"
@Path("/calc/")
public class CalcWSImpl {
    // The Java method will process HTTP GET requests
    @GET 
    @Path("add/{input1}/{input2}")
    // The Java method will produce content identified by the MIME Media
    // type "text/plain"
    @Produces("text/plain")
    public String add(@PathParam("input1") int a, @PathParam("input2") int b) {
        return String.valueOf(a+b);
    }
    
    @GET
    @Path("list/")
    @Produces("application/json")
    public String list() throws JSONException {
        JSONObject list = new JSONObject();
        list.put("mercedes", "20");
        list.put("porsche", "25");
        list.put("audi", "32");
        list.put("lexus", "35");
        return list.toString();
    }
    
    // The Java method will process HTTP GET requests
    @GET 
    @Path("div/{input1}/{input2}")
    // The Java method will produce content identified by the MIME Media
    // type "text/plain"
    @Produces("text/plain")
    public String div (@PathParam("input1") int a, @PathParam("input2") int b) {
        return String.valueOf(a/b);
    }
}

Some brief explanations of the annotations used...

@Path the URI path for a class or method. This is path you add to the base URL for the webservice to access the resource. For example http://localhost:9998/calc/list

@GET method will process HTTP GET methods

@Produces MIME media type of objects returned by a method

@PathParam parameters in the URI path like in the following URI http://localhost:9998/calc/add/6/7/

6 and 7 are parameters as you can see from the @Path annotation add/{input1}/{input2} for the add method.

To find all the JSR-311 annotations refer to JAX-RS annotations

Now to get going we would need the following jars

asm-3.1.jar

jersey-bundle-1.2.jar

jsr311-api-1.11.jar

json.jar

You can download them from here.

We are ready to launch the webservice. Let's look at Main.java

package org.rest.examples;

import com.sun.net.httpserver.HttpServer;
import com.sun.jersey.api.container.httpserver.HttpServerFactory;
import java.io.IOException;

public class Main {
    
    public static void main(String[] args) throws IOException {
        HttpServer server = HttpServerFactory.create("http://localhost:9998/");
        server.start();
        
        System.out.println("Server running");
        System.out.println("Visit: http://localhost:9998/list");
        System.out.println("Hit return to stop...");
        System.in.read();
        System.out.println("Stopping server");   
        server.stop(0);
        System.out.println("Server stopped");
    }
    
    
}

Run Main.java from eclipse as a java application. You have a running web service.

Let's try some examples here using curl.

~$ curl http://localhost:9998/calc/add/6/7/

13
~$

~$ curl http://localhost:9998/calc/list

{"lexus":"35","audi":"32","porsche":"25","mercedes":"20"}
~$

Deploying...
Let's deploy to a servlet container other than Tomcat :-) , Jetty for instance.
here is what the web.xml should look like.

<?xml
version="1.0"
encoding="UTF-8"?>



<!DOCTYPE
web-app PUBLIC
"-//Sun Microsystems, Inc.//DTD Web
Application 2.3//EN"
"http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
 
<servlet>
      
    <servlet-name>JerseyTest</servlet-name>
      
     <servlet-class>
      
         com.sun.jersey.spi.container.servlet.ServletContainer
      
     </servlet-class>
      
      <init-param>
      
   
  <param-name>com.sun.jersey.config.property.packages</param-name>
      
      <param-value>org.rest.examples</param-value>
      
  </init-param>
      
 <load-on-startup>1</load-on-startup>
   
</servlet>
   
<servlet-mapping>
      
    <servlet-name>JerseyTest</servlet-name>
      
 <url-pattern>/*</url-pattern>
   
</servlet-mapping>
</web-app>

With jersey configured as a servlet.

All we are doing here is providing the package name (“ org.rest.examples”) which has resource classes.

We need another class to instantiate Jetty! Yes we are going to start an embedded instance programmatically. Here is RunServlet.java

package org.rest.examples;

import org.mortbay.jetty.Server;
import org.mortbay.jetty.webapp.WebAppContext;

public class RunServlet {

    public static void main(String[] args) throws Exception {
        Server server = new Server(8081);
        
        WebAppContext context = new WebAppContext();
        context.setParentLoaderPriority(true);
        context.setDescriptor("./src/web/WEB-INF/web.xml");
        context.setResourceBase("./src/web");
        
        server.setHandler(context);
        
        server.start();
        server.join();
        
    }

}

Need the following Jetty jars to be in the classpath.
jetty-6.1.18.jar
jetty-util-6.1.1.18.jar
servlet-api-2.5-20081211.jar

On running RunServlet, see the console output below. The JerseyTest servlet scans the package org.rest.examples and finds HelloWorld and CalcWSImpl resource classes.

Now let's see how it works.

All you need is just a few jars and resource classes and you are done creating a RESTful web service! Couldn't be easier than this.

Conclusions:

As you can see, with frameworks like Jersey writing/deploying/testing REST based web services is much simpler and straight forward than it's SOAP equivalent. But of course SOAP has many more features that you may or may not need. So depending on your internal computing environment and external client needs you can choose SOAP or put everything to REST!

Finding process/program using a port

2010-02-24T16:48:00.000-08:00

Many a times you would get a "Bind Exception", "Address already in use" kind of errors and need to find the process that's using the port. For example say the port is 8080. The following command will get you the pid and program name as well.

netstat -nlept | grep "8080"

The result might look something like this...

Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp6 0 0 :::8080 :::* LISTEN 1000 994652 4024/java
Where 4024 is the pid and it's a java app.

Upgraded process manager for Linux

2010-02-02T15:34:00.000-08:00

Check this out htop which is a more interactive and intuitive upgrade of the good old 'top' has many pluses to it like Color themes, mouse enabled and so on.

XML parsing in Java and Groovy

2010-01-04T11:08:00.000-08:00

XML is everywhere. Somebody said that XML is like violence. if it doesn't work for you then you are not using enough of it! Well like it or not you have to deal with XML everywhere. To say that XML handling in java is not easy is an understatement. What if you have to deal with SAX, DOM parsers, etc. Well there are some libraries out there which make the job a little easier. Like XOM, XSTREAM, etc. But what if you want to read an XML config file into a Java object?

XSTREAM has a fairly simple way of doing that. Here is an example.

One other option is to look at the many jvm based languages like groovy, scala, jruby, etc. They all handle XML super easy. Let's see How easy it is in Groovy.
Groovy has two APIs XMLParser and XMLSlurper for dealing with XML.

Groovy lets you easily cut to the metadata instead of going through each node and getting the child and more.
Using a scripting language gives you a tradeoff for complexity and since it is run by the same jvm you don't loose any performance either. A call to groovy functionality can be embedded in java or invoked through a shell. That way any scripting language can be used.

The example here uses a simple xml data file (Cars.xml) shown below.

<records>
<carList>
      <car name='HSV Maloo' make='Holden' year='2006'>
        <country>Australia</country>
        <record type='speed'>Production Pickup Truck with speed of 271kph</record>
      </car>
      <car name='P50' make='Peel' year='1962'>
        <country>Isle of Man</country>
        <record type='size'>Smallest Street-Legal Car at 99cm wide and 59 kg in weight</record>
      </car>
      <car name='Royale' make='Bugatti' year='1931'>
        <country>France</country>
        <record type='price'>Most Valuable Car at $15 million</record>
      </car>
</carList>
</records>

The data structure to represent the XML data is defined in
Records.java.

package org.xml.example;

import java.util.List;

public class Records {
 private List carList = null;
 
 public static class Car {
  String name;
  String make;
  String year;
  Country country;
  Record record;
  
  public static class Record {
   String type;
   String info;
   
   public String toString(){
    return "Record type:" + type + ", info:" + info;
   }
  }


  public static class Country {
   String name;
   
   public String toString(){
    return "Country: " + name;
   }
  }
  
  public String toString(){
   return "name: " + name + ", make:" + make + ", year:" + year + " " 
   + country + " " + record;
  }
 }
 
 public List getCars() {
  return carList;
 } 
 
 public String toString(){
  StringBuilder sb = new StringBuilder();
  for (Car car : carList) {
   sb.append(car.toString());
   sb.append(System.getProperty("line.separator"));
  }
  return sb.toString();
 }
}

The main program is XMLParseExample.java.

package org.xml.example;

import groovy.lang.Binding;
import groovy.util.GroovyScriptEngine;

import java.io.FileNotFoundException;
import java.io.FileReader;

import com.thoughtworks.xstream.XStream;

public class XMLParseExample {

    public static void main(String[] args) throws FileNotFoundException {
//        parseXMLinGroovy();
        parseXMLinXstream();
    }

    private static void parseXMLinXstream() throws FileNotFoundException {
        XStream xstream = new XStream();
        
        xstream.aliasType("records", Records.class);
        xstream.aliasType("car", Records.Car.class);
        xstream.useAttributeFor(Records.Car.class, "name");
        xstream.useAttributeFor(Records.Car.class, "make");
        xstream.useAttributeFor(Records.Car.class, "year");
        xstream.aliasType("country", Records.Car.Country.class);
        xstream.aliasType("record", Records.Car.Record.class);
        xstream.useAttributeFor(Records.Car.Record.class, "type");
                
        Records records = (Records)xstream.fromXML(new FileReader("/home/csrinivasan/Documents/Nigeria/Cars.xml"));
        String xml = xstream.toXML(records);
        System.err.println(xml);
    }

    private static void parseXMLinGroovy()  {
        String[] roots = new String[] { "./scripts/" };
        GroovyScriptEngine gse;
        try {
            gse = new GroovyScriptEngine(roots);
            Binding binding = new Binding();
            gse.run("ParseXML.groovy", binding);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

The groovy script is ParseXML.groovy.

import org.xml.example.*;
import org.xml.example.Records.Car.*

class ParseRecords {
    private List cars

  ParseRecords () {
      cars = new ArrayList()
  }

  def parseRecords() {
    ClassLoader sysClassLoader = ClassLoader.getSystemClassLoader()
      def data = sysClassLoader.getResource("Cars.xml").text
      def xmlSlurper = new XmlParser()
      def records = xmlSlurper.parseText(data)
      Records recs = new Records()
    records.carList.each {
        recs.carList = new ArrayList()
        it.car.each {
            Records.Car car = new Records.Car()
            car.name = it.attribute("name")
            car.make = it.attribute("make")
            car.year = it.attribute("year")
            car.country = new Records.Car.Country()
            car.country.name = it.country.text()
            car.record = new Records.Car.Record()
            it.record.each {
                car.record.type = it.attribute("type")
            }
            car.record.info = it.record.text()
            recs.carList.add(car)
        }
    }
    println recs
  }

ParseRecords algConfig = new ParseRecords()
algConfig.parseRecords()

Lookup Source Filter option in Informatica

2009-10-13T16:42:00.000-07:00

Informatica version 8.6.1 has a feature where Lookup Source filter could be used. This feature wouldn't work if there are multiple lookups with different source filter based on the same source table that uses the same lookup cache. However, this could be overcome by writing a sql override
as shown below.....

Updating default database in Sybase version 9 and above

2009-10-04T13:51:00.000-07:00

If multiple repositories are built in the same database platform and they all reside on the same server it's a good practice to check what the default database is, before installing or upgrading.

Say for example you have Dev and Test repo set on <> server. If Dev repo is installed and now you would like to install Informatica in Test repo. You would have to issue the following command to update the default database in Sybase.

For example in Rapid SQL...

login with uid to old repository database(say old is Dev)
sp_modifylogin uid,defdb, TEST
 go
 sp_displaylogin uid

Bare bones JMX

2009-09-21T16:42:00.000-07:00

Some times you wish there was a way to monitor objects in a app while it is running to check for potential problems, or just to troubleshoot a problem. Well JMX can do that and much more. JMX can also do notification for example if a Queue is full or database is out of space, things like that. But I wanted to get started with just being able to monitor an object whenever I wished to.

The example used here is an application which is basically a Socket server serving clients with data from a queue. The use case here is to look at the app and query for number of clients connected or number of objects in the queue. pretty simple huh?

Here is the code for the app.
We will use JConsole to connect to the app process and invoke the operations defined.

Now you might say that this can be done by logging the information (queue size and number of clients). True but like I said earlier using JMX gives you a way to check on things at runtime. also you could have an operation to clear the queue for a secure user (how? check for later posts!). Also why do you want to waste cpu cycles and disk space to keep logging stuff which you may or not look for.

The Java artifacts you need are an interface that declares the methods that you want to expose, a class that implements that interface. That's it. TestJMX.java is just to create and register a JMX bean.

Interface....

package com.example.testJMX;

public interface DataSocketServerMBean {
 // operations
 public int numberofClients();
 public int dataQueueSize();
}

The implementor...

package com.example.testJMX;

import java.io.BufferedOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
 

 

 

 
import org.apache.log4j.Logger;

public class DataSocketServer extends Thread implements DataSocketServerMBean {
 private final static Logger log = Logger.getLogger(DataSocketServer.class);
 private final static int LISTENING_PORT = 3000;
 private ServerSocket weatherDataServer;
 public boolean shutdown = false;
 protected LinkedBlockingQueue outputDataQueue;
 private ConnectToNewClient newClient;
 private boolean byteData;
 private int listeningPort;

 public DataSocketServer() throws Exception {
  weatherDataServer = new ServerSocket(LISTENING_PORT);
  log.info("Server listening on port " + LISTENING_PORT);
  this.start();
  //start a new thread for writing data to each client socket
  newClient = new ConnectToNewClient();
 }

 public DataSocketServer(LinkedBlockingQueue dataQueue) { 
  this.outputDataQueue = dataQueue;

  try {
   weatherDataServer = new ServerSocket(LISTENING_PORT);
  } catch (IOException e) {
   log.error("Error creating ServerSocket :", e);
  }
  log.info("Server listening on port " + LISTENING_PORT);
  this.setName("WeatherDataSocketServerfor" + listeningPort);
  this.start();
  //start a new thread for writing data to each client socket
  newClient = new ConnectToNewClient();
 }

 public void run() {
  while (true) {
   try {
    log.debug("Waiting for connections.");
    if (weatherDataServer != null){
     Socket client = weatherDataServer.accept();
     log.info("Accepted a connection from: " + client.getInetAddress());
     newClient.addClient(new ClientSocket(client));
    }
    Thread.sleep(500);
   } catch (InterruptedException ix) {
    log.error("Shutting down socket server", ix);
    newClient.interrupt();
    break;
   } catch (Exception e) {
    log.error("Error accepting connections", e);
   }
  }
 }

 public void ConnectToClient(Socket clientSocket) {

 }

 /**
  * Helper class to handle each client as a separate thread.
  */
 class ConnectToNewClient extends Thread {
  // DECLARE
  private List clientSockets = Collections.synchronizedList(new ArrayList());

  // INIT
  public ConnectToNewClient() {
   this.setName("ConnectToNewClientFor" + DataSocketServer.this.listeningPort);
   this.start();
  }

  public void addClient(ClientSocket clientSocket) {
   clientSockets.add(clientSocket);
  }

  public ConnectToNewClient(ArrayList clientList) {
   this.clientSockets = clientList;
   this.setName("ConnectToNewClientFor" + DataSocketServer.this.listeningPort);
   this.start();
  }

  // ACTION
  public void run() {
   Object qobj = null;
   try {
    while (true) {
     while (clientSockets.size() > 0 && (qobj = DataSocketServer.this.outputDataQueue.peek()) != null) {
      log.info("Number of clients =" + clientSockets.size());
      //loop through each client and write to it.
      for (int i = 0; i < clientSockets.size(); i++) {
       ClientSocket clientSocket = null;
       try {
        //check if the socket is active.
        clientSocket = clientSockets.get(i);
        if (clientSocket.checkConnection()) {
         DataSocketServer.log.info("Sending data to client "
           + clientSocket.getInetAddress());
         if (DataSocketServer.this.byteData && qobj instanceof byte[])
          clientSocket.writeBytes((byte[]) qobj);
         else
          clientSocket.writeObject(qobj);
         DataSocketServer.this.outputDataQueue.remove(qobj);
        } else {
         //remove client from list
         DataSocketServer.log.info("Removing client from list : "
           + clientSocket.getInetAddress());
         clientSocket.close();
         clientSockets.remove(clientSocket);
         clientSocket = null;
        }
       } catch (RuntimeException e) {
        DataSocketServer.log.error("Error sending data to client ", e);
        //the client socket may be dead. remove it
        clientSockets.remove(clientSocket);
        clientSocket = null;
       }
      }
     }
     Thread.sleep(1000);
    }
   } catch (Exception e) {
    DataSocketServer.log.error("Error :", e);
    DataSocketServer.log.error("outputDataQueue =" + DataSocketServer.this.outputDataQueue);
    DataSocketServer.log.error("qobj = " + qobj);
   }
}
}

/**
* Socket wrapper for each client.
*/
class ClientSocket {
// DECLARE
private Socket socket = null;
private ObjectInputStream ois = null;
private OutputStream sos = null;
private ExecutorService executor;
private ObjectOutputStream oos = null;
private BufferedOutputStream bos = null;

// INIT
public ClientSocket(Socket clientSocket) {
this.socket = clientSocket;
try {
sos = socket.getOutputStream();
if (DataSocketServer.this.byteData)
bos   = new BufferedOutputStream(sos);
else
oos = new ObjectOutputStream(new BufferedOutputStream(sos));
} catch (Exception e1) {
try {
socket.close();
} catch (Exception e) {
log.error("Error creating client socket :" , e);
}
}
}

public void close() {
try {
socket.close();
if (oos != null)
oos.close();
if (bos != null)
bos.close();
} catch (Exception e) {
}
socket = null;
}

public void writeObject(Object o) {
try {
oos.flush();
if (o instanceof byte[]){
byte[] buf = (byte[])o;
bos.write(buf);
} else {
oos.writeObject(o);
}
oos.flush();
} catch (IOException e) {
// close socket.. connection maybe lost. The only way to find out is when
//we get a socketexception!!!!!!
log.error("Error writing to socket. closing socket...", e);
try {
socket.close();
} catch (IOException e1) {
}
socket = null;
} catch (Exception ex) {
log.error("Error writing data to client socket for client " + this.getInetAddress(),ex);
}
}

// UTIL
public boolean checkConnection() {
if (socket != null && !socket.isClosed() && socket.isConnected() && socket.isBound())
return true;
return false;
}

// ACCESS
public String getInetAddress() {
if (socket != null)
return socket.getInetAddress().getHostName();
return "";
}

public Socket getSocket() {
return socket;
}

public void writeBytes(byte[] buf) {
try {
if (socket != null) {
log.info("writing data " + new String(buf));
bos.flush();
bos.write(buf);
bos.flush();
}
} catch (IOException e) {
// close socket.. connection maybe lost. The only way to find out is when
//we get a socketexception!!!!!!
log.error("Error writing to socket. listening port: " + DataSocketServer.this.listeningPort + " closing socket...", e);
try {
socket.close();
} catch (IOException e1) {
}
socket = null;
} catch (Exception ex) {
log.error("Error writing data to client socket for client " + this.getInetAddress(),ex);
}
}
}

public int dataQueueSize() {
if (outputDataQueue != null)
return outputDataQueue.size();
return 0;
}

public int numberofClients() {
if (newClient != null)
return newClient.clientSockets.size();
return 0;
}

}

and finally the TestJMX class...

package com.example.testJMX;

import java.lang.management.*;
import java.util.Date;
import java.util.concurrent.LinkedBlockingQueue;

import javax.management.*;

import org.json.JSONObject;

public class Main {
    public static void main(String[] args) throws Exception {
    // Get the Platform MBean Server
    MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();

    LinkedBlockingQueue testQueue = new LinkedBlockingQueue();
    JSONObject sampleObj = new JSONObject();
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    
    // Construct the ObjectName for the MBean we will register
    ObjectName name = new ObjectName("com.example.testJMX:type=DataSocketServer");

    // Create the jmx MBean
    DataSocketServer mbean = new DataSocketServer(testQueue);

    // Register the jmx MBean
    mbs.registerMBean(mbean, name);

    // Wait forever
    System.out.println("Waiting forever...");
    Thread.sleep(Long.MAX_VALUE);
    }
}

Now run the TestJMX java class.
Launch JConsole and connect to the process
Invoking dataQueueSize and numberofClients operations...

connect to the socket server using telnet/netcat or any of those tools.
On linux ...
netcat localhost 4233
Now invoking the same operations again gives you the following results...

Capturing jvm Thread Dump in linux

2009-09-08T13:51:00.000-07:00

Many a times it has happened that a Java app froze and I needed to take a thread dump to look for deadlocks or any other issues. In this case you need to perform a Thread dump on a running process. Using kill command with argument 3 produces a Thread dump, but the output goes to the stdout of the process, Which could be a console on your favorite IDE. What if the IDE is frozen too?

Well in that case using the /proc command comes in handy.
ls /proc lists all the processes running on the system currently.

my-desktop:~$ ls  /proc
1      2133   24921  3132  3615  3744         fb             partitions
10     2134   25     3135  3616  4            filesystems    sched_debug
11     2141   25214  3139  3618  4763         fs             schedstat
12     2142   26     3159  3629  5            interrupts     scsi
12087  22     26408  3163  3633  5587         iomem          self
13     2213   26426  3165  3639  6            ioports        slabinfo
14     2258   2881   3183  3645  7            irq            stat
1495   2281   29     3184  3654  722          kallsyms       swaps
15     2283   2913   3198  3656  8            kcore          sys
16     22886  2935   32    3658  856          key-users      sysrq-trigger
16653  22894  2938   3211  3664  9            kmsg           sysvipc
17     2306   2939   3240  3669  acpi         kpagecount     timer_list
18     23177  294    33    3671  asound       kpageflags     timer_stats
1865   23186  3      3316  3673  buddyinfo    latency_stats  tty
19     23191  30     3344  3674  bus          loadavg        uptime
19405  23194  3002   34    3679  cgroups      locks          version
19406  23197  30137  3466  3683  cmdline      mdstat         version_signature
19407  2330   3032   35    3685  cpuinfo      meminfo        vmallocinfo
19445  23701  3077   3536  3691  crypto       misc           vmstat
19988  2389   3080   3549  3694  devices      modules        zoneinfo
2      24     3081   36    3707  diskstats    mounts
20     2431   309    3606  3724  dma          mtrr
20798  24772  3099   3609  3728  driver       net
21     24803  31     3610  3740  execdomains  pagetypeinfo

Say we want to take a thread dump of a java process (PID 24772).
/proc/24772/fd/1 gets you the stdout interface of the Java process where 1 is the file-descriptor for stdout.
You can capture the output to stdoutby issuing the command
cat /proc/24772/fd/1
Now to generate a thread dump issue this from another session

kill -3 24772
You should see the thread dump output like this..

my-desktop:~$ cat  /proc/24772/fd/1
2009-09-08 10:41:19
Full thread dump Java HotSpot(TM) Client VM (14.2-b01 mixed mode, sharing):

"Low Memory Detector" daemon prio=10 tid=0x09060c00 nid=0x60d0 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x0905dc00 nid=0x60ce waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0905c400 nid=0x60cd waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0904dc00 nid=0x60cc in Object.wait() [0xb5326000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x8bcd8918> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x8bcd8918> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x09049400 nid=0x60ca in Object.wait() [0xb5377000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x8bcd89a0> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x8bcd89a0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x09023400 nid=0x60c6 runnable [0xb75e9000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    - locked <0x8b8364b8> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.coastal.log4jext.SocketServerAsWinService.main(SocketServerAsWinService.java:41)

"VM Thread" prio=10 tid=0x09047800 nid=0x60c9 runnable 

"VM Periodic Task Thread" prio=10 tid=0x09062800 nid=0x60d1 waiting on condition 

JNI global references: 945

Heap
 def new generation   total 960K, used 488K [0x8b7d0000, 0x8b8d0000, 0x8bcb0000)
  eden space 896K,  47% used [0x8b7d0000, 0x8b83a3b8, 0x8b8b0000)
  from space 64K, 100% used [0x8b8c0000, 0x8b8d0000, 0x8b8d0000)
  to   space 64K,   0% used [0x8b8b0000, 0x8b8b0000, 0x8b8c0000)
 tenured generation   total 4096K, used 467K [0x8bcb0000, 0x8c0b0000, 0x8f7d0000)
   the space 4096K,  11% used [0x8bcb0000, 0x8bd24f00, 0x8bd25000, 0x8c0b0000)
 compacting perm gen  total 12288K, used 373K [0x8f7d0000, 0x903d0000, 0x937d0000)
   the space 12288K,   3% used [0x8f7d0000, 0x8f82d6a0, 0x8f82d800, 0x903d0000)
    ro space 8192K,  74% used [0x937d0000, 0x93dca2a8, 0x93dca400, 0x93fd0000)
    rw space 12288K,  59% used [0x93fd0000, 0x946e7878, 0x946e7a00, 0x94bd0000)

Search and replace using regular expression groups

2009-07-21T16:45:00.000-07:00

Regular expressions come in very handy for searching and replacing texts in data files. Most IDEs (eclipse on which this was tested) and text editors (TextPad, JEdit, etc.) support regular expressions. Take for example an input data file that you need to create/modify.
Here is the data file we are interested in...


07/10/09,11:10:20 AM,63463284,credit,salary electronic deposit,$1000.00
07/11/09,11:10:20 AM,63463284,credit,CD interest,$22.00
07/12/09,11:10:20 AM,63463284,credit,tax refund,$300.00
07/13/09,11:10:20 AM,63463284,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,63463284,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,63463284,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,63463284,debit,power bill,$80.00
07/17/09,11:10:20 AM,63463284,credit,Check #8774,$350.00
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,63463284,credit,Check #167,$1000.00
07/10/09,11:10:20 AM,77667529,credit,salary electronic deposit,$2000.00
07/11/09,11:10:20 AM,77667529,credit,CD interest,$22.00
07/12/09,11:10:20 AM,77667529,credit,tax refund,$300.00
07/13/09,11:10:20 AM,77667529,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,77667529,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,77667529,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,77667529,debit,power bill,$80.00
07/17/09,11:10:20 AM,77667529,credit,Check #7766,$350.00
07/18/09,11:10:20 AM,77667529,debit,Check copy fee,$10.00
07/19/09,11:10:20 AM,77667529,credit,Check #125,$1000.00
07/10/09,11:10:20 AM,33445398,credit,salary electronic deposit,$2000.00
07/11/09,11:10:20 AM,33445398,credit,CD interest,$22.00
07/12/09,11:10:20 AM,33445398,credit,tax refund,$300.00
07/13/09,11:10:20 AM,33445398,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,33445398,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,33445398,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,33445398,debit,power bill,$80.00
07/17/09,11:10:20 AM,33445398,credit,Check #1255,$350.00
07/18/09,11:10:20 AM,33445398,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,33445398,credit,Check #433,$1000.00

The data is about bank transactions conducted on accounts.
Let's take a look at the first few lines of data...

Date Time account # type description currency amount
------- ------- --------------- ------- ------------------ -------------- ------------
07/10/09,11:10:20 AM,63463284,credit,salary electronic deposit,$1000.00
07/11/09,11:10:20 AM,63463284,credit,CD interest,$22.00
07/12/09,11:10:20 AM,63463284,credit,tax refund,$300.00
07/13/09,11:10:20 AM,63463284,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,63463284,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,63463284,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,63463284,debit,power bill,$80.00
07/17/09,11:10:20 AM,63463284,credit,Check #8774,$350.00
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,63463284,credit,Check #167,$1000.00
..
..
..

Say we want to give credit to all overdraft fees charged for all accounts by adding an adjustment(credit) for the same amount. That is we want to add a new data line for every line which contains 'overdraft fee'. For example if you look at the lines of data above. For every line like this...
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
we want to add a new line like this....
07/18/09,11:10:20 AM,63463284,credit,overdraft fee adjustment,$5.00

The regular expression .*overdraft fee.*\n will get us the line containing overdraft fee. But we also need the timestamp at the beginning of the line. So let's refine the search string by adding a group.
^(.*)debit,overdraft fee,(.*)\n
So everything from the start of line upto debit forms group 1, which is the timestamp and account number.
The replacement string should be the entire last line which is $0 plus the same timestamp and account number($1) plus new text (credit,overdraft fee adjustment, $2) plus platform independent line feed (\R). Putting them all together the replacement string should be
$0$1credit,overdraft fee adjustment,$2\R
This was tested in eclipse IDE. In case of text editors like Textpad, the search string will be
^\$.*\$debit,overdraft fee,\$.*\$\n
and the replace string will be \0\1credit,overdraft fee adjustment,\2\n

Using the search and replace sub menu item of eclipse

Java System Tray icon in Ubuntu

2009-07-09T10:45:00.000-07:00

Download JDIC.

Using Synaptic package manager
search for jdic and download both libjdic -java and libjdic-bin packages.

downloading from JDIC website
download the linux distribution from here.
Say you extract it to the directory /home/myuser/jdic
You should see a linux folder which has jdic.jar and some .so files.

Here is a code example for creating a tray icon.

// Start Tray Icon Code
SystemTray tray = SystemTray.getDefaultSystemTray();
TrayIcon ti;

try {
if (Integer.parseInt(System.getProperty("java.version").substring(2, 3)) >= 5)
System.setProperty("javax.swing.adjustPopupLocationToFit", "false");
menu = new JPopupMenu("MY APP Menu");

// "Quit" menu item
menuItem = new JMenuItem("Exit");
menuItem.addActionListener(this);
menu.add(menuItem);

ImageIcon i = new ImageIcon(MyClass.class.getResource("/images/appImage.png"));

ti = new TrayIcon(i, "Welcome to My App", menu);

ti.setIconAutoSize(true);
ti.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
            MyApp.this.setVisible(!MyApp.this.isVisible());
            if (MyAppFrame.this.isVisible() == false) {
                     ti.displayMessage(
                                             "Welcome to MyApp",
          "To open  MyApp Application click the icon. To exit the application right click the icon and click the Quit menu item.",
          0);
            }
 }
});

tray.addTrayIcon(ti);

// Construct the GUI for balloon message.
ActionListener al = new ActionListener() {

public void actionPerformed(ActionEvent e) {
MyApp.this.setVisible(!MyApp.this.isVisible());
if (MyApp.this.isVisible() == false) {
ti.displayMessage(
  "MyApp",
   "To open the MyApp Application click the icon. To exit the application right click the icon and click the Quit menu item.",
                                          0);
          }
}
};
exitMnu.addActionListener(al);

} catch (Exception e) {
logger.error("Unhandled error", e);
}

try {
this.setIconImage(new ImageIcon(MyApp.class
  .getResource("/images/myApp.png")).getImage());
} catch (Exception e) {
logger.error("Unhandled error", e);
}
// End Tray Icon Code

For this to work, you need to add jdic jar to the build path and also add share library files location(/usr/lib/jni for the first case and /home/myuser/jdic/linux for the second) to the native library path for the project as shown in the picture below.

Mounting windows partition at bootup in Ubuntu

2009-07-06T17:15:00.000-07:00

I have a dual booting linux system at work. I share the java projects (located on the windows partition) on both windows and Linux. But when I launch eclipse, all the projects show up closed since the windows partition is not mounted yet. The solution to this is to mount windows partition at boot up time.

To mount a windows partition at startup, you need to do the following.
- get the UUID of the device

blkid
/dev/sda1: UUID="9044866844865140" TYPE="ntfs" 
/dev/sda2: LABEL="GHOST DATA" UUID="FCF8-3D81" TYPE="vfat"
/dev/sda5: UUID="b2293182-981b-4291-96ff-dba50f73510d" TYPE="ext3"
/dev/sda6: TYPE="swap" UUID="1d3b683c-c559-43e2-ac46-f3500e27dbfd"

- create an entry in /etc/fstab file for the windows NTFS drive

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID="9044866844865140" /media/disk ntfs-3g defaults,locale=en_US.utf8 0 0

Instead of UUID specifying the device also works. see below

/dev/sda1 /media/disk ntfs-3g defaults,locale=en_US.utf8 0 0

Note : both ntfs and ntfs-3g works fine for a windows partition.

The options for fstab are clearly explained in the ubuntu help docs here.

Ant build script for Flex projects

2009-06-04T16:41:00.000-07:00

It took me a while to get the build script working for Flex. I started out by googgling for them. Took some scripts out there and put it together. But ran into some errors like"Unable to transcode swf", "Out of memory heap space". Having fixed them all, wanted to share what i ended up with.

One compile error I was getting regularly was for embedded objects like swf or image files. Care needs to be taken to make sure that the paths are correct. For example inside a class which is deep in the source package tree, the path needs to be absolute. But for mxml files at the root level, relative paths are OK. Flex builder IDE or eclipse with Flex plugin will not raise a compile error for this issue. Only an ant build script will do that. FLEX_HOME environment variable needs to be set to the flex sdk directory.

Also the mxmlc compile task can take a lot of memory, so added maxmemory attribute to take care of that.You need to use fork=true (launch it's own jvm process instead of running under ant's jvm).

build.xml


<?xml version='1.0'?>
<project name='MYUIProject' default='masterbuild' basedir='.'>
<target name='init'>
<!--FLEX_HOME environment property should be set to where flex is installed-->
<property environment="env"/>
<property name="FLEX_HOME" value="${env.FLEX_HOME}"/>
<property name="DEFAULT.LOCAL" value="en_US"/>
<property name="ENCODING" value="UTF-8"/>
<!-- directories-->
<property name="application.name"  value="AppMain"/>
<property name="src.dir" value="src"/>
<property name="bin.dir" value="bin"/>
<property name="build.dir" value="build"/>
<property name="lib.dir" value="lib"/>
<property name="vendor.lib.dir" value="${lib.dir}/vendor"/>
<property name="dist.dir" value="${basedir}/dist"/>
<property name="wrapper.dir" value="html-template"/>
<property name="html.file" value="${dist.dir}/${application.name}.html"/>
<property name="main.class" value="${src.dir}/AppMain.mxml"/>
<property name="swf.export" value="${bin.dir}/AppMain.swf"/>

<!--ASSETS DIR-->
<property name="assets.dir" value="${src.dir}/assets/"/>
<path id="project.classpath">
<pathelement location="${build.dir}" />
<fileset dir="${lib.dir}">
 <include name="**/*.swc" />
</fileset>
<fileset dir="${vendor.lib.dir}">
 <include name="**/*.jar" />
</fileset>
</path>

<!-- Configure the custom Ant tasks -->
<taskdef resource="flexTasks.tasks" classpath="${lib.dir}/vendor/flexTasks.jar" />
<taskdef resource="com/adobe/ac/ant/tasks/tasks.properties" classpath="${FLEX.ANT.TASK.DIR}"/>

</target>

<target name='initdirs' depends="init">
<mkdir dir='${dist.dir}'/>
</target>

<!-- define all .mxml files and .css files that need to be converted into a .swf file
for the application-->
<target name="compile" depends="init, clean">
<antcall target="build-mxml-module">
<param name="module" value="AppMain" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="core" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="day" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="night" />
</antcall>
</target>

<!--Compile the "mxml" file passed in as an argument into a .swf file -->
<target name="build-mxml-module" >
<mxmlc fork="true" maxmemory="128m" file="${src.dir}/${module}.mxml"
keep-generated-actionscript="true"
output="${dist.dir}/${module}.swf"
actionscript-file-encoding="UTF-8"
optimize="true"
debug="false"
incremental="true">
<!-- Include the necessary libraries -->
<load-config filename="${FLEX_HOME}/frameworks/flex-config.xml"/>
<source-path path-element="${FLEX_HOME}/frameworks"/>
<compiler.source-path path-element="${src.dir}"/>
<compiler.include-libraries dir="${lib.dir}" append="true">
 <include name="as3crypto.swc" />
 <include name="AS3IteratorImplementation.swc" />
 <include name="Cairngorm.swc" />
 <include name="flexunit.swc" />
 <include name="flexlib-.2.1.swc" />
</compiler.include-libraries>
</mxmlc>
</target>

<!--Compile the "css" file passed in into a .swf file -->
<target name="build-css-module" >
<mxmlc fork="true" file="${src.dir}/${module}.css"
keep-generated-actionscript="true"
output="${dist.dir}/${module}.swf"
actionscript-file-encoding="UTF-8"
optimize="true"
debug="false"
incremental="true">
<!-- Include the necessary libraries -->
<load-config filename="${FLEX_HOME}/frameworks/flex-config.xml"/>
<source-path path-element="${FLEX_HOME}/frameworks"/>
<compiler.source-path path-element="${src.dir}"/>
<compiler.include-libraries dir="${lib.dir}" append="true">
 <include name="as3crypto.swc" />
 <include name="AS3IteratorImplementation.swc" />
 <include name="as3-rpclib.swc" />
 <include name="Cairngorm.swc" />
 <include name="flexunit.swc" />
 <include name="primitives.swc" />
 <include name="util.swc" />
</compiler.include-libraries>
</mxmlc>
</target>

<target name="clean" depends='initdirs'>
<delete dir="${dist.dir}" />
</target>

<!-- Create HTML wrapper -->
<target name="wrapper" depends="compile" description="Creates the HTML wrapper">
<html-wrapper
           title="Welcome to My Flex App"
           file="${application.name}.html"
           height="800"
           width="800"
           bgcolor="red"
           application="app"
           swf="${application.name}"
           version-major="9"
           version-minor="0"
           version-revision="0"
           history="true"
           template="express-installation"
           output="${dist.dir}"/>

</target>

<!-- Copy non-embedded files to output folder -->
<target name="output" depends="compile" description="Copies non-embedded files to output folder">
<!-- Copy the assets dir -->
<copy todir="${dist.dir}/assets">
<fileset dir="${assets.dir}">
</fileset>
</copy>
<!-- Copy the swf files from src dir -->
<copy todir="${dist.dir}">
<fileset dir="${src.dir}" includes="*.swf">
</fileset>
</copy>
</target>

<!-- Run all, default -->
<target name="masterbuild"
        depends="clean, compile, output, wrapper"
        description="Complete build in efficient sequence"/>
</project>

Watch and Curl

2009-05-22T11:37:00.000-07:00

Watch is a utility I have already discussed in my earlier post. I want to bring your attention to this nifty little utility called curl. It is basically a program to hit a URL and get data from it. There are hundreds of arguments that it can take. I won't be going into all that. I use it just as a tool to get data from a URL and here I combine it with the watch program to repeatedly execute the URL and also see the visible difference(s) in the output response from the URL.

The command used here is
watch -d -n 60 'curl "http://localhost:81/myapp/data/get-sample?userID=abc&password=foo"'

The output produced is shown below....

Where the highlighted text is what has changed in the output during the last time interval (which is 60 seconds in this case).

Top 7 Qualities of an Expert Programmer

2009-04-22T10:28:00.000-07:00

This article from dzone nails it and will probably hold true for the life of a programmer! It is well thought out and well written. Any developer from novice to expert can relate to many of the points outlined.

Looping in Informatica

2009-04-19T15:55:00.000-07:00

Separate mapping/session to get all pid from a file and load the pid's in a relational table(load_result table). The table could consist of the following fields.. pid, datawarehouse transaction datetime, completion_status(with a status of 'I').
Create a mapping/session to ouput a param flat file. The mapping should have a logic to pick the (min) of pid where the completion_status is an 'I'. Snapshot of lod_rlst table before the staging or fact load commences

1 I
01/01/2009 12:00:00:000 AM
Call your next dummy session to load staging or fact tables from post-session command(components tab). The Link between param file(step 2 above) and next dummy session (step 3) should say. Previous_task = succeeded and previous_target_success_rows > 0. (We want to call the next session to load data warehouse tables, if only the param file has any pid in it).
After data is loaded through staging/facts, at the end of all successful staging/facts sessions, create another dummy session with a post_sql to update the completion_status for the pid that was just loaded from 'I' to 'C' (Incomplete status to a Complete Status). Now delete from load_result table where completion_status is 'C'. In post_session success command of this dummy session, it should call step 2 above. Snapshot of load result table after a load is completed successfully but before it's deleted from lod_rslt table

01/01/2009 12:00:00:000 AM

5. The link in step 2 will help the process continue if there are any pid's left in lod_rslt table. If all pid's are done, the session should complete gracefully and send an email.

Monitoring changes over time...

2009-04-12T19:26:00.000-07:00

If you need to run a command repeatedly at some interval and watch the output and know what has changed from the last iteration. For example say you want to monitor an XML feed which is providing data from weather sensors and you want to see the data every 2 minutes and also see the new data highlighted? try the Unix watch utility. It has parameters to specify update interval and also supports matching wildcard.

Monitoring multiple log files simultaneously

2009-04-12T19:06:00.000-07:00

Ever wanted to look at many log files in real time while it's being updated? Multitail is a Unix utility program which can be used to monitor multiple log files and it will break up the display console into one unit for each file. It can also use colors while displaying the logfiles (through regular expressions). Supports monitoring wild card and more. You can monitor entire directories of files.
Also runs on Cygwin.

Search and grouping results in command line unix

2009-01-30T14:39:00.000-08:00

Ever wanted to extract some metrics from log files and group the results. Also print the count for each group. All this without using a loop statement!!!
Just using command line. Here is how it can be done.

In the results, the first number is the count and the second is the group.

For starters here is a bit of explanation...The grep is looking for the string Failed to connect in service.log file. sed is used to make a group. In this case it is the station number. Then the results are sorted, finally uniq gives the unique rows and the -c gives us the count for each group.
BTW this was done on Ubuntu.

$ grep 'Failed to connect' service.log | sed 's/.*Station :$.*$/\1/' | sort | uniq -c
count station

  69    10 - Host : 123.456.103.106 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
356    11 - Host : 123.456.103.105 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    1 - Host : 123.456.103.107 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    2 - Host : 123.456.103.98 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
23    3 - Host : 123.456.103.99 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    4 - Host : 123.456.103.100 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
25    5 - Host : 123.456.103.101 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
25    6 - Host : 123.456.103.102 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
23    7 - Host : 123.456.103.97 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
34    8 - Host : 123.456.103.104 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
28    9 - Host : 123.456.103.103 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect

Regular Expression Cheat sheet for Java

2009-01-27T11:12:00.000-08:00

Regular expression is quirky enough. On top of that it's different for each platform. Even text editors have their own flavors. One way to keep your sanity is to have a cheat sheet for each environment. Here is one for Java and a generic one.

System Clock vs Real Time clock

2009-01-02T13:38:00.000-08:00

Applications which are time sensitive or store and process data that is periodic need to keep up with the Real Time clock (RTC) (hardware clock). Time servers can be used to synch the system clock with external time servers (for example www.nist.gov) but there is no way for applications to synchronize the system clock with the RTC.

Typically at startup the operating system (read windows) sets the system clock to the RTC. After that the operating system generates interrupts(depending on the flavor of windows it could range from 10 ms to 15 ms or more) to update the system clock. Also if Windows finds the time lag between the two clocks to be at least 1 minute (Where exactly this value is set is not known publicly!!!), then it resets the system clock to the RTC. But if you want to keep up the system time much more accurately (say 100 ms, then there is no way to do that.

ClockMon is a simple utility app that can be used to monitor the two clocks or resync the system clock when it drifts away from the RTC. It is a free program that can come quite handy when troubleshooting clock drift issues in applications running on Windows platforms.