Lookup Source Filter option in Informatica

Informatica version 8.6.1 has a feature where Lookup Source filter could be used. This feature wouldn't work if  there are multiple lookups with different source filter based on the same source table that uses the same lookup cache. However, this could be overcome by writing a sql override
as shown below.....

 

Updating default database in Sybase version 9 and above

If multiple repositories are built in the same database platform and they all reside on the same server it's a good practice to check what the default database is, before installing or upgrading. 

Say for example you have Dev and Test repo set on <> server. If Dev repo is installed and now you would like to install Informatica in  Test repo. You would have to issue the following command to update the default database in Sybase.

For example in Rapid SQL...

login with uid to old repository database(say old is Dev)
sp_modifylogin uid,defdb, TEST
 go
 sp_displaylogin uid

Bare bones JMX

Some times you wish there was a way to monitor objects in a app while it is running to check for potential problems, or just to troubleshoot a problem. Well JMX can do that and much more. JMX can also do notification for example if a Queue is full or database is out of space, things like that. But I wanted to get started with just being able to monitor an object whenever I wished to.

The example used here is an application which is basically a Socket server serving clients with data from a queue. The use case here is to look at the app and query for number of clients connected or number of objects in the queue. pretty simple huh?

Here is the code for the app.
We will use JConsole to connect to the app process and invoke the operations defined.

Now you might say that this can be done by logging the information (queue size and number of clients). True but like I said earlier using JMX gives you a way to check on things at runtime. also you could have an operation to clear the queue for a secure user (how? check for later posts!). Also why do you want to waste cpu cycles and disk space to keep logging stuff which you may or not look for.

The Java artifacts you need are an interface that declares the methods that you want to expose, a class that implements that interface. That's it. TestJMX.java is just to create and register a JMX bean.

Interface....

package com.example.testJMX;

public interface DataSocketServerMBean {
 // operations
 public int numberofClients();
 public int dataQueueSize();
}

The implementor...

package com.example.testJMX;

import java.io.BufferedOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
 

 

 

 
import org.apache.log4j.Logger;

public class DataSocketServer extends Thread implements DataSocketServerMBean {
 private final static Logger log = Logger.getLogger(DataSocketServer.class);
 private final static int LISTENING_PORT = 3000;
 private ServerSocket weatherDataServer;
 public boolean shutdown = false;
 protected LinkedBlockingQueue outputDataQueue;
 private ConnectToNewClient newClient;
 private boolean byteData;
 private int listeningPort;

 public DataSocketServer() throws Exception {
  weatherDataServer = new ServerSocket(LISTENING_PORT);
  log.info("Server listening on port " + LISTENING_PORT);
  this.start();
  //start a new thread for writing data to each client socket
  newClient = new ConnectToNewClient();
 }

 public DataSocketServer(LinkedBlockingQueue dataQueue) { 
  this.outputDataQueue = dataQueue;

  try {
   weatherDataServer = new ServerSocket(LISTENING_PORT);
  } catch (IOException e) {
   log.error("Error creating ServerSocket :", e);
  }
  log.info("Server listening on port " + LISTENING_PORT);
  this.setName("WeatherDataSocketServerfor" + listeningPort);
  this.start();
  //start a new thread for writing data to each client socket
  newClient = new ConnectToNewClient();
 }

 public void run() {
  while (true) {
   try {
    log.debug("Waiting for connections.");
    if (weatherDataServer != null){
     Socket client = weatherDataServer.accept();
     log.info("Accepted a connection from: " + client.getInetAddress());
     newClient.addClient(new ClientSocket(client));
    }
    Thread.sleep(500);
   } catch (InterruptedException ix) {
    log.error("Shutting down socket server", ix);
    newClient.interrupt();
    break;
   } catch (Exception e) {
    log.error("Error accepting connections", e);
   }
  }
 }

 public void ConnectToClient(Socket clientSocket) {

 }

 /**
  * Helper class to handle each client as a separate thread.
  */
 class ConnectToNewClient extends Thread {
  // DECLARE
  private List clientSockets = Collections.synchronizedList(new ArrayList());

  // INIT
  public ConnectToNewClient() {
   this.setName("ConnectToNewClientFor" + DataSocketServer.this.listeningPort);
   this.start();
  }

  public void addClient(ClientSocket clientSocket) {
   clientSockets.add(clientSocket);
  }

  public ConnectToNewClient(ArrayList clientList) {
   this.clientSockets = clientList;
   this.setName("ConnectToNewClientFor" + DataSocketServer.this.listeningPort);
   this.start();
  }

  // ACTION
  public void run() {
   Object qobj = null;
   try {
    while (true) {
     while (clientSockets.size() > 0 && (qobj = DataSocketServer.this.outputDataQueue.peek()) != null) {
      log.info("Number of clients =" + clientSockets.size());
      //loop through each client and write to it.
      for (int i = 0; i < clientSockets.size(); i++) {
       ClientSocket clientSocket = null;
       try {
        //check if the socket is active.
        clientSocket = clientSockets.get(i);
        if (clientSocket.checkConnection()) {
         DataSocketServer.log.info("Sending data to client "
           + clientSocket.getInetAddress());
         if (DataSocketServer.this.byteData && qobj instanceof byte[])
          clientSocket.writeBytes((byte[]) qobj);
         else
          clientSocket.writeObject(qobj);
         DataSocketServer.this.outputDataQueue.remove(qobj);
        } else {
         //remove client from list
         DataSocketServer.log.info("Removing client from list : "
           + clientSocket.getInetAddress());
         clientSocket.close();
         clientSockets.remove(clientSocket);
         clientSocket = null;
        }
       } catch (RuntimeException e) {
        DataSocketServer.log.error("Error sending data to client ", e);
        //the client socket may be dead. remove it
        clientSockets.remove(clientSocket);
        clientSocket = null;
       }
      }
     }
     Thread.sleep(1000);
    }
   } catch (Exception e) {
    DataSocketServer.log.error("Error :", e);
    DataSocketServer.log.error("outputDataQueue =" + DataSocketServer.this.outputDataQueue);
    DataSocketServer.log.error("qobj = " + qobj);
   }
}
}

/**
* Socket wrapper for each client.
*/
class ClientSocket {
// DECLARE
private Socket socket = null;
private ObjectInputStream ois = null;
private OutputStream sos = null;
private ExecutorService executor;
private ObjectOutputStream oos = null;
private BufferedOutputStream bos = null;

// INIT
public ClientSocket(Socket clientSocket) {
this.socket = clientSocket;
try {
sos = socket.getOutputStream();
if (DataSocketServer.this.byteData)
bos   = new BufferedOutputStream(sos);
else
oos = new ObjectOutputStream(new BufferedOutputStream(sos));
} catch (Exception e1) {
try {
socket.close();
} catch (Exception e) {
log.error("Error creating client socket :" , e);
}
}
}

public void close() {
try {
socket.close();
if (oos != null)
oos.close();
if (bos != null)
bos.close();
} catch (Exception e) {
}
socket = null;
}

public void writeObject(Object o) {
try {
oos.flush();
if (o instanceof byte[]){
byte[] buf = (byte[])o;
bos.write(buf);
} else {
oos.writeObject(o);
}
oos.flush();
} catch (IOException e) {
// close socket.. connection maybe lost. The only way to find out is when
//we get a socketexception!!!!!!
log.error("Error writing to socket. closing socket...", e);
try {
socket.close();
} catch (IOException e1) {
}
socket = null;
} catch (Exception ex) {
log.error("Error writing data to client socket for client " + this.getInetAddress(),ex);
}
}

// UTIL
public boolean checkConnection() {
if (socket != null && !socket.isClosed() && socket.isConnected() && socket.isBound())
return true;
return false;
}

// ACCESS
public String getInetAddress() {
if (socket != null)
return socket.getInetAddress().getHostName();
return "";
}

public Socket getSocket() {
return socket;
}

public void writeBytes(byte[] buf) {
try {
if (socket != null) {
log.info("writing data " + new String(buf));
bos.flush();
bos.write(buf);
bos.flush();
}
} catch (IOException e) {
// close socket.. connection maybe lost. The only way to find out is when
//we get a socketexception!!!!!!
log.error("Error writing to socket. listening port: " + DataSocketServer.this.listeningPort + " closing socket...", e);
try {
socket.close();
} catch (IOException e1) {
}
socket = null;
} catch (Exception ex) {
log.error("Error writing data to client socket for client " + this.getInetAddress(),ex);
}
}
}

public int dataQueueSize() {
if (outputDataQueue != null)
return outputDataQueue.size();
return 0;
}

public int numberofClients() {
if (newClient != null)
return newClient.clientSockets.size();
return 0;
}

}

and finally the TestJMX class...

package com.example.testJMX;

import java.lang.management.*;
import java.util.Date;
import java.util.concurrent.LinkedBlockingQueue;

import javax.management.*;

import org.json.JSONObject;

public class Main {
    public static void main(String[] args) throws Exception {
    // Get the Platform MBean Server
    MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();

    LinkedBlockingQueue testQueue = new LinkedBlockingQueue();
    JSONObject sampleObj = new JSONObject();
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    sampleObj.put("The current time on host machine", new Date());
    testQueue.add(sampleObj.toString());
    
    // Construct the ObjectName for the MBean we will register
    ObjectName name = new ObjectName("com.example.testJMX:type=DataSocketServer");

    // Create the jmx MBean
    DataSocketServer mbean = new DataSocketServer(testQueue);

    // Register the jmx MBean
    mbs.registerMBean(mbean, name);

    // Wait forever
    System.out.println("Waiting forever...");
    Thread.sleep(Long.MAX_VALUE);
    }
} 

Now run the TestJMX java class.
Launch JConsole and connect to the process
Invoking dataQueueSize and numberofClients operations...

connect to the socket server using telnet/netcat or any of those tools.
On linux ...
netcat localhost 4233
Now invoking the same operations again gives you the following results...

Capturing jvm Thread Dump in linux

Many a times it has happened that a Java app froze and I needed to take a thread dump to look for deadlocks or any other issues. In this case you need to perform a Thread dump on a running process. Using kill command with argument 3 produces a Thread dump, but the output goes to the stdout of the process, Which could be a console on your favorite IDE. What if the IDE is frozen too?

Well in that case using the /proc command comes in handy.
ls /proc lists all the processes running on the system currently.

my-desktop:~$ ls  /proc
1      2133   24921  3132  3615  3744         fb             partitions
10     2134   25     3135  3616  4            filesystems    sched_debug
11     2141   25214  3139  3618  4763         fs             schedstat
12     2142   26     3159  3629  5            interrupts     scsi
12087  22     26408  3163  3633  5587         iomem          self
13     2213   26426  3165  3639  6            ioports        slabinfo
14     2258   2881   3183  3645  7            irq            stat
1495   2281   29     3184  3654  722          kallsyms       swaps
15     2283   2913   3198  3656  8            kcore          sys
16     22886  2935   32    3658  856          key-users      sysrq-trigger
16653  22894  2938   3211  3664  9            kmsg           sysvipc
17     2306   2939   3240  3669  acpi         kpagecount     timer_list
18     23177  294    33    3671  asound       kpageflags     timer_stats
1865   23186  3      3316  3673  buddyinfo    latency_stats  tty
19     23191  30     3344  3674  bus          loadavg        uptime
19405  23194  3002   34    3679  cgroups      locks          version
19406  23197  30137  3466  3683  cmdline      mdstat         version_signature
19407  2330   3032   35    3685  cpuinfo      meminfo        vmallocinfo
19445  23701  3077   3536  3691  crypto       misc           vmstat
19988  2389   3080   3549  3694  devices      modules        zoneinfo
2      24     3081   36    3707  diskstats    mounts
20     2431   309    3606  3724  dma          mtrr
20798  24772  3099   3609  3728  driver       net
21     24803  31     3610  3740  execdomains  pagetypeinfo

Say we want to take a thread dump of a java process (PID 24772).  
/proc/24772/fd/1 gets you the stdout interface of the Java process where 1 is the file-descriptor for stdout.
You can capture the output to stdoutby issuing the command
cat  /proc/24772/fd/1
Now to generate a thread dump issue this from another session

kill -3  24772
You should see the thread dump output like this..

my-desktop:~$ cat  /proc/24772/fd/1
2009-09-08 10:41:19
Full thread dump Java HotSpot(TM) Client VM (14.2-b01 mixed mode, sharing):

"Low Memory Detector" daemon prio=10 tid=0x09060c00 nid=0x60d0 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x0905dc00 nid=0x60ce waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0905c400 nid=0x60cd waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0904dc00 nid=0x60cc in Object.wait() [0xb5326000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x8bcd8918> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x8bcd8918> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x09049400 nid=0x60ca in Object.wait() [0xb5377000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x8bcd89a0> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x8bcd89a0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x09023400 nid=0x60c6 runnable [0xb75e9000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    - locked <0x8b8364b8> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.coastal.log4jext.SocketServerAsWinService.main(SocketServerAsWinService.java:41)

"VM Thread" prio=10 tid=0x09047800 nid=0x60c9 runnable 

"VM Periodic Task Thread" prio=10 tid=0x09062800 nid=0x60d1 waiting on condition 

JNI global references: 945

Heap
 def new generation   total 960K, used 488K [0x8b7d0000, 0x8b8d0000, 0x8bcb0000)
  eden space 896K,  47% used [0x8b7d0000, 0x8b83a3b8, 0x8b8b0000)
  from space 64K, 100% used [0x8b8c0000, 0x8b8d0000, 0x8b8d0000)
  to   space 64K,   0% used [0x8b8b0000, 0x8b8b0000, 0x8b8c0000)
 tenured generation   total 4096K, used 467K [0x8bcb0000, 0x8c0b0000, 0x8f7d0000)
   the space 4096K,  11% used [0x8bcb0000, 0x8bd24f00, 0x8bd25000, 0x8c0b0000)
 compacting perm gen  total 12288K, used 373K [0x8f7d0000, 0x903d0000, 0x937d0000)
   the space 12288K,   3% used [0x8f7d0000, 0x8f82d6a0, 0x8f82d800, 0x903d0000)
    ro space 8192K,  74% used [0x937d0000, 0x93dca2a8, 0x93dca400, 0x93fd0000)
    rw space 12288K,  59% used [0x93fd0000, 0x946e7878, 0x946e7a00, 0x94bd0000)

Search and replace using regular expression groups

Regular expressions come in very handy for searching and replacing texts in data files. Most IDEs (eclipse on which this was tested) and text editors (TextPad, JEdit, etc.) support regular expressions. Take for example an input data file that you need to create/modify.
Here is the data file we are interested in...

07/10/09,11:10:20 AM,63463284,credit,salary electronic deposit,$1000.00
07/11/09,11:10:20 AM,63463284,credit,CD interest,$22.00
07/12/09,11:10:20 AM,63463284,credit,tax refund,$300.00
07/13/09,11:10:20 AM,63463284,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,63463284,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,63463284,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,63463284,debit,power bill,$80.00
07/17/09,11:10:20 AM,63463284,credit,Check #8774,$350.00
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,63463284,credit,Check #167,$1000.00
07/10/09,11:10:20 AM,77667529,credit,salary electronic deposit,$2000.00
07/11/09,11:10:20 AM,77667529,credit,CD interest,$22.00
07/12/09,11:10:20 AM,77667529,credit,tax refund,$300.00
07/13/09,11:10:20 AM,77667529,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,77667529,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,77667529,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,77667529,debit,power bill,$80.00
07/17/09,11:10:20 AM,77667529,credit,Check #7766,$350.00
07/18/09,11:10:20 AM,77667529,debit,Check copy fee,$10.00
07/19/09,11:10:20 AM,77667529,credit,Check #125,$1000.00
07/10/09,11:10:20 AM,33445398,credit,salary electronic deposit,$2000.00
07/11/09,11:10:20 AM,33445398,credit,CD interest,$22.00
07/12/09,11:10:20 AM,33445398,credit,tax refund,$300.00
07/13/09,11:10:20 AM,33445398,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,33445398,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,33445398,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,33445398,debit,power bill,$80.00
07/17/09,11:10:20 AM,33445398,credit,Check #1255,$350.00
07/18/09,11:10:20 AM,33445398,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,33445398,credit,Check #433,$1000.00

The data is about bank transactions conducted on accounts.
Let's take a look at the first few lines of data...

Date Time account # type description currency amount
------- ------- --------------- ------- ------------------ -------------- ------------
07/10/09,11:10:20 AM,63463284,credit,salary electronic deposit,$1000.00
07/11/09,11:10:20 AM,63463284,credit,CD interest,$22.00
07/12/09,11:10:20 AM,63463284,credit,tax refund,$300.00
07/13/09,11:10:20 AM,63463284,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,63463284,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,63463284,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,63463284,debit,power bill,$80.00
07/17/09,11:10:20 AM,63463284,credit,Check #8774,$350.00
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,63463284,credit,Check #167,$1000.00
..
..
..

Say we want to give credit to all overdraft fees charged for all accounts by adding an adjustment(credit) for the same amount. That is we want to add a new data line for every line which contains 'overdraft fee'. For example if you look at the lines of data above. For every line like this...
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
we want to add a new line like this....
07/18/09,11:10:20 AM,63463284,credit,overdraft fee adjustment,$5.00

The regular expression .*overdraft fee.*\n will get us the line containing overdraft fee. But we also need the timestamp at the beginning of the line. So let's refine the search string by adding a group.
^(.*)debit,overdraft fee,(.*)\n
So everything from the start of line upto debit forms group 1, which is the timestamp and account number.
The replacement string should be the entire last line which is $0 plus the same timestamp and account number($1) plus new text (credit,overdraft fee adjustment, $2) plus platform independent line feed (\R). Putting them all together the replacement string should be
$0$1credit,overdraft fee adjustment,$2\R
This was tested in eclipse IDE. In case of text editors like Textpad, the search string will be
^\\(.*\\)debit,overdraft fee,\\(.*\\)\n
and the replace string will be \0\1credit,overdraft fee adjustment,\2\n

Using the search and replace sub menu item of eclipse

Java System Tray icon in Ubuntu

Download JDIC.

• Using Synaptic package manager
  search for jdic and download both libjdic -java and libjdic-bin packages.

• downloading from JDIC website
  download the linux distribution from here.
  Say you extract it to the directory /home/myuser/jdic
  You should see a linux folder which has jdic.jar and some .so files.
  

Here is a code example for creating a tray icon.

// Start Tray Icon Code
SystemTray tray = SystemTray.getDefaultSystemTray();
TrayIcon ti;

try {
if (Integer.parseInt(System.getProperty("java.version").substring(2, 3)) >= 5)
System.setProperty("javax.swing.adjustPopupLocationToFit", "false");
menu = new JPopupMenu("MY APP Menu");

// "Quit" menu item
menuItem = new JMenuItem("Exit");
menuItem.addActionListener(this);
menu.add(menuItem);

ImageIcon i = new ImageIcon(MyClass.class.getResource("/images/appImage.png"));

ti = new TrayIcon(i, "Welcome to My App", menu);

ti.setIconAutoSize(true);
ti.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
            MyApp.this.setVisible(!MyApp.this.isVisible());
            if (MyAppFrame.this.isVisible() == false) {
                     ti.displayMessage(
                                             "Welcome to MyApp",
          "To open  MyApp Application click the icon. To exit the application right click the icon and click the Quit menu item.",
          0);
            }
 }
});

tray.addTrayIcon(ti);

// Construct the GUI for balloon message.
ActionListener al = new ActionListener() {

public void actionPerformed(ActionEvent e) {
MyApp.this.setVisible(!MyApp.this.isVisible());
if (MyApp.this.isVisible() == false) {
ti.displayMessage(
  "MyApp",
   "To open the MyApp Application click the icon. To exit the application right click the icon and click the Quit menu item.",
                                          0);
          }
}
};
exitMnu.addActionListener(al);

} catch (Exception e) {
logger.error("Unhandled error", e);
}

try {
this.setIconImage(new ImageIcon(MyApp.class
  .getResource("/images/myApp.png")).getImage());
} catch (Exception e) {
logger.error("Unhandled error", e);
}
// End Tray Icon Code        

For this to work, you need to add jdic jar to the build path and also add share library files location(/usr/lib/jni for the first case and /home/myuser/jdic/linux for the second) to the native library path for the project as shown in the picture below.

Mounting windows partition at bootup in Ubuntu

I have a dual booting linux system at work. I share the java projects (located on the windows partition) on both windows and Linux. But when I launch eclipse, all the projects show up closed since the windows partition is not mounted yet. The solution to this is to mount windows partition at boot up time.

To mount a windows partition at startup, you need to do the following.
- get the UUID of the device

blkid
/dev/sda1: UUID="9044866844865140" TYPE="ntfs" 
/dev/sda2: LABEL="GHOST DATA" UUID="FCF8-3D81" TYPE="vfat"
/dev/sda5: UUID="b2293182-981b-4291-96ff-dba50f73510d" TYPE="ext3"
/dev/sda6: TYPE="swap" UUID="1d3b683c-c559-43e2-ac46-f3500e27dbfd"

- create an entry in /etc/fstab file for the windows NTFS drive

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID="9044866844865140" /media/disk ntfs-3g defaults,locale=en_US.utf8 0 0

Instead of UUID specifying the device also works. see below

/dev/sda1 /media/disk ntfs-3g defaults,locale=en_US.utf8 0 0 

Note : both ntfs and ntfs-3g works fine for a windows partition.

The options for fstab are clearly explained in the ubuntu help docs here.

Ant build script for Flex projects

It took me a while to get the build script working for Flex. I started out by googgling for them. Took some scripts out there and put it together. But ran into some errors like"Unable to transcode swf", "Out of memory heap space". Having fixed them all, wanted to share what i ended up with.

One compile error I was getting regularly was for embedded objects like swf or image files. Care needs to be taken to make sure that the paths are correct. For example inside a class which is deep in the source package tree, the path needs to be absolute. But for mxml files at the root level, relative paths are OK. Flex builder IDE or eclipse with Flex plugin will not raise a compile error for this issue. Only an ant build script will do that. FLEX_HOME environment variable needs to be set to the flex sdk directory.

Also the mxmlc compile task can take a lot of memory, so added maxmemory attribute to take care of that.You need to use fork=true (launch it's own jvm process instead of running under ant's jvm).

build.xml

<?xml version='1.0'?>
<project name='MYUIProject' default='masterbuild' basedir='.'>
<target name='init'>
<!--FLEX_HOME environment property should be set to where flex is installed-->
<property environment="env"/>
<property name="FLEX_HOME" value="${env.FLEX_HOME}"/>
<property name="DEFAULT.LOCAL" value="en_US"/>
<property name="ENCODING" value="UTF-8"/>
<!-- directories-->
<property name="application.name"  value="AppMain"/>
<property name="src.dir" value="src"/>
<property name="bin.dir" value="bin"/>
<property name="build.dir" value="build"/>
<property name="lib.dir" value="lib"/>
<property name="vendor.lib.dir" value="${lib.dir}/vendor"/>
<property name="dist.dir" value="${basedir}/dist"/>
<property name="wrapper.dir" value="html-template"/>
<property name="html.file" value="${dist.dir}/${application.name}.html"/>
<property name="main.class" value="${src.dir}/AppMain.mxml"/>
<property name="swf.export" value="${bin.dir}/AppMain.swf"/>

<!--ASSETS DIR-->
<property name="assets.dir" value="${src.dir}/assets/"/>
<path id="project.classpath">
<pathelement location="${build.dir}" />
<fileset dir="${lib.dir}">
 <include name="**/*.swc" />
</fileset>
<fileset dir="${vendor.lib.dir}">
 <include name="**/*.jar" />
</fileset>
</path>

<!-- Configure the custom Ant tasks -->
<taskdef resource="flexTasks.tasks" classpath="${lib.dir}/vendor/flexTasks.jar" />
<taskdef resource="com/adobe/ac/ant/tasks/tasks.properties" classpath="${FLEX.ANT.TASK.DIR}"/>

</target>

<target name='initdirs' depends="init">
<mkdir dir='${dist.dir}'/>
</target>

<!-- define all .mxml files and .css files that need to be converted into a .swf file
for the application-->
<target name="compile" depends="init, clean">
<antcall target="build-mxml-module">
<param name="module" value="AppMain" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="core" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="day" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="night" />
</antcall>
</target>

<!--Compile the "mxml" file passed in as an argument into a .swf file -->
<target name="build-mxml-module" >
<mxmlc fork="true" maxmemory="128m" file="${src.dir}/${module}.mxml"
keep-generated-actionscript="true"
output="${dist.dir}/${module}.swf"
actionscript-file-encoding="UTF-8"
optimize="true"
debug="false"
incremental="true">
<!-- Include the necessary libraries -->
<load-config filename="${FLEX_HOME}/frameworks/flex-config.xml"/>
<source-path path-element="${FLEX_HOME}/frameworks"/>
<compiler.source-path path-element="${src.dir}"/>
<compiler.include-libraries dir="${lib.dir}" append="true">
 <include name="as3crypto.swc" />
 <include name="AS3IteratorImplementation.swc" />
 <include name="Cairngorm.swc" />
 <include name="flexunit.swc" />
 <include name="flexlib-.2.1.swc" />
</compiler.include-libraries>
</mxmlc>
</target>

<!--Compile the "css" file passed in into a .swf file -->
<target name="build-css-module" >
<mxmlc fork="true" file="${src.dir}/${module}.css"
keep-generated-actionscript="true"
output="${dist.dir}/${module}.swf"
actionscript-file-encoding="UTF-8"
optimize="true"
debug="false"
incremental="true">
<!-- Include the necessary libraries -->
<load-config filename="${FLEX_HOME}/frameworks/flex-config.xml"/>
<source-path path-element="${FLEX_HOME}/frameworks"/>
<compiler.source-path path-element="${src.dir}"/>
<compiler.include-libraries dir="${lib.dir}" append="true">
 <include name="as3crypto.swc" />
 <include name="AS3IteratorImplementation.swc" />
 <include name="as3-rpclib.swc" />
 <include name="Cairngorm.swc" />
 <include name="flexunit.swc" />
 <include name="primitives.swc" />
 <include name="util.swc" />
</compiler.include-libraries>
</mxmlc>
</target>

<target name="clean" depends='initdirs'>
<delete dir="${dist.dir}" />
</target>

<!-- Create HTML wrapper -->
<target name="wrapper" depends="compile" description="Creates the HTML wrapper">
<html-wrapper
           title="Welcome to My Flex App"
           file="${application.name}.html"
           height="800"
           width="800"
           bgcolor="red"
           application="app"
           swf="${application.name}"
           version-major="9"
           version-minor="0"
           version-revision="0"
           history="true"
           template="express-installation"
           output="${dist.dir}"/>

</target>

<!-- Copy non-embedded files to output folder -->
<target name="output" depends="compile" description="Copies non-embedded files to output folder">
<!-- Copy the assets dir -->
<copy todir="${dist.dir}/assets">
<fileset dir="${assets.dir}">
</fileset>
</copy>
<!-- Copy the swf files from src dir -->
<copy todir="${dist.dir}">
<fileset dir="${src.dir}" includes="*.swf">
</fileset>
</copy>
</target>

<!-- Run all, default -->
<target name="masterbuild"
        depends="clean, compile, output, wrapper"
        description="Complete build in efficient sequence"/>
</project>

Watch and Curl

Watch is a utility I have already discussed in my earlier post. I want to bring your attention to this nifty little utility called curl. It is basically a program to hit a URL and get data from it. There are hundreds of arguments that it can take. I won't be going into all that. I use it just as a tool to get data from a URL and here I combine it with the watch program to repeatedly execute the URL and also see the visible difference(s) in the output response from the URL.

The command used here is
watch -d -n 60 'curl "http://localhost:81/myapp/data/get-sample?userID=abc&password=foo"'

The output produced is shown below....

















Where the highlighted text is what has changed in the output during the last time interval (which is 60 seconds in this case).

Top 7 Qualities of an Expert Programmer

This article from dzone nails it and will probably hold true for the life of a programmer! It is well thought out and well written. Any developer from novice to expert can relate to many of the points outlined.

Looping in Informatica

1. Separate mapping/session to get all pid from a file and load the pid's in a relational table(load_result table). The table could consist of the following fields.. pid, datawarehouse transaction datetime, completion_status(with a status of 'I').
2. Create a mapping/session to ouput a param flat file. The mapping should have a logic to pick the (min) of pid where the completion_status is an 'I'. Snapshot of lod_rlst table before the staging or fact load commences
   
   
   
   
   

   1

   I

   01/01/2009 12:00:00:000 AM

   
   
3. Call your next dummy session to load staging or fact tables from post-session command(components tab). The Link between param file(step 2 above) and next dummy session (step 3) should say. Previous_task = succeeded and previous_target_success_rows > 0. (We want to call the next session to load data warehouse tables, if only the param file has any pid in it).
4. After data is loaded through staging/facts, at the end of all successful staging/facts sessions, create another dummy session with a post_sql to update the completion_status for the pid that was just loaded from 'I' to 'C' (Incomplete status to a Complete Status). Now delete from load_result table where completion_status is 'C'. In post_session success command of this dummy session, it should call step 2 above. Snapshot of load result table after a load is completed successfully but before it's deleted from lod_rslt table
   

1

C

01/01/2009 12:00:00:000 AM

5. The link in step 2 will help the process continue if there are any pid's left in lod_rslt table. If all pid's are done, the session should complete gracefully and send an email.

Monitoring changes over time...

If you need to run a command repeatedly at some interval and watch the output and know what has changed from the last iteration. For example say you want to monitor an XML feed which is providing data from weather sensors and you want to see the data every 2 minutes and also see the new data highlighted? try the Unix watch utility. It has parameters to specify update interval and also supports matching wildcard.

Monitoring multiple log files simultaneously

Ever wanted to look at many log files in real time while it's being updated? Multitail is a Unix utility program which can be used to monitor multiple log files and it will break up the display console into one unit for each file. It can also use colors while displaying the logfiles (through regular expressions). Supports monitoring wild card and more. You can monitor entire directories of files.
Also runs on Cygwin.

Search and grouping results in command line unix

Ever wanted to extract some metrics from log files and group the results. Also print the count for each group. All this without using a loop statement!!!
Just using command line. Here is how it can be done.

In the results, the first number is the count and the second is the group.

For starters here is a bit of explanation...The grep is looking for the string Failed to connect in service.log file. sed is used to make a group. In this case it is the station number. Then the results are sorted, finally uniq gives the unique rows and the -c gives us the count for each group.
BTW this was done on Ubuntu.

$ grep 'Failed to connect' service.log | sed 's/.*Station :\(.*\)/\1/' | sort | uniq -c
count station

  69    10 - Host : 123.456.103.106 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
356    11 - Host : 123.456.103.105 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    1 - Host : 123.456.103.107 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    2 - Host : 123.456.103.98 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
23    3 - Host : 123.456.103.99 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    4 - Host : 123.456.103.100 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
25    5 - Host : 123.456.103.101 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
25    6 - Host : 123.456.103.102 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
23    7 - Host : 123.456.103.97 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
34    8 - Host : 123.456.103.104 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
28    9 - Host : 123.456.103.103 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect 

Regular Expression Cheat sheet for Java

Regular expression is quirky enough. On top of that it's different for each platform. Even text editors have their own flavors. One way to keep your sanity is to have a cheat sheet for each environment. Here is one for Java and a generic one.

System Clock vs Real Time clock

Applications which are time sensitive or store and process data that is periodic need to keep up with the Real Time clock (RTC) (hardware clock). Time servers can be used to synch the system clock with external time servers (for example www.nist.gov) but there is no way for applications to synchronize the system clock with the RTC.

Typically at startup the operating system (read windows) sets the system clock to the RTC. After that the operating system generates interrupts(depending on the flavor of windows it could range from 10 ms to 15 ms or more) to update the system clock. Also if Windows finds the time lag between the two clocks to be at least 1 minute (Where exactly this value is set is not known publicly!!!), then it resets the system clock to the RTC. But if you want to keep up the system time much more accurately (say 100 ms, then there is no way to do that.

ClockMon is a simple utility app that can be used to monitor the two clocks or resync the system clock when it drifts away from the RTC. It is a free program that can come quite handy when troubleshooting clock drift issues in applications running on Windows platforms.