Search and replace using regular expression groups

Regular expressions come in very handy for searching and replacing texts in data files. Most IDEs (eclipse on which this was tested) and text editors (TextPad, JEdit, etc.) support regular expressions. Take for example an input data file that you need to create/modify.
Here is the data file we are interested in...

07/10/09,11:10:20 AM,63463284,credit,salary electronic deposit,$1000.00
07/11/09,11:10:20 AM,63463284,credit,CD interest,$22.00
07/12/09,11:10:20 AM,63463284,credit,tax refund,$300.00
07/13/09,11:10:20 AM,63463284,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,63463284,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,63463284,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,63463284,debit,power bill,$80.00
07/17/09,11:10:20 AM,63463284,credit,Check #8774,$350.00
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,63463284,credit,Check #167,$1000.00
07/10/09,11:10:20 AM,77667529,credit,salary electronic deposit,$2000.00
07/11/09,11:10:20 AM,77667529,credit,CD interest,$22.00
07/12/09,11:10:20 AM,77667529,credit,tax refund,$300.00
07/13/09,11:10:20 AM,77667529,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,77667529,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,77667529,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,77667529,debit,power bill,$80.00
07/17/09,11:10:20 AM,77667529,credit,Check #7766,$350.00
07/18/09,11:10:20 AM,77667529,debit,Check copy fee,$10.00
07/19/09,11:10:20 AM,77667529,credit,Check #125,$1000.00
07/10/09,11:10:20 AM,33445398,credit,salary electronic deposit,$2000.00
07/11/09,11:10:20 AM,33445398,credit,CD interest,$22.00
07/12/09,11:10:20 AM,33445398,credit,tax refund,$300.00
07/13/09,11:10:20 AM,33445398,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,33445398,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,33445398,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,33445398,debit,power bill,$80.00
07/17/09,11:10:20 AM,33445398,credit,Check #1255,$350.00
07/18/09,11:10:20 AM,33445398,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,33445398,credit,Check #433,$1000.00

The data is about bank transactions conducted on accounts.
Let's take a look at the first few lines of data...

Date Time account # type description currency amount
------- ------- --------------- ------- ------------------ -------------- ------------
07/10/09,11:10:20 AM,63463284,credit,salary electronic deposit,$1000.00
07/11/09,11:10:20 AM,63463284,credit,CD interest,$22.00
07/12/09,11:10:20 AM,63463284,credit,tax refund,$300.00
07/13/09,11:10:20 AM,63463284,credit,ATM deposit,$500.00
07/14/09,11:10:20 AM,63463284,debit,credit card payment,$137.00
07/15/09,11:10:20 AM,63463284,credit,checking a/c interest,$20.00
07/16/09,11:10:20 AM,63463284,debit,power bill,$80.00
07/17/09,11:10:20 AM,63463284,credit,Check #8774,$350.00
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
07/19/09,11:10:20 AM,63463284,credit,Check #167,$1000.00
..
..
..

Say we want to give credit to all overdraft fees charged for all accounts by adding an adjustment(credit) for the same amount. That is we want to add a new data line for every line which contains 'overdraft fee'. For example if you look at the lines of data above. For every line like this...
07/18/09,11:10:20 AM,63463284,debit,overdraft fee,$5.00
we want to add a new line like this....
07/18/09,11:10:20 AM,63463284,credit,overdraft fee adjustment,$5.00

The regular expression .*overdraft fee.*\n will get us the line containing overdraft fee. But we also need the timestamp at the beginning of the line. So let's refine the search string by adding a group.
^(.*)debit,overdraft fee,(.*)\n
So everything from the start of line upto debit forms group 1, which is the timestamp and account number.
The replacement string should be the entire last line which is $0 plus the same timestamp and account number($1) plus new text (credit,overdraft fee adjustment, $2) plus platform independent line feed (\R). Putting them all together the replacement string should be
$0$1credit,overdraft fee adjustment,$2\R
This was tested in eclipse IDE. In case of text editors like Textpad, the search string will be
^\\(.*\\)debit,overdraft fee,\\(.*\\)\n
and the replace string will be \0\1credit,overdraft fee adjustment,\2\n

Using the search and replace sub menu item of eclipse

Java System Tray icon in Ubuntu

Download JDIC.

• Using Synaptic package manager
  search for jdic and download both libjdic -java and libjdic-bin packages.

• downloading from JDIC website
  download the linux distribution from here.
  Say you extract it to the directory /home/myuser/jdic
  You should see a linux folder which has jdic.jar and some .so files.
  

Here is a code example for creating a tray icon.

// Start Tray Icon Code
SystemTray tray = SystemTray.getDefaultSystemTray();
TrayIcon ti;

try {
if (Integer.parseInt(System.getProperty("java.version").substring(2, 3)) >= 5)
System.setProperty("javax.swing.adjustPopupLocationToFit", "false");
menu = new JPopupMenu("MY APP Menu");

// "Quit" menu item
menuItem = new JMenuItem("Exit");
menuItem.addActionListener(this);
menu.add(menuItem);

ImageIcon i = new ImageIcon(MyClass.class.getResource("/images/appImage.png"));

ti = new TrayIcon(i, "Welcome to My App", menu);

ti.setIconAutoSize(true);
ti.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
            MyApp.this.setVisible(!MyApp.this.isVisible());
            if (MyAppFrame.this.isVisible() == false) {
                     ti.displayMessage(
                                             "Welcome to MyApp",
          "To open  MyApp Application click the icon. To exit the application right click the icon and click the Quit menu item.",
          0);
            }
 }
});

tray.addTrayIcon(ti);

// Construct the GUI for balloon message.
ActionListener al = new ActionListener() {

public void actionPerformed(ActionEvent e) {
MyApp.this.setVisible(!MyApp.this.isVisible());
if (MyApp.this.isVisible() == false) {
ti.displayMessage(
  "MyApp",
   "To open the MyApp Application click the icon. To exit the application right click the icon and click the Quit menu item.",
                                          0);
          }
}
};
exitMnu.addActionListener(al);

} catch (Exception e) {
logger.error("Unhandled error", e);
}

try {
this.setIconImage(new ImageIcon(MyApp.class
  .getResource("/images/myApp.png")).getImage());
} catch (Exception e) {
logger.error("Unhandled error", e);
}
// End Tray Icon Code        

For this to work, you need to add jdic jar to the build path and also add share library files location(/usr/lib/jni for the first case and /home/myuser/jdic/linux for the second) to the native library path for the project as shown in the picture below.

Mounting windows partition at bootup in Ubuntu

I have a dual booting linux system at work. I share the java projects (located on the windows partition) on both windows and Linux. But when I launch eclipse, all the projects show up closed since the windows partition is not mounted yet. The solution to this is to mount windows partition at boot up time.

To mount a windows partition at startup, you need to do the following.
- get the UUID of the device

blkid
/dev/sda1: UUID="9044866844865140" TYPE="ntfs" 
/dev/sda2: LABEL="GHOST DATA" UUID="FCF8-3D81" TYPE="vfat"
/dev/sda5: UUID="b2293182-981b-4291-96ff-dba50f73510d" TYPE="ext3"
/dev/sda6: TYPE="swap" UUID="1d3b683c-c559-43e2-ac46-f3500e27dbfd"

- create an entry in /etc/fstab file for the windows NTFS drive

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID="9044866844865140" /media/disk ntfs-3g defaults,locale=en_US.utf8 0 0

Instead of UUID specifying the device also works. see below

/dev/sda1 /media/disk ntfs-3g defaults,locale=en_US.utf8 0 0 

Note : both ntfs and ntfs-3g works fine for a windows partition.

The options for fstab are clearly explained in the ubuntu help docs here.

Ant build script for Flex projects

It took me a while to get the build script working for Flex. I started out by googgling for them. Took some scripts out there and put it together. But ran into some errors like"Unable to transcode swf", "Out of memory heap space". Having fixed them all, wanted to share what i ended up with.

One compile error I was getting regularly was for embedded objects like swf or image files. Care needs to be taken to make sure that the paths are correct. For example inside a class which is deep in the source package tree, the path needs to be absolute. But for mxml files at the root level, relative paths are OK. Flex builder IDE or eclipse with Flex plugin will not raise a compile error for this issue. Only an ant build script will do that. FLEX_HOME environment variable needs to be set to the flex sdk directory.

Also the mxmlc compile task can take a lot of memory, so added maxmemory attribute to take care of that.You need to use fork=true (launch it's own jvm process instead of running under ant's jvm).

build.xml

<?xml version='1.0'?>
<project name='MYUIProject' default='masterbuild' basedir='.'>
<target name='init'>
<!--FLEX_HOME environment property should be set to where flex is installed-->
<property environment="env"/>
<property name="FLEX_HOME" value="${env.FLEX_HOME}"/>
<property name="DEFAULT.LOCAL" value="en_US"/>
<property name="ENCODING" value="UTF-8"/>
<!-- directories-->
<property name="application.name"  value="AppMain"/>
<property name="src.dir" value="src"/>
<property name="bin.dir" value="bin"/>
<property name="build.dir" value="build"/>
<property name="lib.dir" value="lib"/>
<property name="vendor.lib.dir" value="${lib.dir}/vendor"/>
<property name="dist.dir" value="${basedir}/dist"/>
<property name="wrapper.dir" value="html-template"/>
<property name="html.file" value="${dist.dir}/${application.name}.html"/>
<property name="main.class" value="${src.dir}/AppMain.mxml"/>
<property name="swf.export" value="${bin.dir}/AppMain.swf"/>

<!--ASSETS DIR-->
<property name="assets.dir" value="${src.dir}/assets/"/>
<path id="project.classpath">
<pathelement location="${build.dir}" />
<fileset dir="${lib.dir}">
 <include name="**/*.swc" />
</fileset>
<fileset dir="${vendor.lib.dir}">
 <include name="**/*.jar" />
</fileset>
</path>

<!-- Configure the custom Ant tasks -->
<taskdef resource="flexTasks.tasks" classpath="${lib.dir}/vendor/flexTasks.jar" />
<taskdef resource="com/adobe/ac/ant/tasks/tasks.properties" classpath="${FLEX.ANT.TASK.DIR}"/>

</target>

<target name='initdirs' depends="init">
<mkdir dir='${dist.dir}'/>
</target>

<!-- define all .mxml files and .css files that need to be converted into a .swf file
for the application-->
<target name="compile" depends="init, clean">
<antcall target="build-mxml-module">
<param name="module" value="AppMain" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="core" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="day" />
</antcall>
<antcall target="build-css-module">
<param name="module" value="night" />
</antcall>
</target>

<!--Compile the "mxml" file passed in as an argument into a .swf file -->
<target name="build-mxml-module" >
<mxmlc fork="true" maxmemory="128m" file="${src.dir}/${module}.mxml"
keep-generated-actionscript="true"
output="${dist.dir}/${module}.swf"
actionscript-file-encoding="UTF-8"
optimize="true"
debug="false"
incremental="true">
<!-- Include the necessary libraries -->
<load-config filename="${FLEX_HOME}/frameworks/flex-config.xml"/>
<source-path path-element="${FLEX_HOME}/frameworks"/>
<compiler.source-path path-element="${src.dir}"/>
<compiler.include-libraries dir="${lib.dir}" append="true">
 <include name="as3crypto.swc" />
 <include name="AS3IteratorImplementation.swc" />
 <include name="Cairngorm.swc" />
 <include name="flexunit.swc" />
 <include name="flexlib-.2.1.swc" />
</compiler.include-libraries>
</mxmlc>
</target>

<!--Compile the "css" file passed in into a .swf file -->
<target name="build-css-module" >
<mxmlc fork="true" file="${src.dir}/${module}.css"
keep-generated-actionscript="true"
output="${dist.dir}/${module}.swf"
actionscript-file-encoding="UTF-8"
optimize="true"
debug="false"
incremental="true">
<!-- Include the necessary libraries -->
<load-config filename="${FLEX_HOME}/frameworks/flex-config.xml"/>
<source-path path-element="${FLEX_HOME}/frameworks"/>
<compiler.source-path path-element="${src.dir}"/>
<compiler.include-libraries dir="${lib.dir}" append="true">
 <include name="as3crypto.swc" />
 <include name="AS3IteratorImplementation.swc" />
 <include name="as3-rpclib.swc" />
 <include name="Cairngorm.swc" />
 <include name="flexunit.swc" />
 <include name="primitives.swc" />
 <include name="util.swc" />
</compiler.include-libraries>
</mxmlc>
</target>

<target name="clean" depends='initdirs'>
<delete dir="${dist.dir}" />
</target>

<!-- Create HTML wrapper -->
<target name="wrapper" depends="compile" description="Creates the HTML wrapper">
<html-wrapper
           title="Welcome to My Flex App"
           file="${application.name}.html"
           height="800"
           width="800"
           bgcolor="red"
           application="app"
           swf="${application.name}"
           version-major="9"
           version-minor="0"
           version-revision="0"
           history="true"
           template="express-installation"
           output="${dist.dir}"/>

</target>

<!-- Copy non-embedded files to output folder -->
<target name="output" depends="compile" description="Copies non-embedded files to output folder">
<!-- Copy the assets dir -->
<copy todir="${dist.dir}/assets">
<fileset dir="${assets.dir}">
</fileset>
</copy>
<!-- Copy the swf files from src dir -->
<copy todir="${dist.dir}">
<fileset dir="${src.dir}" includes="*.swf">
</fileset>
</copy>
</target>

<!-- Run all, default -->
<target name="masterbuild"
        depends="clean, compile, output, wrapper"
        description="Complete build in efficient sequence"/>
</project>

Watch and Curl

Watch is a utility I have already discussed in my earlier post. I want to bring your attention to this nifty little utility called curl. It is basically a program to hit a URL and get data from it. There are hundreds of arguments that it can take. I won't be going into all that. I use it just as a tool to get data from a URL and here I combine it with the watch program to repeatedly execute the URL and also see the visible difference(s) in the output response from the URL.

The command used here is
watch -d -n 60 'curl "http://localhost:81/myapp/data/get-sample?userID=abc&password=foo"'

The output produced is shown below....

















Where the highlighted text is what has changed in the output during the last time interval (which is 60 seconds in this case).

Top 7 Qualities of an Expert Programmer

This article from dzone nails it and will probably hold true for the life of a programmer! It is well thought out and well written. Any developer from novice to expert can relate to many of the points outlined.

Looping in Informatica

1. Separate mapping/session to get all pid from a file and load the pid's in a relational table(load_result table). The table could consist of the following fields.. pid, datawarehouse transaction datetime, completion_status(with a status of 'I').
2. Create a mapping/session to ouput a param flat file. The mapping should have a logic to pick the (min) of pid where the completion_status is an 'I'. Snapshot of lod_rlst table before the staging or fact load commences
   
   
   
   
   

   1

   I

   01/01/2009 12:00:00:000 AM

   
   
3. Call your next dummy session to load staging or fact tables from post-session command(components tab). The Link between param file(step 2 above) and next dummy session (step 3) should say. Previous_task = succeeded and previous_target_success_rows > 0. (We want to call the next session to load data warehouse tables, if only the param file has any pid in it).
4. After data is loaded through staging/facts, at the end of all successful staging/facts sessions, create another dummy session with a post_sql to update the completion_status for the pid that was just loaded from 'I' to 'C' (Incomplete status to a Complete Status). Now delete from load_result table where completion_status is 'C'. In post_session success command of this dummy session, it should call step 2 above. Snapshot of load result table after a load is completed successfully but before it's deleted from lod_rslt table
   

1

C

01/01/2009 12:00:00:000 AM

5. The link in step 2 will help the process continue if there are any pid's left in lod_rslt table. If all pid's are done, the session should complete gracefully and send an email.