Top 7 Qualities of an Expert Programmer

This article from dzone nails it and will probably hold true for the life of a programmer! It is well thought out and well written. Any developer from novice to expert can relate to many of the points outlined.

Looping in Informatica

1. Separate mapping/session to get all pid from a file and load the pid's in a relational table(load_result table). The table could consist of the following fields.. pid, datawarehouse transaction datetime, completion_status(with a status of 'I').
2. Create a mapping/session to ouput a param flat file. The mapping should have a logic to pick the (min) of pid where the completion_status is an 'I'. Snapshot of lod_rlst table before the staging or fact load commences
   
   
   
   
   

   1

   I

   01/01/2009 12:00:00:000 AM

   
   
3. Call your next dummy session to load staging or fact tables from post-session command(components tab). The Link between param file(step 2 above) and next dummy session (step 3) should say. Previous_task = succeeded and previous_target_success_rows > 0. (We want to call the next session to load data warehouse tables, if only the param file has any pid in it).
4. After data is loaded through staging/facts, at the end of all successful staging/facts sessions, create another dummy session with a post_sql to update the completion_status for the pid that was just loaded from 'I' to 'C' (Incomplete status to a Complete Status). Now delete from load_result table where completion_status is 'C'. In post_session success command of this dummy session, it should call step 2 above. Snapshot of load result table after a load is completed successfully but before it's deleted from lod_rslt table
   

1

C

01/01/2009 12:00:00:000 AM

5. The link in step 2 will help the process continue if there are any pid's left in lod_rslt table. If all pid's are done, the session should complete gracefully and send an email.

Monitoring changes over time...

If you need to run a command repeatedly at some interval and watch the output and know what has changed from the last iteration. For example say you want to monitor an XML feed which is providing data from weather sensors and you want to see the data every 2 minutes and also see the new data highlighted? try the Unix watch utility. It has parameters to specify update interval and also supports matching wildcard.

Monitoring multiple log files simultaneously

Ever wanted to look at many log files in real time while it's being updated? Multitail is a Unix utility program which can be used to monitor multiple log files and it will break up the display console into one unit for each file. It can also use colors while displaying the logfiles (through regular expressions). Supports monitoring wild card and more. You can monitor entire directories of files.
Also runs on Cygwin.

Search and grouping results in command line unix

Ever wanted to extract some metrics from log files and group the results. Also print the count for each group. All this without using a loop statement!!!
Just using command line. Here is how it can be done.

In the results, the first number is the count and the second is the group.

For starters here is a bit of explanation...The grep is looking for the string Failed to connect in service.log file. sed is used to make a group. In this case it is the station number. Then the results are sorted, finally uniq gives the unique rows and the -c gives us the count for each group.
BTW this was done on Ubuntu.

$ grep 'Failed to connect' service.log | sed 's/.*Station :\(.*\)/\1/' | sort | uniq -c
count station

  69    10 - Host : 123.456.103.106 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
356    11 - Host : 123.456.103.105 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    1 - Host : 123.456.103.107 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    2 - Host : 123.456.103.98 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
23    3 - Host : 123.456.103.99 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
24    4 - Host : 123.456.103.100 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
25    5 - Host : 123.456.103.101 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
25    6 - Host : 123.456.103.102 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
23    7 - Host : 123.456.103.97 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
34    8 - Host : 123.456.103.104 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect
28    9 - Host : 123.456.103.103 -  Failed to connect - Error :java.net.ConnectException: Connection timed out: connect 

Regular Expression Cheat sheet for Java

Regular expression is quirky enough. On top of that it's different for each platform. Even text editors have their own flavors. One way to keep your sanity is to have a cheat sheet for each environment. Here is one for Java and a generic one.

System Clock vs Real Time clock

Applications which are time sensitive or store and process data that is periodic need to keep up with the Real Time clock (RTC) (hardware clock). Time servers can be used to synch the system clock with external time servers (for example www.nist.gov) but there is no way for applications to synchronize the system clock with the RTC.

Typically at startup the operating system (read windows) sets the system clock to the RTC. After that the operating system generates interrupts(depending on the flavor of windows it could range from 10 ms to 15 ms or more) to update the system clock. Also if Windows finds the time lag between the two clocks to be at least 1 minute (Where exactly this value is set is not known publicly!!!), then it resets the system clock to the RTC. But if you want to keep up the system time much more accurately (say 100 ms, then there is no way to do that.

ClockMon is a simple utility app that can be used to monitor the two clocks or resync the system clock when it drifts away from the RTC. It is a free program that can come quite handy when troubleshooting clock drift issues in applications running on Windows platforms.