PHP - OpenID library integration

An excellent introduction to the openID library as a consumer from the PHP here.

Erlang based WebSocket client in place

An erlang based WebSocket client in place here , for clients that deal with web socket protocol as yet. 

Storing Data in a Hash - Erlang

Came across this nice example in the Erlang mailing list for storing data in an hashmap.

-module(db_server).

-export([start/0, init/1, write/2, read/1, delete/1]).

start() ->
register(server, spawn(db_server, init, [dict:new()])).

init(Records) ->
receive
{add, Pid, Key, Value} ->
RecordsNew = dict:store(Key, Value, Records),
Pid ! {ok, Key, Value},
init(RecordsNew);
{show, Pid, Key} ->
case dict:find(Key, Records) of
{ok, Value} -> Pid ! {ok, Value};
error ->  Pid ! {error, no_such_value}
end,
init(Records);
{delete, Pid, Key} ->
RecordsNew = dict:erase(Key, Records),
Pid ! {ok, ok},
init(RecordsNew)           
end.

write(Key, Value) ->
server ! {add, self(), Key, Value},
receive Res ->
Res
end.

read(Key) ->
server ! {show, self(), Key},
receive Res ->
Res
end.

delete(Key) ->
server ! {delete, self(), Key},
receive Res ->
Res
end.

Deleting tags from remote in git

Happened to go through a build process , and while flipping around with versions - it created some tags on the remote repository that I wanted to get rid of entirely.

For example - lets assume the tag name is artifactA-0.1.0 .

$ git tag -d artifactA-0.1.0 

This deletes the tag locally ( in your local clone )
To push the change to remote and to delete the tag remotely as well - we can give -

$ git push origin :artifactA-0.1.0

where , I assume origin is the name of the remote branch from which I cloned initially.

This should delete the tag remotely as well.

GPG agent

When preparing some artifacts to be published to a maven repository - needed some help with gpg publishing.

More often that not - when the gpg key verification was happening - it was reporting about a missing file -    ~/.gnupg/S.gpg-agent .

'touch'ing would not help because that is not a file , but a socket for the agent to listen on.

$ gpg-agent --use-standard-socket --daemon 2>/dev/null

This makes the agent listen on the socket.

Google Collections SVN repository

Google Collections 1.0 was released recently towards the end of December 2009. While very useful from an API and performance perspective, the API had quite an amount of surprises / deprecations / removals in its later stages ( 0.7 / 0.8 etc.) .

With 1.0 the API seems to have been stabilized and as an added benefit for those integrating with mvn - it is also available here as the mvn repository - http://google-maven-repository.googlecode.com/svn/repository/com/google/collections/google-collections/1.0/ .

com.google.collections / google-collections / 1.0 should do the trick in ivy.xml / pom.xml as appropriate.

libevent 2.0 released

As per this update on the google developer blog , libevent 2.0 seems to be released.

For those of you new to the library - libevent provides a platform agnostic event handling library so that the user does deal with the quirks of the operating systems like Linux and Solaris and chooses the best event handling adapter present in the kernel ( Eg: In Linux, from 2.6 - epoll performs much better than poll / select . The former in O(1) in handling of connections whereas the latter group is O(n) in event handling proportional to the number of active connections at that time instant ).

More details are available in the book available here.

Specifically about the update - it seems like the developers on Windows would benefit a lot from the API changes made. I will probably write a more detailed review of the same after playing around with the software / API on various platforms and distributions.

Resetting / Overwriting /etc/resolv.conf in EC2 instance

Creates a new EC2 instance ( CentOS ) and by default it comes with a dns resolver to the outside world.

For our purposes - we had set up an internal dns server running (bind - process 'named' on a particular host).

When we were launching our pool of servers - we wanted to make sure that the new instances fall under the same domain that we specify it to be.

We also wanted to set the nameserver of the newly created instances pointing to the internal DNS server we have to resolve the ambiguity we have.

Before making the change - the file /etc/resolv.conf was looking as follows.

# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search some.internal.aws.domain
nameserver 172.x.y.z 

where 172.x.y.z was something set by Amazon EC2 by default.

Edit /etc/dhclient.conf ( Create one , if it does not exist )

supersede  domain-name "ec2.mycompany.com" ;
prepend domain-name-servers 10.p.q.r ;my internal company dns

I also found it useful to create an Elastic IP and associate the dns server instance with the elastic ip,
and then have the prepend domain-name-servers refer to the elastic ip , instead of the internal ip.
So - even if the internal dns server fails = we can reconstruct it from another AMI and associate with the elastic ip without affecting the rest of the system.

To see the changes to /etc/dhclient , do the following

$ dhclient -r ; dhclient 

This command forcibly renews the dhcp lease that will force the new credentials from /etc/dhclient.conf to be read.

After doing this , verify /etc/resolv.conf

search ec2.mycompany.com
nameserver 10.p.q.r    ; our DNS server
nameserver 172.x.y.z  ; the one set by amzn

So that completes the process and we are good about it.

Zookeeper build

Was trying to build Zookeeper from trunk ( 3.3.0 ) .

Came across this error  -

create-cppunit-configure:
     [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found in library
     [exec] libtoolize: putting auxiliary files in `.'.
     [exec] libtoolize: copying file `./config.guess'
     [exec] libtoolize: copying file `./config.sub'
     [exec] libtoolize: copying file `./install-sh'
     [exec] libtoolize: copying file `./ltmain.sh'
     [exec] libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and
     [exec] libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree.
     [exec] libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
     [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found in library
     [exec] configure.ac:33: error: possibly undefined macro: AM_PATH_CPPUNIT
     [exec]       If this token and others are legitimate, please use m4_pattern_allow.
     [exec]       See the Autoconf documentation.
     [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1

Installed -

$  sudo apt-get install libcppunit-dev

Ended up with ..

     [exec] .../zookeeper/src/c/configure: line 5015: syntax error near unexpected token `1.10.2'
     [exec] .../zookeeper/src/c/configure: line 5015: `   AM_PATH_CPPUNIT(1.10.2)'

Hmm.. bad times.

Installing Thrift Continued

Continuing my previous post of installing Thrift - needed to install the following obvious ones as well.

 sudo apt-get install flex bison 

And then , run ./configure once again to make sure all the libraries are linked together.

 ./configure

This should regenerate the makefiles once again, after installing flex and bison.

Installing Thrift

Checked out thrift from the trunk on my ubuntu box ( 9.10 ) .

The first step on the installation was running the program

$ ./bootstrap.sh

It failed with some errors , some fairly obvious (missing autoconf ) - some not so obvious.
The first step is to get autoconf installed.

$ sudo apt-get install autoconf

Then ran into this error.

$ ./bootstrap.sh
configure.ac:44: error: possibly undefined macro: AC_PROG_LIBTOOL
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure.ac:26: installing `./install-sh'
configure.ac:26: installing `./missing'
compiler/cpp/Makefile.am: installing `./depcomp'
configure.ac: installing `./ylwrap'
lib/cpp/Makefile.am:24: Libtool library used but `LIBTOOL' is undefined
lib/cpp/Makefile.am:24:   The usual way to define `LIBTOOL' is to add `AC_PROG_LIBTOOL'
lib/cpp/Makefile.am:24:   to `configure.ac' and run `aclocal' and `autoconf' again.
lib/cpp/Makefile.am:24:   If `AC_PROG_LIBTOOL' is in `configure.ac', make sure
lib/cpp/Makefile.am:24:   its definition is in aclocal's search path.
test/Makefile.am:30: Libtool library used but `LIBTOOL' is undefined
test/Makefile.am:30:   The usual way to define `LIBTOOL' is to add `AC_PROG_LIBTOOL'
test/Makefile.am:30:   to `configure.ac' and run `aclocal' and `autoconf' again.
test/Makefile.am:30:   If `AC_PROG_LIBTOOL' is in `configure.ac', make sure
test/Makefile.am:30:   its definition is in aclocal's search path.

The fix was to install libtool.
$ sudo apt-get install libtool


And then comes the boost libraries -

$ sudo apt-get install libboost1.40-dev libboost1.40-doc

(Your boost library version might be different from mine but you get the idea !!).

HDFS Permissions issues

If you are running into permissions issue in hdfs installation (Eg: When you try to write to hdfs and that does not seem to work ) - then you may want to *relax* the permissions by the following setting in all hdfs cluster nodes , in hdfs-site.xml ( starting from 0.20 ). Of course - the namednode needs to be restarted for the changes to be effective.

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

Important - this has a gaping security hole in itself by relaxing the permissions and currently the hdfs team is actively working on enabling better permission based access rules. So - this change is best in the early stages of development to get started and should be revisited once again soon after.

iBator eclipse plugin

If you are like me and you are into using iBatis eclipse plugin - check out http://ibatis.apache.org/ibator.html. It comes with an eclipse plugin that is quite useful if you are working with the ibatis configuration files.

Handling signals in Java

Signals by definition are specific to the underlying implementation ( read, operating system) .

Java, being a platform independent language , it is often discouraged to write platform specific stuff except for the rarest cases and when there is a clear business justification for the same.

Java does have an undocumented API that talks about signals where we can override the default signal handler for a given signal.  Scenarios where this might be useful and necessary are when we try to release resources ( connections etc.).

Note: When we override default signal handlers, it is important to delegate the behavior to the default handler implementation (Hint: Save the old handler when overriding the same) after we finish the current implementation.

package experiment;

import java.util.HashMap;
import java.util.Map;
import java.util.logging.Logger;

import sun.misc.Signal;
import sun.misc.SignalHandler;

public class CustomSignalHandler implements SignalHandler
{

  private static final Logger               LOGGER   = Logger.getLogger(CustomSignalHandler.class.getName());

  private static Map<Signal, SignalHandler> handlers = new HashMap<Signal, SignalHandler>();


  @Override
  public void handle(Signal signal)
  {
    LOGGER.info("received " + signal);

    // Delegate to the existing handler after handling necessary clean-up.
    handlers.get(signal).handle(signal);
  }


  /**
   * Important: This API is not portable but heavily platform dependent as the signal name depends
   * on the underying operating system. 

   * 
   * @param signalName
   * @param signalHandler
   */
  public static void delegateHandler(final String signalName,
                                     final SignalHandler signalHandler)
  {
    try { 
      Signal signal = new Signal(signalName);
      SignalHandler oldhandler = Signal.handle(signal, signalHandler);
    } finally { 
      handlers.put(signal, oldhandler);
    } 
  }


  public static void main(String[] args)
  {
    final int LONG_TIME = 50000;

    SignalHandler example = new CustomSignalHandler();
    delegateHandler("TERM", example);
    delegateHandler("INT", example);
    delegateHandler("ABRT", example);

    try
    {
      Thread.sleep(LONG_TIME);
    }
    catch (InterruptedException ie)
    {
      ie.printStackTrace();
    }
  }
}

Advanced Linux Programming

There is a new free e-book on advanced linux programming available here at - http://www.advancedlinuxprogramming.com/alp-folder  .

The chapter list is as follows.

• Chapter 01 - Advanced Unix Programming with Linux
• Chapter 02 - Writing Good GNU/Linux Software
• Chapter 03 - Processes
• Chapter 04 - Threads
• Chapter 05 - Interprocess Communication
• Chapter 06 - Mastering Linux
• Chapter 07 - The /proc File System
• Chapter 08 - Linux System Calls
• Chapter 09 - Inline Assembly Code
• Chapter 10 - Security
• Chapter 11 - A Sample GNU/Linux Application

Among the list of chapters - chapter 7 and chapter 8 to some extent are especially useful and well-written where there are some good hidden treasures even for a seasoned linux professional.

Segmentation Fault - but no core dump ?

When I was working on a given binary - it gave me a "Segmentation Fault" but no core dump.

Started with the usual suspects .
* Checked the permission of the directory to see if the user has write permission to write the core file. It was ok.
* Checked the /tmp directory just in case.

Then I realized that I was working on bash, that sets core file size to be 0 automatically.

$ ulimit -a
core file size          (blocks, -c)   0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20

..

Reset the same as below

$ ulimit -c unlimited

Now I ran my executable again and yes , this time the core dump is available for processing.

Linux AIO Examples

Tim Jones gives an excellent introduction and motivation behind aio here.  All 2.6 kernels have aio as a standard feature now.

Elastic Block Store

Amazon comes up with another *potential* low-margin - high volume business by announcing Elastic Block Store today.
The most interesting feature is of course the facility to provide block level storage. The S3 service is already extremely popular - thanks to getting away from the relational model (which sometimes could end up being an overkill ) and reducing the headache of IT management for a potential entrepreneur .

EC2 service, no doubt - is much better to create your own instance of an image from scratch and use it. In spite of having free REST requests to the S3 service , the absence of persistence as such on the image instance was a drawback.

EBS provides us with block-level storage volumes that could be attached to an EC2 instance. As opposed to the other tools that has a much stepper adoption curve ( s3 needs some sort of wrapper around REST - the popular being JetS3t ) - this is probably as simple as it could get and hence it might increase the adoption rate compared to the rest.

I have not got the time to compare the pricing of EBS against the rest, but my guess is that people probably would not mind paying up for this given the level of comfort it gives to making the EC2 instances more usable.

Graphite - Visualization Tool

Orbitz, the popular travel planning website, had recently brought some of their (previously proprietary) projects into the public domain by making them open source.

Graphite is a scalable, real-time graph visualization tool, released under the Apache License.

Some of the interesting aspects of the same (courtesy, the FAQ of the software):

• Written in Python, based on the Django project.
  
• The rendering engine is based on the Cairo framework, the same rendering engine used for the rendering of content in the Firefox 3 browser.
  
• The input data has to be a numeric time series. (This seems intuitive since graph visualization schemes, differences ought to be based on some quantitative measure eventually). And then, of course - any categorical metric could be mapped to preset numerical values to achieve a similar effect.

Graphite, seems to achieve the scalability by storing the entries in a distributed in-memory database, similar to what LiveJournal implements using the memcached service. And more recently, microsoft has started offering Velocity , a competing product in the same space (with subtle differences though- which I will cover later ).

Ct - Programming language for multi-core processor

With multi-core processors becoming the norm, the responsibility of exploiting the parallelism / improving the performance has increased on the software development rather than the hardware.

Intel has recently come out with a prototype implementation of Ct, a new programming language, for multi-core processors. As per the release notes, the learning curve of Ct is expected to be smoother, as the fundamental language construct seems to be based on the C/C++ programming language, in addition to the language specific features that enable the programmer to refer to parallelism.

A brief introduction to the language construct is available here .

With multi-core systems - it obviously makes more sense to extract / specify data-level parallelism ( + related instructions), as opposed to instruction level parallelism only, to get the best results. New constructs are available in the programming language to specify the same.

• Mention of a new Generic Vector Type (TVEC), that exist in the managed space. It is important to note that TVECs could be a flat vector or a multi-dimensional vector.


• Restricted operator overloading on TVEC objects, with the important restriction of allowing those with no side-effects.

As a proof of concept, the examples listed in the tutorial talk about the Black-Scholes option pricing model and the Convolution operator (widely applied in Computer Vision / Image processing applications).


I have not been able to confirm if the implementation + runtime is made available to the public yet. One of the components, The Threading building blocks, has been available as a open source project for sometime though.


This interesting release brings some interesting questions.

• The last C++ standard (C++03) was written for a single-threaded abstract machine , and threading as yet - is not part of the current C++ standard (current, as supported by the compilers). With fragmented threading libraries across platforms and implementations, portability had always been an issue with threading libraries on C++. But more recently, with Boost Threads providing a nice wrapper over the implementation-specific thread libraries - it is becoming less of a concern. And there is a very good chance that most of these primitives / APIs would be used in the upcoming C++0x standard as well. Given that, the standardization process of introducing thread support into the languages is a little bit late and C++ look-alikes specific to multi-core processors, pushed by the architecture vendor themselves, what would the first choice of technology developers to implement high frequency applications ?


• Functions with no-side effects, List Comprehension are all first class citizens, welcome in the Functional Programming world. More specifically, recently , I am fascinated with the Erlang Programming Language with native constructs supporting concurrency (no shared memory, thanks) and based on message passing. So - can the job of extracting better performance from multi-core processors be split between providing a robust interpreter / compiler for the functional programming languages and the functional programming language developer ?

We need to wait and see the way things take shape regarding the above mentioned scenarios.