Tuning Postgresql database server

Background

As you may know, I am Java/JEE developer by profession and part time stock trader(Not a DBA). I have been developing and maintaining, proprietary financial trading platform for past few years. As any other financial application, it does periodical historical updates (bulk inserts) and daily updates (insert/update) to database (OLTP). Also on daily basis, we have to run our algorithms to prepare reports(OLAP). Therefore, the most important component of this application is database and I/O.

Environment

Name Value
VPS Server Xen Based VPS with 2GB RAM
OS Ubuntu 10.04 LTS Server 32 bit Minimal
Database Postgresql 9.1

Tuning

As I have said earlier, I am not DBA by profession, following configurations are based on Gregory Smith’s,
PostgreSQL 9.0 High Performance
. I highly recommend this book and please read for more information. Before editing any of the configuration files,please make a backup of existing configuration. Also it is good idea to version control them in your repository.

/etc/sysctl.conf

Following are the configurational changes that I made to sysctl.conf,

##############################################################
# VM Parameters
#
#
#
##############################################################
vm.swappiness=0
vm.overcommit_memory=2
##############################################################
# Kernel Parameters
#
#
#
##############################################################
kernel.shmmax = 1081798656
kernel.shmall = 264111
kernel.sem=250 32000 64 512
##############################################################

##############################################################
#File Parameters
##############################################################
fs.file-max = 100000
##############################################################

kernel.shmmax and kernel.shmall is very important to increase wal-buffers size for postgres. Following bash script (copied from the book) will
help you to determine those parameters,

#!/bin/bash
page_size=`getconf PAGE_SIZE`
phys_pages=`getconf _PHYS_PAGES`
shmall=`expr $phys_pages / 2`
shmmax=`expr $shmall * $page_size`
echo kernel.shmmax = $shmmax
echo kernel.shmall = $shmall

/etc/…./postgresql.conf


wal_buffers = 64MB
checkpoint_segments = 10
checkpoint_completion_target = 0.9
effective_cache_size = 256MB

/etc/fstab

Change /dev/xxx / ext3 defaults,noatime 0 0

My Opinion on Latest Linux Desktops Ubuntu 11.10 Unity vs Fedora 15/Ubuntu Gnome 3 shell

My Frustration

I have been using extensively Ubuntu and Puppy for my development and day to day desktop activities for a while and I am one of the Linux Geeks who gives a damn about how my Desktop looks and do not like any changes in the way I use my desktop. As long as I have access to terminal,my Java IDE’s,video and audio players, I am a happy Linux desktop user. But as you may know, both Ubuntu and Fedora have decided to ditch Gnome2 and move forward with ambitious new desktop layouts similar to Windows/Mac, Fedora chose Gnome3 as its default interface and Ubuntu with its own interface Unity. I did not realize how it will affect me until I upgraded my OS from Ubuntu 10.04 LTS to 11.04. I was shocked with unity and how many clicks I have to make to just to get to terminal and I missed all my shortcuts in gnome panels. I was furious and did whine for long time. Six months back I uninstalled Ubuntu 11.04 and went back to Ubuntu 10.04 LTS.

My experience with Fedora 15/Ubuntu 11.10 and Gnome3

After reading and researching few Linux forums, realized that, whether I like it or not I have to let go Gnome2 and have to accept either Gnome3 or Ubuntu Unity. I decided to dump Ubuntu (obviously because of the way they decided to make those drastic changes) and try Fedora 15 with Gnome 3 for a while. For some reason, Fedora was way too clunky and enterprisey for me..Also I have been tinkering with Ubuntu file system for long time, I did struggle in Fedora to setup my softwares,media drives etc.. I moved back to Ubuntu 11.10 with Gnome3 on pretty soon …:) As you may know except the name Gnome 3 and Gnome 2 have nothing in common. They have revamped the GUI drastically. In my opinion, Gnome3 is really impressive and it makes me to think Am I in Windows world all over again sometimes :).. But at the moment Gnome3 is way too buggy for my liking… My Java IDE’s do not work as expected and in my opinion Gnome3 has a long, long way to go..

My choice

After one month of struggle,believe it or not, I am finally comfortable with Ubuntu 11.10 and its Unity Interface. Personally I think its quite mature right now and it works smoothly and faster. Also compiz and Docky works as expected :) (It was broken in Ubuntu 11.04 ).. I have to admit, it takes a while to get use to new menu futures and forget how we navigate in Gnome2,but I guess we will get used to it. I have managed to achieve what I do on daily basis with Ubuntu 11.10 Unity and docky , Below is my Dual Monitor Ubuntu desktop screenshot with Docky on,

Class Loader settings in GlassFish (like IBM Parent First/Last)

If you are developing your web applications in Glassfish and running your production code in other Application Servers (IBM in my case) ,you would like to provide your own version of
JSF,JPA ..etc by yourself than relying on container. Or you would provide your own libraries if you want to use latest JEE standards. The way we tell IBM class loader to load classes from application first than container is by setting Parent Last rather than Parent First. By default glassfish also looks for classes in the container than the application, if you want to override that, provide a glassfish-web.xml in your WEB-INF folder and add

Restart your application now, it will pick up the libraries from your WAR now.

Setting up Scala on your Linux Machine and IDE

In this article, we will setup Scala in linux machine and will compare three popular IDE’s which support Scala and provide you my opinion which one is best.

Installation

  • Install Java

    Scala runs on JVM, (Java Virtual Machine) so I am going to assume you have JRE/JDK setup in your machine already if not go ahead and install sudo apt-get or yum install open-jdk or sun-java-6 etc

  • Download Scala

    Download Scala and copy to /usr/local/share/scala (I got it in /home/xxx/Programs/scala)

  • Make some sym links in your /usr/local/bin

    cd /usr/local/bin
    sudo ln -s ../share/scala/bin/scala scala
    sudo ln -s ../share/scala/bin/fsc fsc
    sudo ln -s ../share/scala/bin/scalac scalac
  • Type scala and you should expect a console something like this,
    scala
    Welcome to Scala version 2.9.0.1 (OpenJDK Server VM, Java 1.6.0_22).
    Type in expressions to have them evaluated.
    Type :help for more information.

    scala> println("Hello World")
    Hello World

  • Another Example

    scala> val testList = List(1,2,3,4,5)
    list: List[Int] = List(1, 2, 3, 4, 5)
    scala> testList.foreach {i=> println(i)}
    1
    2
    3
    4
    5

If you could execute above examples, then you have installed Scala successfully in your machine.

IDE Comparision

I have written classic Inheritance Shape example in Scala ,


Shape

Circle Triangle Square

using three competitive Java IDE’s.

I ruled out Netbeans quickly just because there is not much documentation out there and also not much developers are involved in writing Scala plugin right now. Intellij and Eclipse both support Scala very well. Both Intellij Community/Ultimate edition supports scala, To install Scala in Intellij go to plugins section and search scala and install Scala. In Eclipse, go to the link above and get the update site url and install plugin in Eclipse using update site option. For me Eclipse is better than Intellij because, I was able to write Scala code in Eclipse as the way I used to write Java code in Eclipse. This is my opinion for now, it will change as my Scala codebase grows. In my next post, I will try to explain how we convert a bloated Java application to Scala application.

Using Method Chaining in POJO/DTO/ Entities.

Background

In recent years method chaining is becoming popular in Java/JEE world. Frameworks that I am aware of, that have used Method Chaining effectively are, Hibernate/JPA,EasyMock/JMock and jQuery. jQuery has revolutionized JavaScript world by using readable fluent interfaces. Method chaining has been around for years from Small Talk era. In C++,it is very common practice to use method chaining in getter/setter classes.
Let us see how we can apply method chaining to Java POJO classes.

Typical Java POJO Example

Let us consider following daily stock price example,

DailyPriceData.java

public class DailyPriceData {

private Date transDate;
private Double open;
private Double high;
private Double low;
private Double close;
private Double volume;

public DailyPriceData () {

}
public DailyPriceData (Date transDate,
Double open, Double close) {
this.transDate =new Date(transDate.getTime());
this.open = open;
this.close = close;
}
public Double getClose() {
return this.close;
}
public void setClose(Double close) {
this.close = close;
}
................./getters and setters
}

Above DailyPriceData class is typical way we generate POJO’s or the way our IDE’s will generate. The Builder/Converter class that will construct the above object has to use setter methods or a constructor has to provided. We certainly will not provide a bloated constructor with six arguments, the way we solve this issue generally is,we provide getters/setters and expect Builder/Converter class to construct object using getter/setters. If we want object to be immutable, then we
would break the object finer like Price(open,high,low,close)etc… and provide a constructor with finer nested objects. We will not discuss the later case here, and we will assume we
are providing getters and setters like our typical Java POJO way.

Converter/Builder

In Enterprise Java world, it is quite common to convert Persistence objects to DTOs or JAXB Beans etc.. Also it is quite common for us to write Builder classes
which construct/build objects based on certain attributes or from other objects. See the example below how we construct the object below with normal setters,

PriceBuilder.java

DailyPriceData priceData = new DailyPriceData ();
priceData.setClose(priceBean.getClose());
priceData.setHigh(priceBean.getHigh());
priceData.setLow(priceBean.getLow());
priceData.setVolume(priceBean.getVolume());

POJO using Method Chaining

public class DailyPriceData {
private Date transDate;
private Double open;
private Double high;
private Double low;
private Double close;
private Double volume;

public DailyPriceData () {

}
public Double getOpen() {
return open;
}
public DailyPrice setOpen(Double open) {
this.open = open;
return this;
}
public Double getHigh() {
return high;
}
public DailyPrice setHigh(Double high) {
this.high = high;
return this;
}
.......
}

As you can see the only difference between our classical Java POJO and the class above is ,it returns the object (DailyPrice in our case) that invokes the setter method. Now we will see how we construct
the object in our builder class,
PriceBarData priceData = new PriceBarData().setClose(priceBean.getClose())
.setHigh(priceBean.getHigh())
.setLow(priceBean.getLow().
setVolume(priceBean.getVolume());

One of the major advantage by using this method is ,try writing the above code in IDE’s like Intellij or Netbeans you would see how fast you would write the above code by tabbing :).. Also above technique will not disturb your Hibernate generators,or Dozer frameworks.

Fluent Interfaces

Martin Fowler wrote an article based on Eric Evans presentation and named Interfaces ‘that are primarly designed to be readable and to flow’ as fluent interfaces. Though, we use method chaining in the above method, it is not readable and flowable as we wish to describe as
price().openedAt().closedAt().surgedAt()...etc... We will definitely break our Hibernate or JAXB frameworks,if we are using any fluent interfaces. As he describes in his article, writing a fluent interface requires good bit of thought.

Finding jar for a class in IBM Websphere

Find Jar in Websphere

If you want to find a class file,Most of the time googling it or going to Maven Repo or Findjar will tell you where the jar is. But our friendly IBM Websphere will get you to its API :) not jar .Today, I ran into Websphere specific error,
java.lang.NoClassDefFoundError: com/ibm/ws/bootstrap/RASWsLoggerFactory
I have to do something like this below to find the class ‘RASWsLoggerFactory’ from websphere folder,

mambalav/cygdrive/c/IBM/Websphere/runtimes/base_v7/
$ find . -name '*.jar' -type f|xargs -i bash -c "jar tvf {}|grep 'RASWsLoggerFactory' && echo {}"

If you are a unix geek,you can tweak the above command and make it as a script and search any class from anywhere. But Trust me googling is faster in most of the cases :).
After a while, found out RASWsLoggerFactory is in IBM JDK ext folder and I was pointing to Sun JDK in my Netbeans IDE :).

platform1

Netbeans RCP Stock Trading Plaform

Background

As any other trader or sucker, I got attracted by Foreign Exchange/Stock market to make ‘Quick Money’ :). After loosing half of my initial account in FX, I learnt there is no such thing as ‘Quick Money’. After that huge loss, I took trading seriously, started learning Technical Analysis and Fundamental analysis. As part of my learning, I decided to develop /programmatically make trade setups based on my understanding which lead to this rich client platform development.

Why another Trading Platform

  • Though all Brokers provide fantastic platform, it is still hard to write your own algorithm and do whatever you want on your own.
  • A platform which helps to make risk/money management according to my needs. As I know my own strengths and weakness, it is easy to customize and build rules for my needs
  • Main reason in this helps me to improve my Programming skills as well as Trading skills (It took me a while to understand,mental skills and keeping thing simple is important :) )

Why Netbeans RCP

As I am a Java developer,I started developing my application with Java Swing based on JFreeChart (http://www.jfree.org/jfreechart/). After spending one month, I dropped this project realizing developing an interactive charting ‘Good Looking (:P)’ application is going to take years with JFree Chart and Java Swing. After a while, I stumbled across Netbeans based open source HumaiTrader (http://sourceforge.net/projects/humaitrader/). It is possibly the best free Trading platform available still now,but unfortunately this project is tightly coupled to its framework and also the project is dead. This lead to own trading platform by just using(ripping) HumaiTrader charting package. I decided to stick with netbeans rcp because,

  • Very Rich Application desktop framework(Rich Window ,Rich explorer,Toolbar,Quick Search, Progress Bar…)
  • Pure Java Swing Based Framework(not like Eclipse SWT )
  • Very active community

Screen Shots





Error Handling in Jersey/JAX-RS(REST)

In this article we will try to explain handling errors in your jersey(JAX-RS) restful web application using simple example. Let us consider simple StockQuoteRestService example,the rest service will fetch stock quotes for a given stock quote.

Stock Quote Rest Service Example

REST URL

Let us consider the following Rest URL ,
resources/equities/price/?exchange=NASDAQ&symbol=AAPL&fromDate=20050101&endDate=20110101
Above rest url will fetch stock quote based on following query parameters,

  • symbol
  • stock exchange
  • fromDate
  • endDate

If any of the above query parameters is wrong you will get something like this,
In worst case scenario, if you are being naughty and haven’t handled exceptions in service layer you will end up with our infamous NullPointer Exception.
Note:Always validate your query parameters and handle exceptions in your rest service layer before delegating your call to service layer. You rest application is prone to security attacks if you do not have strict validation.

Error Handling

The exception class to use for throwing your business exceptions is JAX-RS WebApplicationException.

Create your business exception class and extend this class and generate super constructor as required,

Exception class example


/**
*
* @author mohan
*/
public class SymbolNotSupportedException extends WebApplicationException {

public SymbolNotSupportedException(String message) {
super(Response.status(Status.NOT_FOUND).
entity(message).type(MediaType.TEXT_PLAIN).
build());
}
.......................
....................
}

Assuming you will have more specific business exceptions, we can push this constructor to AbstractGenericException class

/**
*
* @author mohan
*/
public abstract class AbstractGenericRestException extends WebApplicationException {

public AbstractGenericRestException(String message) {
super(Response.status(Status.NOT_FOUND).entity(message).type(MediaType.TEXT_PLAIN).build());
}
.......
}
public class SymbolNotSupportedException extends AbstractGenericRestException {

public SymbolNotSupportedException(String message) {
super(message);
}
......
}

Sample QuoteRestService Code Snippet


.................................
@GET
@Produces("application/xml")
public List getPriceBarListDateAsXML(@QueryParam symbol,@QueryParam("fromDate") String fromDate,
@QueryParam("endDate") String endDate)
{
if (!validator.validSymbol(symbol) {
throw new SymbolNotSupportedException("No data found.We currently do no support symbol :" + symbol);
}
........
}

Now if you try the following URL,
resources/equities/price/?exchange=NASDAQ&symbol=XXXXXXX&fromDate=20050101&endDate=20110101
you will get error as below,