DigitalJoel

2011/08/10

Backing up your database to Amazon S3

Filed under: development — Tags: , , — digitaljoel @ 10:17 pm

So now that you have your application running on an AWS EC2 instance, you need to backup the data somehow.  In my case, it’s a postgres database and I wanted to back it up into Amazon S3 within my same AWS account.  I wanted to have a backup for every day of the week, which would roll.  What I mean is that I would have a backup for Monday, and every Monday it would overwrite the previous Monday’s backup.  That way I would have a rolling 7 day backup. but not have a bazillion copies of the database that I have to manually get rid of.  Anyway, on to the code.

I wrote a little bash script that I then put into a cron job.  First, there’s a touch of setup to be done.  Wherever you are going to be running the job, you will need to install and configure s3cmd.  It’s a great little utility for hitting s3 from the command line.  The very simple instructions for configuring s3cmd are on that first page and shouldn’t take you more than 5 minutes.  I’ve run it on OSX Lion and also on my AWS instance and had no issues.

Next, is the bash script.  Here it is.

#!/bin/bash

PGDUMP=pg_dump
EXPORTFILE=`date +%A`.sql
COMPRESSEDFILE=${EXPORTFILE}.tgz
BUCKET=<your bucket name>
S3CMD=~/bin/s3cmd-1.0.1/s3cmd

$PGDUMP -f ./$EXPORTFILE -cb -F p --inserts -U <your user> <your database>
tar -czf ${COMPRESSEDFILE} ${EXPORTFILE}

$S3CMD put ./${COMPRESSEDFILE} s3://${BUCKET}

You’ll need to set PGDUMP to point to your pg_dump script if it isn’t in your path. Also set S3CMD to point to wherever you installed s3cmd.  If you prefer other options for pg_dump, or if you are using some other database, you can modify the $PGDUMP line to do whatever you need.

On Monday the script will create a file named Monday.sql and a compressed archive named Monday.sql.tgz.  It’ll then upload Monday.sql.tgz to your s3 account.  You could easily add another line at the end of this script to remove the exported file and the archive using


rm $EXPORTFILE $COMPRESSEDFILE

Finally, you’ll need to schedule this to be run once per day.  This can be done by running crontab -e and then using the following line in the crontab file:


0 2 * * * ~/backupdb.sh

That will run the script every morning at 2.  You can change the hour for whatever fits your needs.

The final task for me is going to be creating a similar script that will run every week and keep the last 4 weeks of backup.  I’m planning to do that using %W on the date command to get the week of year and do some math using the week number in the file name to create the new file and remove the old ones.  I guess I’ll leave that as an exercise for the reader.

Advertisements

2011/08/04

Amazon Web Services Alarm for HTTP Server

Filed under: development — Tags: , , , — digitaljoel @ 10:53 pm

So you’ve written an app and you’re hosting it on an AWS EC2 instance.  For whatever reason you have only the one server up with no load balancer in front of it.  You want to set an alarm in AWS so that if the server goes down you’ll know right away, but how can you do it?

I wrote a simple bash script to ping a special URL in my web application.  The response from the URL is simply the text “healthcheck ok” with a 200 response code.  The script checks for that text.  If it exists in the response, then it sends a 1 up to AWS as a custom metric.  If it doesn’t, then it sends a 0.

#!/bin/bash
while :
do
  stat=0
  healthcheck=`curl --connect-timeout 5 --max-time 7 --fail --insecure --silent https://localhost/healthcheck`
  if [ "healthcheck ok" = "$healthcheck" ]
  then
    stat=1
  fi
  mon-put-data --metric-name HttpHealthCheck --namespace YourNamespace --dimensions "server=prod" --value $stat
  sleep 60
done

In order for the script to run, you’ll need to have done all the authentication setup for the AWS scripts and ensure you have a version of them that includes the mon-put-data script.  For testing, you can run the curl command on the command line.  You can do the same with mon-put-data.

In my experience, it took a few minutes for the custom metric to show up the first time I sent it.  Once it settles in you should be able to select it from the metrics in CloudWatch.  The final step is to setup the alarm.

You should be able to set an alarm to go off when the value of the metric is <= 0.  I tested it by shutting down my web server and I got the alarm notification within about a minute.

If your health check isn’t started (which you can do with $nohup ./healthcheck.sh &) then you won’t get samples and in my test no alarm was sounded.  So, I set another alarm.  For any metric, you can set an alarm based on the value, or based on the samples.  Just choose the “samples” statistic from the drop down.  Set the alarm to go off if samples <= 0.  Also add another action and set it to go off on INSUFFICIENT_DATA, meaning that there are not enough samples, which likely means your script wasn’t started, or has failed.

Once your app is super popular, you can look at the load balancer, which I believe allows for setting alarms based on HTTP response times etc. but I think this’ll do until I get there.

2011/06/15

Spring ConverterFactory Implementation

Filed under: java, spring — Tags: , , , — digitaljoel @ 10:12 pm

In my Spring MVC 3 based application I had recently implemented a few Converters for some of my JPA based data objects. It started with one, then another, and so on. By the time I got around to adding my fourth converter to the spring configuration file I knew it was time to pull it out and abstract it a bit. Thankfully, Spring allows you to implement a ConverterFactory that is responsible for creating the converters for some types.

Each of my entities extend an abstract base class that looks basically like this

@MappedSuperclass
public abstract class DataObjectAbstract<K extends Serializable>
        implements DataObject<K>
{
    protected transient String[] excludedEqualsFields = new String[] { "key", "version" };

    @Version
    protected int version;

    @Override
    public boolean equals( Object that )
    {
        return EqualsBuilder.reflectionEquals( this, that, excludedEqualsFields );
    }

    @Override
    public int hashCode()
    {
        return HashCodeBuilder.reflectionHashCode( this, excludedEqualsFields );
    }

    @Override
    public String toString()
    {
        return ToStringBuilder.reflectionToString( this, ToStringStyle.MULTI_LINE_STYLE );
    }
}

The DataObject interface simply declares a getKey and setKey method.

So, in my Spring MVC Controller methods I was originally accepting a String or Long, then using my own data access objects to lookup the entities I needed. The next iteration in my implementation was to implement the Converters as I mentioned above. That was very simple and worked well, but having many data objects I didn’t want to copy that implementation over and over again. This is where the ConverterFactory comes in. Here’s my implementation:

@Component
public class DataObjectConverterFactory
        implements ConverterFactory<String, DataObject<Long>>
{
    @PersistenceContext
    EntityManager em;

    @Override
    public <T extends DataObject<Long>> Converter<String, T> getConverter( Class<T> type )
    {
        return new GenericLongKeyedDataObjectConverter<T>( type, em );
    }
}

The ConverterFactory interface is basically as simple as the Converter interface. The Class<T> type parameter to the getConverter method tells us what type we are going to convert to.  One option from here is to have a big nasty if/else statement with a bunch of instanceof methods that create a new Converter.  I thought about doing this and passing in the appropriate data access object and performing the lookup.  That would be only two classes and then I could convert all of my DataObjects, but I didn’t like the idea of a bajillion instanceof statements.  So you can see I implemented a GenericLongKeyedDataObjectConverter which takes the target type and the EntityManager as a parameters.  Here’s the implementation of the generic converter class:

/**
 * A generic converter used for converting from a string representation of an entity key to the DataObject itself.
 *
 * @param <T> The type that is to be converted to.
 */
public class GenericLongKeyedDataObjectConverter<T extends DataObject<Long>>
        implements Converter<String, T>
{
    private Class<T> type;
    private EntityManager em;

    /**
     *
     * @param type An instance of Class for the type being converted to
     * @param em EntityManager used to perform the lookup.
     */
    public GenericLongKeyedDataObjectConverter( Class<T> type, EntityManager em )
    {
        this.type = type;
        this.em = em;
    }

    @Override
    public T convert( String stringKey )
    {
        Long key = Long.parseLong( stringKey );
        return em.find( type, key );
    }
}

An extremely simple parameterized class implementation of the Converter interface. Here, with no use of instanceof, I’m creating the appropriate converter implementation for all of my persisted classes.  If you have a group of objects that you want converted and they all inherit from a base class, a ConverterFactory may be a better solution than implementing a bunch of converters manually.

Finally, here’s the bean xml configuration:

<bean id="conversionService" class="org.springframework.format.support.FormattingConversionServiceFactoryBean">
    <property name="converters">
        <list>
            <ref bean="dataObjectConverterFactory" />
        </list>
    </property>
</bean>

Notice that we reference the dataObjectConverterFactory bean, but I never defined it in my xml config.  That’s because I used the @Component annotation on my implementation class.

2011/04/20

Removing A Dragged And Dropped List Item

Filed under: development, jquery — Tags: , — digitaljoel @ 11:11 pm

In a previous post I mentioned how to take a table row and drag it onto a sortable list. The problem with that is that there was no way to remove the item once it was dropped on the list. So, I modified the original code so that the dropped item now has a button that allows for removal of the item. Here is the new version in its entirety:

        var qTable;
        var newSurvey;
        // create the fancy datatable
        $(function() {
            // setup the datatable
            qTable = $('#questionTable').dataTable( {
                    "aoColumns": [
                                  { "asSorting": [ "desc", "asc" ] },
                                  { "asSorting": [ "desc", "asc", "asc" ] },
                              ]
                    , "bJQueryUI": true
                }
            );
            
            $(qTable.fnGetNodes()).draggable({
                opacity: 0.7,
                helper: function() {
                    var text = this.children[0].innerText;
                    var result = "<li id='"+this.id+"'>"+text+"</li>";
                    return result;
                },
                connectToSortable: '#newSurvey'
            });

            newSurvey = $('#newSurvey');
            newSurvey.sortable({
                beforeStop: function( event, ui ) {
                    var id = ui.helper.attr( "id" );
                    if ( id.indexOf( 'li' ) == -1 ) {
                        id = 'li' + id;
                    }
                    var text = ui.helper.text();
                    var li = "<li id='"+id+"'><span class='ui-icon ui-icon-circle-close' onclick='remove(\""
                            +id+"\")'></span>"+text+"</li>";
                    $(ui.item).replaceWith( li );
                },
            }).disableSelection();
        });
        
        function remove(id)
        {
            var li = $('#'+id);
            li.fadeOut('fast', function() { li.remove();});
        }

So, the biggest difference between this and the previous is in the “beforeStop” function. The first being this block:

                    if ( id.indexOf( 'li' ) == -1 ) {
                        id = 'li' + id;
                    }

The problem I had was that when dropping from the table row, everything was awesome, but if I re-ordered within the list, then I kept pre-pending another ‘li’ to the front of the id. So I would end up with a row with an id of ‘lilili123’ or something like that. Undesirable at best. So now, I check to ensure it only has one li prefix.

The second difference, and the main one for this post, is the addition of the remove function and the button to remove it when dropped, contained here:

                    var li = "<li id='"+id+"'><span class='ui-icon ui-icon-circle-close' onclick='remove(\""
                            +id+"\")'></span>"+text+"</li>";
                    $(ui.item).replaceWith( li );

and here:

        function remove(id)
        {
            var li = $('#'+id);
            li.fadeOut('fast', function() { li.remove();});
        }

The first section is the new code to replace the helper from the table row draggable with the list item, including the button for removal. If that sentence didn’t make sense, then go back and read the post linked above to get the details. Since this is all based on jquery, I used a jquery icon for the button. It is nice because then it will mesh with whatever jquery theme you are using.

The remove function uses a jquery animation to quickly fade the list item out and then remove it from the list. You must call .remove() to get it out of the list altogether.

The last wrinkle I have in this is that a user can drag the same item from the table onto the list, resulting in multiple copies of the list item, but that’s a problem for another day.

2011/04/05

Get the “Next” value in a Java Enum

Filed under: development, java — Tags: — digitaljoel @ 10:32 pm

Java Enums. An awesome addition to Java 1.5 so we could avoid using public static ints for that purpose. I’ve been using them for some time with success and never noticed one deficiency until now. You can get the ordinal of an enum value with the ordinal() method. That is basically the index in the order the values were declared. So, if your enum look something like this:

 public enum Planet { MERCURY, VENUS, EARTH, MARS, JUPITER, SATURN, URANUS, NEPTUNE }

In this case, MERCURY would have an ordinal of 0, then VENUS 1, and so forth. Now, what if you want to iterate through them? You can get all the values of an enumerated type as an array using the values() method. Cool, right? Well, what if I don’t want to iterate through them, but I want to simply progress from one to the next. It would be cool if math operators (like + and -) would let you go from one to the next, but that’s not the case. I had a need to go from one to the next, so I changed my enum by adding the following method.

 private enum Planet { MERCURY, VENUS, EARTH, MARS, JUPITER, SATURN, URANUS, NEPTUNE;
    public Planet getNext() {
      return this.ordinal() < Planet.values().length - 1
          ? Planet.values()[this.ordinal() + 1]
          : null;
    }
  }

Now, if I do Planet.MERCURY.getNext() I would get VENUS. This takes advantage of the ordinal of each entry (which you cannot assign in any other way than the order in which you declare the enum values) and the values method, indexing into the values array to get the next value. If you attempt to go off the end, it’ll return null. It would be simple to make it wrap instead if that makes sense for your case. It would also be trivial to take this and implement a “getPrevious” if you have a need to go in reverse.

2011/03/31

Erlang Concurrency Demo Application

Filed under: development — Tags: , — digitaljoel @ 9:37 pm

As I said in my last post I was responsible for a demo in Erlang today. Before starting this application, I had about 2 hours of experience programming in Erlang. Day 3 in Bruce Tate’s book, Seven Languages in Seven Weeks focused on the concurrency aspects of Erlang. I was looking for an example program I could write that would demonstrate this, and didn’t want to just copy the example from the book. I asked on StackOverflow.com and didn’t really get anything I wanted to tackle.

Finally, last night while on my way to Kenpo I was able to think up this cool application. One process would be the air traffic control tower. Then, other processes would be all of the airplanes. Here’s the whole program

-module(aircontrol).
-export([run/0, fastrun/0, tower/0, createPlane/5, removePlane/2, addPlane/3]).

% method to run the program, creating a tower and some planes.
% in this run, 4 of the 5 planes crash
run() ->
  Tower = tower(),
  createPlane(Tower, 1, 5, 5, 2000),
  createPlane(Tower, 2, 1, 1, 4000),
  createPlane(Tower, 3, 1, 5, 6000),
  createPlane(Tower, 4, 5, 3, 10000),
  createPlane(Tower, 5, 5, 1, 5000).

% all planes survive here
fastrun() ->
  Tower = tower(),
  createPlane(Tower, 1, 5, 5, 30),
  createPlane(Tower, 2, 1, 1, 10),
  createPlane(Tower, 3, 1, 5, 30),
  createPlane(Tower, 4, 5, 3, 10),
  createPlane(Tower, 5, 5, 1, 10).

 

%%%%%%%%%%%%% Tower methods %%%%%%%%%%%%
tower() -> 
  io:format( "Tower ready for duty~n" ),
  spawn( fun() -> tower(lists:duplicate(25, null ))end).

tower(Occupied) ->
  receive
    {arrive, {Plane, Id, 3, 3 }} ->
      io:format("Tower: Flight ~p, welcome home.~n", [Id] ),
      tower(removePlane(Plane, Occupied));

    {arrive, {Plane, Id, X, Y }} ->
      {NewX, NewY} = getBearing( X, Y),
      io:format( "Tower: Flight ~p, please head to bearing (~p, ~p)~n", [Id, NewX, NewY]),
      Plane ! {bearing, getBearing( X, Y )},
      tower( addPlane( Plane, ((Y-1)*5)+X, removePlane( Plane, Occupied )));

    _ -> io:format( "Tower: I don't understand. ~n" )
end.

removePlane(Plane, List) -> lists:map(
  fun(X) -> 
    if 
      X == Plane -> null; 
      true -> X
    end
  end
, List).

addPlane(Plane, Index, List) -> 
  Occupant = lists:nth( Index, List ),
  if
    Occupant /= null ->
      Occupant ! crash,
      Plane ! crash,
      removePlane(Occupant, List );
    true ->
      lists:append([lists:sublist(List, Index-1), [Plane], lists:sublist(List, Index+1, 25)])
  end.

% get the next bearing as an X,Y tuple
getBearing(X, Y) ->
  if
    Y /= 3 -> {X, getBearing(Y)};
    true -> {getBearing(X), Y}
  end.

% Get a component of the next bearing
getBearing(Num) ->
  case Num of
    1 -> 2;
    2 -> 3;
    3 -> 3;
    4 -> 3;
    5 -> 4
  end.


%%%%%%%% Plane methods %%%%%%%%

createPlane(Tower, Id, X, Y, Speed) ->
  Plane = spawn(fun() -> plane(Tower, Id, Speed) end ),
  Plane ! {new, {X, Y}}.

plane(Tower, Id, Speed) ->
  receive
    {new, {X, Y}} ->
      io:format("Flight ~p created at (~p, ~p)~n", [Id, X, Y]),
      Tower ! {arrive, {self(), Id, X, Y }},
      plane(Tower, Id, Speed);
  
    {bearing, {X, Y}} ->
      io:format("Roger that tower, Flight ~p heading to bearing (~p, ~p)~n", [Id, X, Y]),
      timer:sleep(Speed),
      Tower ! {arrive, {self(), Id, X, Y}},
      plane(Tower, Id, Speed);

    crash ->
      io:format("MAYDAY! MAYDAY!, Flight ~p is going down!~n", [Id])
end.

Alright, what is going on here. You have a playing field that is a 5×5 grid. Each plane will be in one section of the grid at a time, so a plane that is at location (1,1) is in the top left corner, and a plane at (5,5) is in the bottom right corner. The airport (and the control tower) is at (3,3) and if a plane gets to the airport then it lands and is not tracked anymore.

Each plane has a flight number, coordinates, and a speed. The speed may be counter intuitive because the lower the number, the faster the plane flies. A plane with a speed of 2000 means that it takes 2000 milliseconds (or 2 seconds) to go from one square to the next. If two planes are in the same square in the playing field, then they collide and crash. Obviously we don’t want this to happen, but whoever is running the tower isn’t very good at their job.

When a plane arrives at a new square it checks in with the tower. The tower then tells it which square to go to next. The plane travels at its speed to the next square, checking in with the tower when it arrives, and getting new coordinates from the tower, and so on, until the plane arrives at the airport.

So, let’s step through and see where in the code this is all happening.

run() ->
  Tower = tower(),
  createPlane(Tower, 1, 5, 5, 2000),
  % create more planes here.

The run method takes care of creating the tower, and the planes, which sets everything in motion. When we call tower(), here’s what happens


tower() -> 
  % print a method to the user telling them that the tower is starting up.
  io:format( "Tower ready for duty~n" ),

  % spawn the tower process.  This is a little different than the spawn code in the book.  In the book none of the functions
  % used to spawn a process took any parameters.  Here we need to pass a list that says which places on the grid are 
  % currently occupied by planes.  I seed that list with 25 null values.  25 for the 5x5 grid, using an offset scheme within
  % the list to represent the different rows in the grid.
  spawn( fun() -> tower(lists:duplicate(25, null ))end).

After spawning the tower, it goes into its receive loop. The receive loop accepts one type of message which contains the atom
receive and a tuple containing the Plane process, the flight number, and the (X,Y) coordinates that the plane arrived at. There are two cases I handle here, the first being when the plane has arrived at the airport safely, and the second being where the tower needs to tell the plane where to go to next.

tower(Occupied) ->
  receive
    {arrive, {Plane, Id, 3, 3 }} ->
      % in this case, the plane has arrived at the airport
      io:format("Tower: Flight ~p, welcome home.~n", [Id] ),

      % since the plane has landed, remove it from the list of planes that are in the air.
      % and recursively call ourselves with the new list so we can receive messages from more planes.
      tower(removePlane(Plane, Occupied));

    {arrive, {Plane, Id, X, Y }} ->
      % in this case the plane has arrived at some square and needs directions on where to go next.

      % Here we get the new bearings for the flight
      {NewX, NewY} = getBearing( X, Y),
      io:format( "Tower: Flight ~p, please head to bearing (~p, ~p)~n", [Id, NewX, NewY]),

      % and here we send the bearings to the plane.
      Plane ! {bearing, getBearing( X, Y )},

      % next we change the location of the plane in the grid.
      tower( addPlane( Plane, ((Y-1)*5)+X, removePlane( Plane, Occupied )));

    _ -> io:format( "Tower: I don't understand. ~n" )
end.

I’m not going to go into the guts of how we figure out where the plane should go. The tower will always move a plane on the Y axis until it gets to 3, then it moves it along the X axis toward the airport. This isn’t the greatest algorithm in order to avoid a collision, but it is what it is.

The code where we figure out if we have a crash is in the addPlane method, which is as follows:

addPlane(Plane, Index, List) -> 
  % get the current occupant of the target space
  Occupant = lists:nth( Index, List ),

  if
    Occupant /= null ->
      % if the occupant isn't null, then we have a collision.
      % send a message to each of the planes in the space letting them know there's a problem.
      Occupant ! crash,
      Plane ! crash,

      % because Plane is not in the list, we only need to remove the Occupant.
      removePlane(Occupant, List );

    true ->
      % otherwise, Occupant is null, so we can just add this plane to the list.
      lists:append([lists:sublist(List, Index-1), [Plane], lists:sublist(List, Index+1, 25)])
  end.

Now, on to the code for the airplanes. First, the code to create a new plane. This is a convenience method since we could obviously spawn them and send the message ourselves


createPlane(Tower, Id, X, Y, Speed) ->
  % spawn the process, passing in the tower, flight id, and speed
  Plane = spawn(fun() -> plane(Tower, Id, Speed) end ),

  % then we send a message to the process, letting it know its starting location
  Plane ! {new, {X, Y}}.

Next, we need a receive loop for the plane so it can get the messages back from the tower. Receive accepts three messages, either the atom new with coordinates, the atom bearing with coordinates, or the atom crash. Perhaps the first two could be combined into a single match, but I implemented them separate. Here is the implementation.

plane(Tower, Id, Speed) ->
  receive
    {new, {X, Y}} ->
      io:format("Flight ~p created at (~p, ~p)~n", [Id, X, Y]),

      % we need to get on the radar with the tower, so tell them where we are so they can tell us where to go next
      Tower ! {arrive, {self(), Id, X, Y }},

      % now recurse into our receive loop again
      plane(Tower, Id, Speed);
  
    {bearing, {X, Y}} ->
      io:format("Roger that tower, Flight ~p heading to bearing (~p, ~p)~n", [Id, X, Y]),

      % the second receive pattern sleeps for Speed ms to simulate the time it takes to fly from
      % one square to the next
      timer:sleep(Speed),

      % once we arrive, we need to tell the tower where we are, then recurse into the loop again to accept more messages.
      Tower ! {arrive, {self(), Id, X, Y}},
      plane(Tower, Id, Speed);

    crash ->
      % if we crash, we don't recurse back into the receive loop because our process is done.
      io:format("MAYDAY! MAYDAY!, Flight ~p is going down!~n", [Id])
end.

So, there you go. A sample concurrency implementation written in erlang made up completely in my head. I was very happy with how this turned out, and it only took me a couple hours to program it. Yeah, the meat of the implementation (without the run methods) is only about 80 lines, but it does a lot in those lines, and I think it’s something coming from an entrenched OO programmer.

Project Euler problem 22 in Erlang

Filed under: development — Tags: , — digitaljoel @ 7:24 pm

At work I’m in a reading group covering Bruce Tate’s Seven Languages in Seven Weeks. Each week we get together for an hour to discuss the current language. One person or duo is responsible for giving a presentation on the current language. This week was Erlang, and I was partly responsible for the presentation. After reading the chapter, I was trying to find a small program I could write in order to demonstrate some of the concepts discussed. I settled on problem 22 of Project Euler.

The problem is as follows:

Using names.txt (right click and ‘Save Link/Target As…’), a 46K text file containing over five-thousand first names, begin by sorting it into alphabetical order. Then working out the alphabetical value for each name, multiply this value by its alphabetical position in the list to obtain a name score.

For example, when the list is sorted into alphabetical order, COLIN, which is worth 3 + 15 + 12 + 9 + 14 = 53, is the 938th name in the list. So, COLIN would obtain a score of 938 53 = 49714.

What is the total of all the name scores in the file?

This is the very first problem I tried to solve with Erlang. Fortunately, the documentation on the language is pretty good, and the standard libraries are nice too. When I started programming in 1993 in college, my first language was Pascal, but I haven’t done much in a functional language since then, and working in Java exclusively for my occupation since 2000 hasn’t really kept me in the functional frame of mind. Anyway, to the code.

-module(euler).
-export([prob22/0]).

% http://projecteuler.net/index.php?section=problems&id=22
% problem 22 in project euler.
% Load the names in names.txt, then sort them alphabetically
% Then working out the alphabetical value for each name, multiply this value 
% by its alphabetical position in the list to obtain a name score.
% Then find the total of all name scores in the file.
%
% I massaged input by putting all names on their own line and remove quote characters
% before running this program.

prob22() -> 
    UnsortedNameList = readlines("names.txt"),
    SortedNameList = lists:sort(UnsortedNameList),
    % convert the Names to the integers representing the characters.
    % but don't accept the \n at the end of each name.
    NumberNameList = lists:map(fun(Name) -> [X-64 || X <- Name, X>64] end, SortedNameList),
    % NumberNameList contains all the names adjusted so A=1, B=2, etc.
    addNameValues( NumberNameList, 1).

% add the name values in the list, as specified in the requirements of problem 22.
addNameValues( [], _ ) -> 0;
addNameValues( [Head|Tail], Index ) -> 
    HeadValue = lists:foldl(fun(X, Sum) -> X + Sum end, 0, Head),
    HeadValue*Index + addNameValues(Tail, Index+1).
    

% file reading blatantly borrowed from http://www.trapexit.org/Read_File_to_List
readlines(FileName) ->
    {ok, Device} = file:open(FileName, [read]),
    get_all_lines(Device, []).

get_all_lines(Device, Accum) ->
    case io:get_line(Device, "") of
        eof  -> file:close(Device), lists:reverse(Accum);
        Line -> get_all_lines(Device, [Line|Accum])
    end.

Not any big rocket science. Note that I modified the source file by putting each name on its own line and removing the quotes around each name, so maybe that’s cheating, but I only had a little while to work on this, so I didn’t want to spend the time really looking into getting around that.

In the book, each of the functions were quite terse with their variable names and most were a single line. I didn’t feel like that did the reader any favors, but just for fun I thought I would try to condense it a bit. Here’s what I came up with

-module(eulerSmall).
-export([prob22/0]).

prob22() -> add( lists:map(fun(N) -> [X-64 || X <- N, X>64] end, lists:sort(readlines("names.txt"))), 1).

% add the name values in the list, as specified in the requirements of problem 22.
add( [], _ ) -> 0;
add( [H|T], I ) -> I * lists:foldl(fun(X, Sum) -> X + Sum end, 0, H) + add(T, I+1).

% file reading blatantly borrowed from http://www.trapexit.org/Read_File_to_List
readlines(FileName) ->
    {ok, Device} = file:open(FileName, [read]),
    get_all_lines(Device, []).

get_all_lines(Device, Accum) ->
    case io:get_line(Device, "") of
        eof  -> file:close(Device), lists:reverse(Accum);
        Line -> get_all_lines(Device, [Line|Accum])
    end.

Notice that by combining all of the list manipulation together, the prob22() function now takes a single line. That’s kind of fun.

2011/03/17

How to create a custom taglib containing an EL function for JSP

Filed under: java — Tags: , — digitaljoel @ 10:49 pm

At some point in your use of JSP, there’s something you’re going to need to do for which you can’t find a spring or jstl tag. In that case, you can create a custom function in your custom tag library. It sounds more difficult than it is. All you will need is a tag library descriptor, and your class that implements the function. That’s about it. Here’s my TLD file.

<taglib xmlns="http://java.sun.com/xml/ns/j2ee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee/web-jsptaglibrary_2_0.xsd" 
    version="2.0">

    <tlib-version>2.0</tlib-version>
    <uri>http://www.your-domain.com/taglib</uri>

    <function>
        <name>doMyStuff</name>
        <function-class>com.mydomain.util.ElFunctions</function-class>
        <function-signature>java.lang.String doMyStuff( java.util.Collection )</function-signature>
    </function>
</taglib>

This file should be placed in your WEB-INF directory. In the function-signature, be sure to use fully qualified names.

Next is the class that implements the function.

package com.mydomain.util.ElFunctions;

import java.util.Collection;

/**
 * Functions for use in expression language in the jsp views.
 */
public class ElFunctions
{

    /**
     * This is the function that is called by the Expression Language processor.  It must be static.
     * @param myparam
     * @return
     */
    public static String doMyStuff( Collection<SomeType> myparam )
    {
        // do stuff here and return the results
    }
}

Finally, just reference the function in my jsp file.

<%-- where you declare your taglibs, include this one, which references the tld we created in the first step. --%>

<%@ taglib prefix="my" uri="/WEB-INF/my.tld" %>

<!-- more html and whatever, in my case I'm using spring:message to output the results of my method call -->

<spring:message text="${my:doMyStuff(bean.collection)}" />

The call to ${my:doMytuff(bean.collection)} causes the EL processor to call my function when it evaluates that snippet. In this case, ‘bean’ would be some java bean available to the view, and ‘collection’ would be a property on the bean that returns the collection expected as input to doMyStuff.

2011/02/23

jQuery Dragging a Table Row, Dropping a List Item

Filed under: jquery — Tags: , , , — digitaljoel @ 11:14 pm

I have a collection of survey questions. I’m using the jQuery datatables plugin to show those questions in a way that is sortable and filterable. The datatables plugin is awesome for this. You can look at the reference for the datatables plugin for more information.

Next, I wanted to be able to drag the question as represented in a table row into a list. The list needs to be orderable, because the order of the questions in the survey is important. This leads me to the conclusion that I need to use the jQuery Sortable functionality.

Here’s my problem. If I drag the table row and drop it on the ordered list, I end up with a list full of table rows. That’s not great. The key is in the reception of the dropped element in the sortable list. Let’s get to some source.

            // variables to hold the table and the list.
            var qTable;
            var newSurvey;

            // create the fancy datatable
            $(function() {
                // setup the datatable according to the docs at datatables.net
                // my table only has two columns, with the question text being in the first column.
                qTable = $('#questionTable').dataTable( {
                        "aoColumns": [
                                      { "asSorting": [ "desc", "asc" ] },
                                      { "asSorting": [ "desc", "asc", "asc" ] },
                                  ]
                        , "bJQueryUI": true
                    }
                );
                
                // Get all the table rows and make them draggable
                $(qTable.fnGetNodes()).draggable({
                    opacity: 0.7,
                    helper: function() {
                        var text = this.children[0].innerText;
                        var result = "<li id='"+this.id+"'>"+text+"</li>";
                        return result;
                    },
                    connectToSortable: '#newSurvey'
                });

                // setup the sortable ordered list
                newSurvey = $('#newSurvey');
                newSurvey.sortable({
                    beforeStop: function( event, ui ) {
                        var id = ui.helper.attr( "id" );
                        var text = ui.helper.text();
                        var li = "<li id='"+id+"'>"+text+"</li>";
                        $(ui.item).replaceWith( li );
                    }
                }).disableSelection();
                
            });

There are a couple of subtleties here that you probably would miss if I didn’t point them out to you.

First, this line:

$(qTable.fnGetNodes()).draggable({

Looks simple enough. This line adds the draggable functionality to all rows of the datatable. This was a key. If you use another selector, something like “#tableId tr” which you would think would work, you’ll be in trouble. That’s because if you change the data that is viewed in the table, i.e. by filtering, then the draggable functionality will be lost on the new rows that are shown. So, in order to apply it to ALL rows of the table, you must call the function supplied by the datatables plugin.

Next, the helper function in the draggable setup.

    helper: function() {
        var text = this.children[0].innerText;
        var result = "<li id='"+this.id+"'>"+text+"</li>";
        return result;
    },

The helper is what is going to be displayed to the user when they are dragging your table row. I didn’t really care about showing the second column, so here I convert it to an li. This conversion isn’t absolutely necessary, but I left it in for kicks. If you decide not to write your own helper, you can use a built in “clone” helper. This will leave the row in the table and clone it for what is dragged rather than removing it from the table when you drag it out.

Finally, the beforeStop function of the Sortable.

    beforeStop: function( event, ui ) {
        var id = ui.helper.attr( "id" );
        var text = ui.helper.text();
        var li = "<li id='"+id+"'>"+text+"</li>";
        $(ui.item).replaceWith( li );
    }

While we did the transformation in the helper of the draggable, the helper isn’t what is dropped in the sortable, the actual item is, which in our case, is still a tr. So, once again, I do the transformation. Since it’s easier to get the information out of the helper than out of the table row again, I used that here. It seems like I ought to be able to just get the text of the li in the helper, but I didn’t pursue that much further. Also, if you attempt to change the item to an LI in the draggable definition, then you may end up with LIs in your table. It could get ugly.

So, there you have it. You are no longer dropping TRs in your ULs. Your tables have TRs and your ULs have LIs.

One caveat. I attempted to upgrade from 1.4.4 to 1.5 of jquery core tonight and found that dropping into the list was broken if I dragged the item out of the top of the drop zone.

2011/02/17

Java Equals Implementation Performance

Filed under: java — Tags: , — digitaljoel @ 4:13 pm

I had some questions about various implementations of the equals (and by association hashcode) methods in Java. I recently implemented a solution in a project I’m working on that uses Apache’s EqualsBuilder in order to create a simple, elegant looking implementation. Knowing that the solution used reflection, and that equals may be called a lot more than you would think. So, I implemented a little test today in order to see the performance of various implementations. The contenders are the equals method generated by the Eclipse IDE, Apache Commons EqualsBuilder implementation using append, and also using reflection, and finally Pojomatic.

Here’s the source of the test

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

import org.apache.commons.lang.builder.EqualsBuilder;
import org.apache.commons.lang.builder.HashCodeBuilder;
import org.pojomatic.Pojomatic;
import org.pojomatic.annotations.AutoProperty;

public class EqualsTest {

  // how many objects we want to create
  static long count = 10000;
  // seed for our random number generator so all tests get the same variation
  static long seed = 12345;
  // range of variation in the values for our objects
  static int variation = 15;
  // how many times equals was called from the eclipse method
  static long plainCounter = 0;
  // how many times equals was called from the reflection method
  static long reflectCounter = 0;
  // how many times equals was called from the append method
  static long appendCounter = 0;
  // how many times equals was called from the pojomatic method
  static long pojomaticCounter = 0;
  // how many times equals was called from the broken method.
  static long brokenCounter = 0;

  public static void main(String[] args) {
    EqualsTest test = new EqualsTest();
    System.gc();
    test.testPlainEquals();
    System.gc();
    test.testAppendEquals();
    System.gc();
    test.testReflectEquals();
    System.gc();
    test.testPojomaticEquals();
    System.gc();
    test.testBrokenEquals();
  }

  // Do our little test that calls equals a lot of times.
  private void testSomething(List<Object> objects, Object object) {
    for (Object o : objects) {
      if (o.equals(object)) {
        return;
      }
    }
    objects.add(object);
  }

  // Test the performance of equals as implemented using pojomatic
  // http://pojomatic.sourceforge.net/pojomatic/index.html
  private void testPojomaticEquals() {
    Random rand = new Random(seed);
    List<Object> list = new ArrayList<Object>();
    long start = System.currentTimeMillis();
    for (long i = 0; i < count; i++) {
      String s = "asdf";
      long suffix = i * rand.nextInt(variation);
      MyObjectPojomatic obj = new MyObjectPojomatic(s + suffix, s + (suffix + 1), s + (suffix - 1),
          suffix, suffix + 1, suffix - 1);
      testSomething(list, obj);
    }
    long end = System.currentTimeMillis();
    System.out.println("Time taken for pojomatic is " + (end - start) + "ms" + " set size is "
      + list.size() + " calls = " + pojomaticCounter);
  }

  // test a broken implementation of equals to show a difference in the set size and total calls
  private void testBrokenEquals() {
    Random rand = new Random(seed);
    List<Object> list = new ArrayList<Object>();
    long start = System.currentTimeMillis();
    for (long i = 0; i < count; i++) {
      String s = "asdf";
      long suffix = i * rand.nextInt(variation);
      MyObjectBroken obj = new MyObjectBroken(s + suffix, s + (suffix + 1), s + (suffix - 1),
          suffix, suffix + 1, suffix - 1);
      testSomething(list, obj);
    }
    long end = System.currentTimeMillis();
    System.out.println("Time taken for broken is " + (end - start) + "ms" + " set size is "
      + list.size() + " calls = " + brokenCounter);
  }

  // test EqualsBuilder.reflectionEquals from apache commons.
  // http://commons.apache.org/lang/api-2.6/org/apache/commons/lang/builder/EqualsBuilder.html
  private void testReflectEquals() {
    Random rand = new Random(seed);
    List<Object> list = new ArrayList<Object>();
    long start = System.currentTimeMillis();
    for (long i = 0; i < count; i++) {
      String s = "asdf";
      long suffix = i * rand.nextInt(variation);
      MyObjectReflect obj = new MyObjectReflect(s + suffix, s + (suffix + 1), s + (suffix - 1),
          suffix, suffix + 1, suffix - 1);
      testSomething(list, obj);
    }
    long end = System.currentTimeMillis();
    System.out.println("Time taken for reflection is " + (end - start) + "ms" + " set size is "
      + list.size() + " calls = " + reflectCounter);
  }

  // test EqualsBuilder.append equals from apache commons.
  // http://commons.apache.org/lang/api-2.6/org/apache/commons/lang/builder/EqualsBuilder.html
  private void testAppendEquals() {
    Random rand = new Random(seed);
    List<Object> list = new ArrayList<Object>();
    long start = System.currentTimeMillis();
    for (long i = 0; i < count; i++) {
      String s = "asdf";
      long suffix = i * rand.nextInt(variation);
      MyObjectAppend obj = new MyObjectAppend(s + suffix, s + (suffix + 1), s + (suffix - 1),
          suffix, suffix + 1, suffix - 1);
      testSomething(list, obj);
    }
    long end = System.currentTimeMillis();
    System.out.println("Time taken for append is " + (end - start) + "ms" + " set size is "
      + list.size() + " calls = " + appendCounter);
  }

  // Test the equals method as generated by Eclipse IDE
  // http://eclipse.org/
  private void testPlainEquals() {
    Random rand = new Random(seed);
    List<Object> list = new ArrayList<Object>();
    long start = System.currentTimeMillis();
    for (long i = 0; i < count; i++) {
      String s = "asdf";
      long suffix = i * rand.nextInt(variation);
      MyObjectPlain obj = new MyObjectPlain(s + suffix, s + (suffix + 1), s + (suffix - 1), suffix,
          suffix + 1, suffix - 1);
      testSomething(list, obj);
    }
    long end = System.currentTimeMillis();
    System.out.println("Time taken for plain is " + (end - start) + "ms" + " set size is "
      + list.size() + " calls = " + plainCounter);
  }

  // Object that uses EqualsBuilder.append for equals.
  public class MyObjectAppend {
    public String s1;
    public String s2;
    public String s3;
    public Long l1;
    public Long l2;
    public Long l3;

    public MyObjectAppend(String a, String b, String c, Long d, Long e, Long f) {
      s1 = a;
      s2 = b;
      s3 = c;
      l1 = d;
      l2 = e;
      l3 = f;
    }

    @Override
    public boolean equals(Object obj) {
      appendCounter += 1;
      if (obj instanceof MyObjectAppend) {
        MyObjectAppend that = (MyObjectAppend) obj;
        EqualsBuilder builder = new EqualsBuilder();
        builder.append(this.s1, that.s1);
        builder.append(this.s2, that.s2);
        builder.append(this.s3, that.s3);
        builder.append(this.l1, that.l1);
        builder.append(this.l2, that.l2);
        builder.append(this.l3, that.l3);
        return builder.isEquals();
      }
      return false;
    }

    @Override
    public int hashCode() {
      HashCodeBuilder builder = new HashCodeBuilder();
      builder.append(this.s1);
      builder.append(this.s2);
      builder.append(this.s3);
      builder.append(this.l1);
      builder.append(this.l2);
      builder.append(this.l3);
      return builder.toHashCode();

    }
  }

  // Object that uses EqualsBuilder.reflectEquals for the implementation
  public class MyObjectReflect {
    public String s1;
    public String s2;
    public String s3;
    public Long l1;
    public Long l2;
    public Long l3;

    public MyObjectReflect(String a, String b, String c, Long d, Long e, Long f) {
      s1 = a;
      s2 = b;
      s3 = c;
      l1 = d;
      l2 = e;
      l3 = f;
    }

    @Override
    public boolean equals(Object obj) {
      reflectCounter += 1;
      return EqualsBuilder.reflectionEquals(this, obj);
    }

    @Override
    public int hashCode() {
      return HashCodeBuilder.reflectionHashCode(this);
    }
  }

  // Object that users pojomatic's equals implementation
  @AutoProperty
  public class MyObjectPojomatic {
    public String s1;
    public String s2;
    public String s3;
    public Long l1;
    public Long l2;
    public Long l3;

    public MyObjectPojomatic(String a, String b, String c, Long d, Long e, Long f) {
      s1 = a;
      s2 = b;
      s3 = c;
      l1 = d;
      l2 = e;
      l3 = f;
    }

    @Override
    public boolean equals(Object obj) {
      pojomaticCounter += 1;
      return Pojomatic.equals(this, obj);
    }

    @Override
    public int hashCode() {
      return Pojomatic.hashCode(this);
    }
  }

  // Object with a broken equals implementation.
  public class MyObjectBroken {
    public String s1;
    public String s2;
    public String s3;
    public Long l1;
    public Long l2;
    public Long l3;

    public MyObjectBroken(String a, String b, String c, Long d, Long e, Long f) {
      s1 = a;
      s2 = b;
      s3 = c;
      l1 = d;
      l2 = e;
      l3 = f;
    }

    @Override
    public boolean equals(Object obj) {
      brokenCounter += 1;
      return false;
    }

    @Override
    public int hashCode() {
      return 1;
    }
  }

  // Object that uses Eclipse's generated equals implementation.
  public class MyObjectPlain {
    public String s1;
    public String s2;
    public String s3;
    public Long l1;
    public Long l2;
    public Long l3;

    public MyObjectPlain(String a, String b, String c, Long d, Long e, Long f) {
      s1 = a;
      s2 = b;
      s3 = c;
      l1 = d;
      l2 = e;
      l3 = f;
    }

    @Override
    public int hashCode() {
      final int prime = 31;
      int result = 1;
      result = prime * result + getOuterType().hashCode();
      result = prime * result + ((l1 == null)
          ? 0
          : l1.hashCode());
      result = prime * result + ((l2 == null)
          ? 0
          : l2.hashCode());
      result = prime * result + ((l3 == null)
          ? 0
          : l3.hashCode());
      result = prime * result + ((s1 == null)
          ? 0
          : s1.hashCode());
      result = prime * result + ((s2 == null)
          ? 0
          : s2.hashCode());
      result = prime * result + ((s3 == null)
          ? 0
          : s3.hashCode());
      return result;
    }

    @Override
    public boolean equals(Object obj) {
      plainCounter += 1;
      if (this == obj)
        return true;
      if (obj == null)
        return false;
      if (getClass() != obj.getClass())
        return false;
      MyObjectPlain other = (MyObjectPlain) obj;
      if (!getOuterType().equals(other.getOuterType()))
        return false;
      if (l1 == null) {
        if (other.l1 != null)
          return false;
      }
      else if (!l1.equals(other.l1))
        return false;
      if (l2 == null) {
        if (other.l2 != null)
          return false;
      }
      else if (!l2.equals(other.l2))
        return false;
      if (l3 == null) {
        if (other.l3 != null)
          return false;
      }
      else if (!l3.equals(other.l3))
        return false;
      if (s1 == null) {
        if (other.s1 != null)
          return false;
      }
      else if (!s1.equals(other.s1))
        return false;
      if (s2 == null) {
        if (other.s2 != null)
          return false;
      }
      else if (!s2.equals(other.s2))
        return false;
      if (s3 == null) {
        if (other.s3 != null)
          return false;
      }
      else if (!s3.equals(other.s3))
        return false;
      return true;
    }

    private EqualsTest getOuterType() {
      return EqualsTest.this;
    }
  }
}

And here’s the output

Time taken for plain is 804ms set size is 8696 calls = 39185108
Time taken for append is 3030ms set size is 8696 calls = 39185108
Time taken for reflection is 20765ms set size is 8696 calls = 39185108
Time taken for pojomatic is 4094ms set size is 8696 calls = 39185108
Time taken for broken is 604ms set size is 10000 calls = 49995000

Given that the set size and the number of equals calls are the same, you can be reasonably sure that each implementation is as correct as the others, other than the intentionally broken one, which was meant to help illustrate this point.

The results speak for themselves. The Eclipse generated solution is fastest, followed by append, followed closely by pojomatic. Finally, the reflection based implementation took over 25x the amount of time as the fastest solution.

« Newer PostsOlder Posts »

Blog at WordPress.com.