Note to self: Java Grails application hot deployment updates with Maven, Cargo, Apache Tomcat, PostgreSQL

This is not an attempt at buzzword bingo, nor listing all existing Open Source projects.

This is actually just a blog entry to help me remember how to get started with the production environment for my Grails Application for my new service at http://www.freescan.dk - which is hopefully going to be a nice project in itself, but also a commercial success later by sending customers to my company :-)

The application is about portscanning, but this blog entry is about Project Infrastructure and getting my development and test environment up and running, and later be able to move into a production environment. If you just want a quick portscan try going to the site http://www.freescan.dk/ with the changes being done I will do most of my development in another "context" on Tomcat, so unless the application has crashed you should be fine - and poke me on IRC if it doesn't work :-)

The goals as such are:

• Be able to develop the software - allow me to use Textmate on the Grails files, like any other small project, if others join in let them use whatever they like
• Be able to deploy a new version for testing - on an environment which resembles the real environment as much as possible
• Do not repeat myself, nor do tedious manual work when trying out new versions in the testing environment

To begin with it is easy to get started using Grails, just download from http://www.grails.org and run the command grails create-app and start hacking. Lots of fun and quickly you can start working with domain classes and controllers - and let the Grails web application framework "scaffold" and build views and CRUD elements.

I have a lot of parts already in place, like basic PostgreSQL installation (being used for Prelude and other stuff) and have switched to Subversion for versioning of files a few years back. PostgreSQL was selected for the excellent documentation and because it is a real database, and works very consistent. I have selected Subversion mostly for the easy moving/renaming of directories and files - feels more "modern" fixing various stuff which CVS does badly.

The Java Servlet container I have chosen is Apache Tomcat which I have found is easy to setup and run.

It is easy to setup a Subversion repository and a PostgreSQL database, so I won't get into this - if you like CVS or MySQL then fine with me. To summarize I have to begin with:

• Development environment selected Grails framework
• Subversion installed - if you add this as the first thing after creating the project or later is up to you
• PostgreSQL installation - if you decide to run with in-memory HSQLDB database or immediately switch to a real database is also up to you
• Tomcat installation - Grails use Jetty while you do development which is not really a production environment
• Operating system, backup procedures for files and databases, hardened server, RAID mirror, passwords, SSH keys etc.

Documentation

I did not write anything myself for the project infrastructure, I have downloaded and used some existing tools - what is the great thing is that they work nicely together. To get them to work together required some magic dust - which I found on these pages:

• http://www.grails.org/Maven+Integration which has the basic information about Maven and Grails Note: this is for the Grails release candidates! I currently use 1.1beta2
• http://thedevelopercorner.blogspot.com/2007/01/using-maven-to-deploy-application-to.html - listing the actual configuration you need to add deployment to the Maven configuration file (pom.xml) to allow you to easily deploy
  

Tools needed

The tools I used below are found from various pages:

• PostgreSQL database http://www.postgresql.org/ and also the JDBC connector http://jdbc.postgresql.org/download.html
• Maven http://maven.apache.org/ make sure you at least read the introduction Maven in 5 minutes
• Grails - but this will actually get installed by Maven

Most likely these projects are available in the ports/packaging system that your favorite operating system uses. I use a Mac for my development so I got Maven from Macports project.

Creating a deployable project

The thing that got me started was reading the release notes for the Grails 1.1beta2 - while thinking about deploying new versions on Tomcat without doing the manual work of creating and copying the WAR file to the server and deplying (grails war ; scp application.war server: ; ssh server do-something). These release notes pointed me to the Maven integration and step 1.

Step 1. Maven installed and ready

The first step is to download an install Maven - I use Maven 2.0.9 which is minimum requirement for the Grails+Maven stuff

hlk@bigfoot:hlk$ cd src/maven
hlk@bigfoot:maven$ mvn -version
Maven version: 2.0.9
Java version: 1.5.0_16
OS name: "mac os x" version: "10.5.6" arch: "i386" Family: "unix"

Now would perhaps be a good time to experiment with Maven and reading the introduction from the website? Make sure that Java and Maven works before moving on.

Step 2. Create new Grails project with Maven

Then we are ready to start a new project using the command specified, I call my group FreeScan.dk and "application" DeployTester.

hlk@bigfoot:maven$ mvn archetype:generate -DarchetypeGroupId=org.grails \
-DarchetypeArtifactId=grails-maven-archetype \
-DarchetypeVersion=1.0-beta2 \
-DarchetypeRepository=http://snapshots.repository.codehaus.org \
-DgroupId=FreeScan.dk -DartifactId=DeployTester

Yes, that is a bitch - but as discussed elsewhere Maven can do a lot of stuff and you really do need to specify what you want to have done. The output from the command when you run it the first time is HUGE - since Maven will actually download quite a lot of JAR files, tools and even Grails! My output from creating yet another project is much smaller :-)

[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'archetype'.
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Default Project
[INFO]    task-segment: [archetype:generate] (aggregator-style)
[INFO] ------------------------------------------------------------------------
[INFO] Preparing archetype:generate
[INFO] No goals needed for project - skipping
[INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO] [archetype:generate]
[INFO] Generating project in Interactive mode
[INFO] Archetype defined by properties
Define value for version:  1.0-SNAPSHOT: : initial
Confirm properties configuration:
groupId: FreeScan.dk
artifactId: DeployTester
version: initial
package: FreeScan.dk
Y: :
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating OldArchetype: grails-maven-archetype:1.0-beta2
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: FreeScan.dk
[INFO] Parameter: packageName, Value: FreeScan.dk
[INFO] Parameter: basedir, Value: /userdata/src/maven
[INFO] Parameter: package, Value: FreeScan.dk
[INFO] Parameter: version, Value: initial
[INFO] Parameter: artifactId, Value: DeployTester
[WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference : template = archetype-resources/pom.xml [line 120,column 20] : ${java.version} is not a valid reference.
[WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference : template = archetype-resources/pom.xml [line 122,column 23] : ${java.home} is not a valid reference.
[INFO] ********************* End of debug info from resources from generated POM ***********************
[INFO] OldArchetype created in dir: /userdata/src/maven/DeployTester
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11 seconds
[INFO] Finished at: Thu Jan 22 10:38:03 CET 2009
[INFO] Final Memory: 8M/14M
[INFO] ------------------------------------------------------------------------

Step 3. Initialize this Grails project

The next step is going into the application directory and make this project ready for development:

hlk@bigfoot:maven$ cd DeployTester/
hlk@bigfoot:DeployTester$ ls
grails-app/     pom.xml         src/

This does not really look like a Grails project yet, you need to initialize - as described on the web page from Grails.org:

hlk@bigfoot:DeployTester$ mvn initialize
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - FreeScan.dk:DeployTester:war:initial
[INFO]    task-segment: [initialize]
[INFO] ------------------------------------------------------------------------
[INFO] [grails:validate {execution: default}]
[INFO] No Grails application found - skipping validation.
[INFO] [grails:init {execution: default}]
[INFO] Cannot read application info, so initialising new application.
[INFO] Creating '/userdata/src/maven/DeployTester/target/grails-lib' directory for Grails JARs
Running pre-compiled script
Environment set to development
[mkdir] Created dir: /userdata/src/maven/DeployTester/src/java
[mkdir] Created dir: /userdata/src/maven/DeployTester/src/groovy
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/controllers
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/services
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/domain
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/taglib
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/utils
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/views
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/views/layouts
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/i18n
[mkdir] Created dir: /userdata/src/maven/DeployTester/test
[mkdir] Created dir: /userdata/src/maven/DeployTester/test/unit
[mkdir] Created dir: /userdata/src/maven/DeployTester/test/integration
[mkdir] Created dir: /userdata/src/maven/DeployTester/scripts
[mkdir] Created dir: /userdata/src/maven/DeployTester/web-app
[mkdir] Created dir: /userdata/src/maven/DeployTester/web-app/js
[mkdir] Created dir: /userdata/src/maven/DeployTester/web-app/css
[mkdir] Created dir: /userdata/src/maven/DeployTester/web-app/images
[mkdir] Created dir: /userdata/src/maven/DeployTester/web-app/META-INF
[mkdir] Created dir: /userdata/src/maven/DeployTester/lib
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/conf/spring
[mkdir] Created dir: /userdata/src/maven/DeployTester/grails-app/conf/hibernate
[propertyfile] Creating new property file: /userdata/src/maven/DeployTester/application.properties
[copy] Copying 1 resource to /userdata/src/maven/DeployTester
[unjar] Expanding: /userdata/src/maven/DeployTester/grails-shared-files.jar into /userdata/src/maven/DeployTester
[delete] Deleting: /userdata/src/maven/DeployTester/grails-shared-files.jar
[copy] Copying 1 resource to /userdata/src/maven/DeployTester
[unjar] Expanding: /userdata/src/maven/DeployTester/grails-app-files.jar into /userdata/src/maven/DeployTester
[delete] Deleting: /userdata/src/maven/DeployTester/grails-app-files.jar
[move] Moving 1 file to /userdata/src/maven/DeployTester
[move] Moving 1 file to /userdata/src/maven/DeployTester
[propertyfile] Updating property file: /userdata/src/maven/DeployTester/application.properties
Created Grails Application at /userdata/src/maven/DeployTester
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15 seconds
[INFO] Finished at: Thu Jan 22 10:44:37 CET 2009
[INFO] Final Memory: 17M/34M
[INFO] ------------------------------------------------------------------------

Much better! Now we see the usual structure of a Grails project (some parts pruned to shorten output)

|-- DeployTester.launch
|-- DeployTester.tmproj
|-- application.properties
|-- build.xml
|-- grails-app
| |-- conf
| | |-- BootStrap.groovy
| | |-- BuildConfig.groovy
| | |-- Config.groovy
| | |-- DataSource.groovy
| | |-- UrlMappings.groovy
| | |-- hibernate
| | `-- spring
| |     `-- resources.groovy
| |-- controllers
| |-- domain
| |-- i18n
| | |-- messages.properties
| | |-- messages_de.properties
| | |-- messages_es.properties
| | `-- messages_zh_CN.properties
| |-- services
| |-- taglib
| |-- utils
| `-- views
|     |-- error.gsp
|     `-- layouts
|         `-- main.gsp
|-- lib
|-- pom.xml
|-- scripts
|-- src
| |-- groovy
| |-- java
| `-- main
|     |-- resources
|     `-- webapp
|         `-- WEB-INF
|             `-- web.xml
|-- target
| `-- grails-lib
|-- test
| |-- integration
| `-- unit
`-- web-app
    |-- META-INF
    |-- WEB-INF
    | |-- applicationContext.xml
    | |-- sitemesh.xml
    | `-- tld
    |     |-- grails.tld
    |     `-- spring.tld
    |-- css
    | `-- main.css
    |-- images
    | |-- favicon.ico
    | |-- grails_logo.jpg
    | |-- skin
    | | |-- database_add.png
    | | |-- database_delete.png
    | | |-- database_edit.png
   | | `-- sorted_desc.gif
    | `-- spinner.gif
    |-- index.gsp

35 directories, 57 files

Step 4. Test the Grails app

This might be a good time to try out the Grails application we just initialized

hlk@bigfoot:DeployTester$ mvn grails:run-app
[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'grails'.
[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - FreeScan.dk:DeployTester:war:initial
[INFO]    task-segment: [grails:run-app]
[INFO] ------------------------------------------------------------------------
[INFO] [grails:run-app]
Running pre-compiled script
Environment set to development
[mkdir] Created dir: /Users/hlk/.grails/1.1-beta2/projects/DeployTester/classes
[groovyc] Compiling 7 source files to /Users/hlk/.grails/1.1-beta2/projects/DeployTester/classes
[mkdir] Created dir: /Users/hlk/.grails/1.1-beta2/projects/DeployTester/resources/grails-app/i18n
[native2ascii] Converting 11 files from /userdata/src/maven/DeployTester/grails-app/i18n to /Users/hlk/.grails/1.1-beta2/projects/DeployTester/resources/grails-app/i18n
[copy] Copying 1 file to /Users/hlk/.grails/1.1-beta2/projects/DeployTester/classes
[copy] Copying 1 file to /Users/hlk/.grails/1.1-beta2/projects/DeployTester/resources
Running Grails application..
Server running. Browse to http://localhost:8080/DeployTester

This Maven command runs the Grails framework command run-app which compiles the necessary files and then run a Jetty server - allowing us the run a browser against the URL shown:

Step 5. Add the Cargo deployment

The above created an application, and so what? It used Maven, so what? It can build a WAR using Maven - but we need to deploy! So we need to specify where to deploy and also some authentication :-) This is from the blogspot.com link above - with some small edits, correcting the groupid to groupId etc. and should be inserted into the pom.xml which is the Maven configuration:

<build>

<plugins>
...

<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<configuration>
<!-- Container configuration -->
<container>
<containerId>tomcat6x</containerId>
<type>remote</type>
</container>
<!-- Configuration to use with the Container -->
<configuration>
<type>runtime</type>
<properties>
<cargo.tomcat.manager.url>http://server:8080/manager</cargo.tomcat.manager.url>
<cargo.remote.username>adminuser</cargo.remote.username>
<cargo.remote.password>adminuserpassword</cargo.remote.password>
</properties>
</configuration>
<!-- Deployer configuration -->
<deployer>
<type>remote</type>
<deployables>
<deployable>
<groupId>FreeScan.dk</groupId>
<artifactId>DeployTester</artifactId>
<type>war</type>
</deployable>
</deployables>
</deployer>
</configuration>
</plugin>

</plugins>
</build>

With this added you can do a build with mvn package and then deploy this packaged application to the server with mvn cargo:deploy. The interesting part of the output is the deploy part, which is:

hlk@bigfoot:DeployTester$ mvn cargo:deploy
[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'cargo'.
[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - FreeScan.dk:DeployTester:war:initial
[INFO]    task-segment: [cargo:deploy]
[INFO] ------------------------------------------------------------------------
[INFO] [cargo:deploy]
[INFO] [mcat6xRemoteDeployer] Deploying [/userdata/src/maven/DeployTester/target/DeployTester-initial.war]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 50 seconds
[INFO] Finished at: Thu Jan 22 11:09:24 CET 2009
[INFO] Final Memory: 9M/18M
[INFO] ------------------------------------------------------------------------

Step 6. Add to the application - do some development

With everything in place (hopefully along the way you added versioning with CVS, Subversion or whatever) we can do some development.

Some ideas might be:

• Add the PostgreSQL JDBC jar-file to the lib directory, needed for the database integration - either Grails lib or the Tomcat lib library!
• Add some domain classes, read any Grails tutorial or the excellent Grails books from Apress.com - I have bought the PDF versions also.

Whenever you feel like it build and upload your application, using the Maven commands - with all the consistency that Maven provides. When the application is already deployed use the mvn cargo:deployer-redeploy to do undeploy and then deploy.

I did some commands adding a domain class User and controller UserController:

• mvn grails:create-domain-class added class User with strings Firstname, Lastname, Email
• mvn grails:create-controller added UserController which is just "scaffold = User"
  

Then some testing with Grails run-app and finally again the Maven mvn package and mvn cargo:deployer-redeploy :

Step 7. Change the application production database

I promised something about databases, by talking about PostgreSQL, so lets move on. First I created a database as the PostgreSQL user with createdb deploytest on Pumba - my Tomcat host. You get to choose your own host, database and name :-)

Then I reconfigured the production environment in my Grails application, which is defined in the file grails-app/conf/DataSource.groovy:

dataSource {
        pooled = true
        driverClassName = "org.hsqldb.jdbcDriver"
        username = "sa"
        password = ""
}
hibernate {
cache.use_second_level_cache=true
cache.use_query_cache=true
cache.provider_class='com.opensymphony.oscache.hibernate.OSCacheProvider'
}
// environment specific settings
environments {
 development {
              dataSource {
                       dbCreate = "create-drop" // one of 'create', 'create-drop','update'
                        url = "jdbc:hsqldb:mem:devDB"
              }
  }
  test {
             dataSource {
                       dbCreate = "update"
                        url = "jdbc:hsqldb:mem:testDb"
             }
  }
  production {
               dataSource {
                       dbCreate = "update"
                        //url = "jdbc:hsqldb:file:prodDb;shutdown=true"
                    driverClassName = "org.postgresql.Driver" //the database driver class name
                 username = "_postgresql" //database name
                   password = "123456" //database password
                    url = "jdbc:postgresql://localhost:5432/deploytest"         }
        }
}

The changes are in the production environment, where I have added the driverClassName for the PostgreSQL, username, password and then changed the URL part.

Then after the package'ing and deploy I have the database on the server Pumba :-)

$ psql deploytest 
Welcome to psql 8.3.5, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

deploytest=# \d
                  List of relations
 Schema |        Name        |   Type   |    Owner    
--------+--------------------+----------+-------------
 public | hibernate_sequence | sequence | _postgresql
 public | users              | table    | _postgresql
(2 rows)

deploytest=# select * from users;
 id | version |     email     | firstname | lastname 
----+---------+---------------+-----------+----------
  1 |       0 | hlk@kramse.org | Henrik    | Kramshøj
(1 row)

deploytest=# 

Warning: I had to cheat! It seems user is a reserved word in Postgresql, so either name your class something else - I changed user to users, or map this using a static mapping = { table "`user`" } in your application. Sorry, but I spent to much time writing this blog entry already to go back and changing everything ;-)

What is the next step?

• Develop some more
• When ready - lock down the database, so it won't accidently is wiped out due to some bad change etc.
• Investigate the things Maven can do for you
• Define criteria for releases and do releases
• ... etc.

This is to much hassle!

You might find this a bit to much, especially the Maven initialize will download quite a lot of files - but really it does a lot of work later on, so don't worry.

I find it nice to setup things once and then be able to use simple commands to do complex stuff!