Tuning the Application Server

For most peoples needs JBoss would work adequately straight out of the box especially with today's hardware, however there are a number of companies that have large volumes of traffic that require you to tune JBoss to obtain the optimal performance for your application. As I discussed in my Tomcat tuning section performance tuning is a complex area, there are many parts to the equation networking, hardware, application, databases, JVM, etc. There is even a possibility to make the system worse, the best approach to take with tuning is an iterative one, for an example you should use something like below

1. Set up a test bed
2. Do performance testing to obtain a baseline
3. Investigate bottlenecks within the system these include network, hardware, database, etc
4. Identify areas where improvements are needed which may or may not include changes to your web app
5. Make one fix
6. Repeat steps 2 through 5

The two areas that a user is concerned about are

Before you start to performance tune you should set yourself some goals on what you want to achieve (see below for some idea's), normally these are backed up by an agreement with the business called an SLA (Service Level Agreement), the SLA is a contract to provide a certain level of service between the vendor who's providing the service and the customer, this agreement will define the excepted response times and throughput levels. With this information you then may have a idea on the sort of hardware that many be purchase and how this will fit into your technical architecture.

• How long should a web transaction take?
• How long should a user wait before a page loads?
• How many users should the Web site support?
• What kind of traffic with will be expecting (bursty or constant)

I am not going to go into too much details here as there are plenty of documents on the internet that do better justice then I can, so what I am going to present is a simple guide on what options are available to tune and some recommendations, but the best source would be the JBoss web site.

Hardware Tuning

We first start with the network card, you need to make sure that these are running at their maximum speed as there are a number of speeds that they can run at, most networks now days should support at least 100MB full-duplex, I point you to my Unix cheat sheet on how to obtain if your network cards have been setup correctly.

Having multiple CPU's in a server obviously gives you better performance, however adding additional CPU's to an existing system many not improve performance, as I noted above there are many parts and the CPU many not be the bottleneck. Having multiple CPU's works well with Java threads, as they don't have to wait for long before getting onto a CPU, however there can be a problem by using too many CPU's for a thread this is called processor affinity

When the processor is executing the thread of a process, the data and the code used by the thread is stored within a cache within the processor. When the thread is interrupted (thread is either blocked while waiting on some i/o or used up its time allotment), another thread runs and starts to overwriting some of that cache code and data. If the cache is large enough there many be a chance that when the thread gets back on the processor that some of the code and data is still there but if the thread is moved to another processor then the thread must fill the cache with the code and data again, remember that accessing the processors cache is many times faster than accessing RAM. So there is a memory latency if the thread is moved from processor to processor, thus you are trying to limit the number of processors that the thread moves hoping that cached data is still there and not over written.

So with the information above the best processor affinity setup would be to allow a thread to run on two - four processors, this does not mean you cannot have a server with 32 CPU's what it means is that a thread will only try and use 4 out of the 32 CPU's on that server, which four is up to the O/S.

JVM Tuning

There are two JVM's

• server
• client

The difference between them is how they handle compiling Java bytecode into native machine code and how they manage the heap (memory allocation), I will go into more on the heap in a moment. The client is intended for short running programs (Ant) and the server for long-running programs (JBoss). You can choose one to run by using the JAVA_OPTS setting and the -server or the -client option must be the first setting.

The JVM allocates Java objects in an area of memory called the Java heap, what I am about to describe is the Sun's JVM (HotSpot) other JVM's may differ. A Java application will place objects onto the heap, when the application has finished with the object it removes all references to that object, which means the object is now unreachable. When an object becomes unreachable it is deemed as available to the garbage collector, the garbage collector will free up memory by removing any unreachable objects, the garbage collector typically runs when memory resources are low.

Most JVM objects have a short lifespan and are said to die young, the JVM's heap is divided into several areas, each area is collected individually and each has its own collection algorithms.

There are a number of JVM heap sizing arguments that you can supply

There is a recommendation that you should set the min and max size of the heap to the same, the reason behind this is that if the values are different the JVM takes away time from processing the application to determine if it should adjust the sizes after garbage collection, this also applies to the young generation sizes, also it is recommended that the young generation size should be one-third to one-quarter the size of the heap. An example of adjusting the memory is below, I suggest you play around with your settings to obtain the optimal setting, how do you now when they are optimal well keep read and all will be explained later.

There are two categories of garbage collection

• Minor - cleans out the young generation only
• Major - otherwise know as a full collection cleans out the tenured and young generations

When the young generation is collected, all objects still in use are moved to the survivor space (to space), these objects are said to be alive and objects no longer in use are dead. After the collection the eden space and survivor space (from space) are empty. If the to space fills up during the collection any excess surviving objects are placed in the tenured generation, also any objects that have survived several minor collections are also placed into the tenured generation. Once the collection is finished the eden space and the from space are empty and within the young generation space only the to space contains in-use objects, At this point in time, the from space and the to space change destinations because after the collection the to space is always empty.

Before a minor collection the JVM looks at the tenured generation and determines if there's ample space to hold any objects from the young generation that overflow into the tenured generation, if not, then it performs a collection on the tenured generation first, this is known as a full collection, the tenured generation is also compacted moving all the objects to one end of the tenured generation space.

There are two collector types

• Serial - pauses the Java application until the collection has finished
• Concurrent - performs most of the collection while the Java application continues to run, only two shorts phases causes the application to pause

By default if the server has multiple processors and at least 1GB of memory the JVM turns on UseParallelGC automatically for the serial collector, the JVM uses one collector thread for every processor on the server. The arguments that you can use are the following

You can also adjust the number of threads using the -XXParallelGCThreads option.

To gather garbage collection data you can use a number of parameters

I will leave you to investigate the log files, but here is a Java program (but you can use what ever you want) that you can use to put the data and into a CVS file that excel can chart.

The type of graph you are looking for is one that starts performing garbage collections but then progresses into a smooth line (see below diagram).

You are trying to achieve the following by adjusting the heap sizes, be careful making the heap size to large, as this will make garbage collection take longer (could take 20mins if you set it to 20GB), try to make the size just enough. You are trying to archive the following

• No full (major) Collections - These are expensive
• As few minor collections as possible - Each collection takes time

There are some more tuning parameters that you can use relating to the JVM

JBoss Tuning

There are several things that can be tuned in JBoss, some affect all applications other only affect applications that many use a particular service.

JBoss maintains a pool of database connections which are used by the application, in the *-ds.xml file you can specify information about the pool size, try not to set a too high value as each connection uses systems resources which are wasted if you are not using them.

There is a way to monitor the database connection pool usage, when a datasource is deployed the application server creates three MBeans, one of which manages the connection pool

Another configuration option regarding the data source is the prepared statement cache, this cache holds the recently used prepared statements in case they are used again.

You can also tune the HTTP request pool and I point you to the topic configuring connectors for more information on this.

The server/xxx/deployers/jbossweb.deployer/web.xml file contains initialization parameters for the JSP servlet, I have listed some of the parameters that you can change to increase performance