niceideas.ch

Bytecode manipulation with Javassist for fun and profit part II: Generating toString and getter/setters using bytecode manipulation

Jerome Kehrli — Mon, 24 Apr 2017 16:38:42 -0400

Following my first article on Bytecode manipulation with Javassist presented there: Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code, I am here presenting another example of Bytecode manipulation with Javassist: generating toString method as well as property getters and setters with Javassist.

While the former example was oriented towards understanding how Javassist and bytecode manipulation comes in help with implementing IoC concerns, such as what is done by the spring framework of the pico IoC container, this new example is oriented towards generating boilerplate code, in a similar way to what Project Lombok is doing.
As a matter of fact, generating boilerplate code is another very sound use case for bytecode manipulation.

Boilerplate code refers to portions of code that have to be included or written in the same way in many places with little or no alteration.
The term is often used when referring to languages that are considered verbose, i.e. the programmer must write a lot of code to do minimal job. And Java is unfortunately a clear winner in this regards.
Avoiding boilerplate code is one of the main reasons (but by far not the only one of course !) why developers are moving away from Java in favor of other JVM languages such as Scala.

In addition, as a reminder, a sound understanding of the Java Bytecode and the way to manipulate it are strong prerequisites to software analytics tools, mocking libraries, profilers, etc. Bytecode manipulation is a key possibility in this regards, thanks to the JVM and the fact that bytecode is interpreted.
Traditionally, bytecode manipulation libraries suffer from complicated approaches and techniques. Javassist, however, proposes a natural, simple and efficient approach bringing bytecode manipulation possibilities to everyone.

So in this second example about Javassist we'll see how to implement typical Lombok features using Javassist, in a few dozen lines of code.

Part of this article is available as a slideshare presentation here: https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit.

You might want to have a look at the first article in this serie available here : Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code.

Summary

1. Introduction / Purpose
2. Javassist
3. Java Instrumentation framework and applications
- 3.1 Lombok
- 3.2 Java agents and the linkage problem
4.0 BCG: a simple approach for generating boilerplate code using Javassist
5. Conclusion

1. Introduction / Purpose

I am giving only a few lines here and return the user to my first article on this topic for the complete introduction to bytecode manipulation.

Bytecode manipulation consists in modifying the classes - represented by bytecode - compiled by the Java compiler, at runtime. It is used extensively for instance by frameworks such as Spring (IoC) and Hibernate (ORM) to inject dynamic behaviour to Java objects at runtime.
Bytecode manipulation is traditionally difficult. Out of all the various libraries and tools to achieve it, Javassist stands out due to its natural and simple yet efficient approach to it.

2. Javassist

I am giving only a few lines here and return the user to my first article on this topic for the complete introduction to Javassist.

Quoting Wikipedia:

Javassist (Java programming assistant) is a Java library providing a means to manipulate the Java bytecode of an application.

Javassist enables Java programs to define a new class at runtime and to modify a class file when the JVM loads it. Unlike other similar bytecode editors, Javassist provides two levels of API: source level and bytecode level. Using the source-level API, programmers can edit a class file without knowledge of the specifications of the Java bytecode; the whole API is designed with only the vocabulary of the Java language. Programmers can even specify inserted bytecode in the form of Java source text; Javassist compiles it on the fly. On the other hand, the bytecode-level API allows the users to directly edit a class file as other editors.

3. Java Instrumentation framework and applications

Java 5 was the first version seeing the proper implementation of JSR-163 (Java Platform Profiling Architecture API) support including a bytecode instrumentation mechanism through the introduction of the Java Programming Language Instrumentation Services (JPLIS). At first that JSR only mentioned native (C) interfaces but evolved fast towards a pretty convenient Java API.
This was an interesting breakthrough since it allowed, with the help of an agent, to modify the content of a class bytecode inherent to the methods of a class in such a way as to modify its behavior at runtime.

The key point of the JSR-163 is JVMTI. JVMTI - or Java Virtual Machine Tool Interface - allows a program to inspect the state and to control the execution of applications running in the Java Virtual Machine. JVMTI is designed to provide an Application Programming Interface (API) for the development of tools that need access to the state of the JVM. Examples for such tools are debuggers, profilers or runtime boilerplate code generator.

The scope of this article is to focus on presenting how a boilerplate code generator can benefit from JVMTI and bytecode manipulation to achieve unprecedented comfort and easiness of programming than ever before on the JVM, unless considering a different language than Java, such as Scala or Clojure.

As a way to illustrate Boilerplate code concerns, We will focus on Project Lombok and its features and try to see how to reproduce some of them using Javassist.

3.1 Lombok

Project Lombok is a Boilerplate code generator that addresses one of the most frequent criticism against the Java Programming Language: the volume of this type of code that is found in most projects.
Boilerplate code is a term used to describe code that is repeated in many parts of an application with only slight contextual changes and with little added value.
Project Lombok reduces the need of some of the worst offenders by replacing each of them with a simple annotation. It then takes care of generating the boilerplate code at runtime

Project Lombok also integrates well in IDEs making it possible to use the usual IDE features such as refactoring and usage analysis pretty transparently.

Importantly in our context, since we will focus below on reproducing Lombok features using runtime bytecode generation, I have to mention right away that Lombok doesn't just generate Java sources or bytecode: it transforms the Abstract Syntax Tree (AST), by modifying its structure at compile-time.
The AST is a tree representation of the parsed source code, created by the compiler, similar to the DOM tree model of an XML file. By modifying (or transforming) the AST, Lombok keeps the source code trim and free of bloat, unlike plain-text code-generation.
Lombok's generated code is also visible to classes within the same compilation unit, unlike direct bytecode manipulation.
Our approach, as we will see below, using runtime bytecode manipulation shall not benefit from the same comfort and is hence less efficient than what Lombok is doing.

3.1.1 Example class

Let's see an example. Imagine the following Java POJO:

public class DataExample {

    private final String name;

    private int age;

    private double score;

    private String[] tags;
}

Typical boilerplate code involved when considering such a POJO are:

Getters and Setters for all private fields, making them JavaBean properties
A nice toString method giving the values of its properties when an object is output on the console
Consistent hashCode and equals methods enabling to compare and manipulate two different objects with same values
A default constructor without any argument (Javabean standard)
An all args constructor taking all values as argument to build the instance

3.1.2 Without Lombok

Without Lombok, writing all this code is a nightmare, and the simple class above becomes as follows:

import java.util.Arrays;

public class DataExample {

    private final String name;
    private int age;
    private double score;
    private String[] tags;
  
    public DataExample(String name) {
        this.name = name;
    }

    public DataExample(String name, int age, double score, String[] tags) {
        this.name = name;
        this.age = age;
        this.score = score;
        this.tags = tags;
    }
  
    public String getName() {
        return this.name;
    }
  
    void setAge(int age) {
        this.age = age;
    }
    public int getAge() {
        return this.age;
    }
  
    public void setScore(double score) {
        this.score = score;
    }
    public double getScore() {
        return this.score;
    }
  
    public String[] getTags() {
        return this.tags;
    }
    public void setTags(String[] tags) {
        this.tags = tags;
    }
  
    @Override 
    public String toString() {
        return "DataExample(" + this.getName() + 
            ", " + this.getAge() + 
            ", " + this.getScore() + 
            ", " + Arrays.deepToString(this.getTags()) + ")";
    }

    @Override 
    public boolean equals(Object o) {
        if (o == this) return true;
        if (!(o instanceof DataExample)) return false;
        DataExample other = (DataExample) o;
        if (this.getName() == null ? 
                other.getName() != null : 
                !this.getName().equals(other.getName())) 
            return false;
        if (this.getAge() != other.getAge()) return false;
        if (Double.compare(this.getScore(), other.getScore()) != 0) return false;
        if (!Arrays.deepEquals(this.getTags(), other.getTags())) return false;
        return true;
    }
  
    @Override 
    public int hashCode() {
        final int PRIME = 59;
        int result = 1;
        final long temp1 = Double.doubleToLongBits(this.getScore());
        result = (result*PRIME) + (this.getName() == null ? 43 : this.getName().hashCode());
        result = (result*PRIME) + this.getAge();
        result = (result*PRIME) + (int)(temp1 ^ (temp1 >>> 32));
        result = (result*PRIME) + Arrays.deepHashCode(this.getTags());
        return result;
    }
}

One needs to understand that while the features implemented by the code above are pretty useful and very important, the code itself has no added value whatsoever. Most IDEs generate this code for you using some right-click here and there. A machine can write this code for you, can you imagine this? ... And yet with java it HAS to be written. This makes no sense.

So without Lombok, for a 4 properties, 5 lines of code class ... One has to write more than 60 lines of boilerplate code, that's a ratio of [Boilerplate code / Useful Code] of more than 1600% !!!
This is the main rationality behind project Lombok.

3.1.3 With Lombok

With Lombok we can use the following set of annotations on top of the class to generate the very same boilerplate code that we had to write on our own above:

@Getter
@Setter
@ToString
@EqualsAndHashCode
@RequiredArgsConstructor
@AllArgsConstructor
public class DataExample {

  private final String name;

  private int age;

  private double score;

  private String[] tags;
}

All these annotations are straightforward to understand so I won't be describing them any further.

You might want to look at the Lombok documentation to learn more about these annotations.

With Lombok, we get a much better ratio of boilerplate code concerns vs. useful code of 100% instead of more than 1600% without Lombok.

3.1.4 Lombok Apporoach (AST transformation) vs. Bytecode Manipulation vs. Code Generation

Code generation

Of course there are alternatives to Lombok. For instance, most if not all IDEs enable the developer to generate these boilerplate methods in the class source file just as Lombok does it.
But generating this code in the class source file is IMHO not the right approach. At the end of the day, I believe that generated code - i.e. code that can be written by a machine - should simply not be written! If a machine can write this code then this code has no added value at all! I do not want to see such code in my class files, I do not want to be aware of it.
Lombok generates this code transparently and seamlessly. By modifying (or transforming) the AST, Lombok keeps your source code trim and free of bloat, unlike plain-text code-generation. With Lombok, I do not need to be aware of this code, it's not polluting my code coverage computation (in Sonar for instance or else) and everything behaves as if this code was actually written except that one doesn't see it and doesn't need to care about it.

Bytecode Manipulation

Bytecode manipulation is preferable over Code generation IMHO in regards to generating (at least some of the) boilerplate code. For the same reason indicated above: bytecode manipulation makes the boilerplate code transparent and avoids it polluting my source code.
It is however not as efficient as AST Transformation such as Lombok is doing. Lombok's generated code is also visible to classes within the same compilation unit, unlike direct bytecode manipulation.

For instance when it comes to generating getters and setters, Lombok makes them visible at compile time, making it possible for client code to use them without them appearing in the Java Bean class.
Using bytecode manipulation to generate getters and setters, they become invisible to the compiler and cannot be used by another class, except using runtime reflection, which may be fine for some use cases (hibernate, etc) but not for most of them.

AST Transformation

More information on Lombok's internal is availanle there: Custom AST transformations with Project Lombok.

3.1.5 Just a note on concerns

I sometime hear (... read) arguments online against the use of Lombok. Most of the time, people against Lombok complain about the magic added by Lombok.

It is true that Lombok adds a lot of magic to the application. But that magic is related to boilerplate code with no added value and easy to debug and understand in case of any doubt. In addition, it is very well recognized by most common IDEs such as eclipse and IDEA using well known or even official plugins.

3.2 Java agents and the linkage problem

Javassist cannot modify a class after it has been loaded by a classloader ... as far as this classloader is concerned.
Whenever one tries to modify a class already loaded by the referenced classloader, that attempt to call pool.makeClass( ... ) will fail and complain that class is frozen (i.e. already created via toClass().
Being able to do that would require to unload the class first from the reference Classloader.

The problem here is that one cannot unload a single class from a ClassLoader. A class may be unloaded if it and its ClassLoader became unreachable but since every class refers to its loader that implies that all classes loaded by this loader must have to become unreachable too.
Of course one can (re-)create the class using a different ClassLoader but that would require to make the whole program use that new Classloader and this becomes fairly complicated.
At the end of the day, that would well require reloading the whole application and initializing everything all over again. This makes no sense.

Let's just accept here that a class cannot be changed by Javassist once it has been already loaded by the Classloader.

3.2.1 Overcoming the Linkage problem with Java Agents

The only (easy) way to overcome this problem is to change the class implementation using bytecode manipulation before the class is loaded by any Classloader. And happily, as often in the Java World, the JVM provides a mechanism for this, the JPLIS - Java Programming Language Instrumentation Services - and the concept of Java Agent.

In its essence, a Java agent is a regular Java class which follows a set of strict conventions. The agent class must implement a public static void premain(String agentArgs, Instrumentation inst) method which becomes an agent entry point (similar to the main method for regular Java applications).

Once the Java Virtual Machine (JVM) has initialized, each such premain(String agentArgs, Instrumentation inst) method of every agent will be called in the order the agents were specified on JVM start. When this initialization step is done, the real Java application main method will be called.

The instrumentation capabilities of Java agents are truly unlimited. Most noticeable one is the ability to redefine classes at run-time. The redefinition may change method bodies, the constant pool and attributes. The redefinition must not add, remove or rename fields or methods, change the signatures of methods, or change inheritance.

Please notice that re-transformed or redefined class bytecode is not checked, verified and installed just after the transformations or redefinitions have been applied. If the resulting bytecode is erroneous or not correct, the exception will be thrown and that may crash JVM completely.

3.2.2 ClassFileTransformer

A Java agent premain method takes the Instrumentation entry point - class java.lang.instrument.Instrumentation - as argument.

The Instrumentation entry point provides several commodity methods to check the possibilities of the JVM but the most important API of the java.lang.instrument.Instrumentation class is the method void addTransformer(ClassFileTransformer transformer); that enable the developer to register several java.lang.instrument.ClassFileTransformer.

The java.lang.instrument.ClassFileTransformer interface defines one single method byte[] transform(...) that is responsible to apply transformations (as far as complete rewriting if required) the java classes being loaded by the JVM.
The transform(...) method is called for each and every class being loaded by a classloader. Both the class being loaded and the classloader actually loading it as well as other information are given in argument.

The transform(...) method is the ideal place where bytecode manipulation libraries can be used to modify classes just before they are loaded by the classloader.

(Source : http://www.barcelonajug.org/2015/04/java-agents.html)

3.2.3 Caution

As a sidenote, the Java agent class may also have a public static void agentmain(String agentArgs, Instrumentation inst) method which is used when the agent is started after JVM startup.

A common practice when developing agents is to implement both the agentmain and premain methods and delegate one to the other. See implementation of Java Agent of BCG below.

3.2.4 Simple Example

package ch.niceideas.common.agent;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.security.ProtectionDomain;

public class ClassLoadingLoggingAgent {

    public static void premain(String agentArgument, Instrumentation instrumentation){     
        System.out.println("Hello, Agent [ " + agentArgument + " ]");

        instrumentation.addTransformer (new ClassFileTransformer() {

            @Override
            public byte[] transform(
                    ClassLoader loader, 
                    String className, 
                    Class<?> classBeingRedefined, 
                    ProtectionDomain protectionDomain, 
                    byte[] classfileBuffer) throws IllegalClassFormatException {

                // Transform is called just before class loading occurs :-)
                System.out.println("Class being loaded : " + className);

                // No transformation ...
                return classfileBuffer;
            }
        });
    }
}

Let's now see how to invoke a simple program using this simple agent.

3.2.5 Invoking the Agent

When running from the command line, the Java agent could be passed to JVM instance using -javaagent argument which has following semantic -javaagent:<path-to-jar>[=options].

A java agent needs to be packaged in a jar file and that jar file needs to have a specific and proper MANIFEST.MF file indicating the class containing the premain method.

A proper manifest file for the agent above should be packages within the jar archive containing the agent classes under META-INF/MANIFEST.MF and would be as follows:

Manifest-Version: 1.0
Premain-Class: ch.niceideas.common.agent.ClassLoadingLoggingAgent

Now let's imagine we invoke our agent on a simple program defined as follows:

package ch.niceideas.common.enhancer;

public class TestMain {

    public static void main (String args[]) {
        System.out.println ("Program Main");
    }
}

Sample result on this simple program is as follows:

badtrash@badbook:/data/work/niceideas-commons/target/test-classes$ java \
    -javaagent:/home/badtrash/ClassLoadingLoggingAgent.jar=007 \
    ch.niceideas.common.enhancer.TestMain

Hello, Agent [ 007 ]
Class being loaded : java/lang/invoke/MethodHandleImpl
Class being loaded : java/lang/invoke/MethodHandleImpl$1
Class being loaded : java/lang/invoke/MethodHandleImpl$2
Class being loaded : java/util/function/Function
Class being loaded : java/lang/invoke/MethodHandleImpl$3
Class being loaded : java/lang/invoke/MethodHandleImpl$4
Class being loaded : java/lang/ClassValue
Class being loaded : java/lang/ClassValue$Entry
Class being loaded : java/lang/ClassValue$Identity
Class being loaded : java/lang/ClassValue$Version
Class being loaded : java/lang/invoke/MemberName$Factory
Class being loaded : java/lang/invoke/MethodHandleStatics
Class being loaded : java/lang/invoke/MethodHandleStatics$1
Class being loaded : sun/launcher/LauncherHelper
Class being loaded : java/util/concurrent/ConcurrentHashMap$ForwardingNode
Class being loaded : sun/misc/URLClassPath$FileLoader$1
Class being loaded : java/lang/Package
Class being loaded : java/io/FileInputStream$1
Class being loaded : ch/niceideas/common/enhancer/TestMain
Class being loaded : sun/launcher/LauncherHelper$FXHelper
Class being loaded : java/lang/Class$MethodArray
Class being loaded : java/lang/Void
Program Main
Class being loaded : java/lang/Shutdown
Class being loaded : java/lang/Shutdown$Lock

3.2.6 Workaround

As a sidenote, and to conclude this section, let's just mention that using a java agent to inject behaviour at runtime using bytecode manipulation is not always a requirement, it depends eventually on the use case.
A pretty common approach favored over java agents usage is the subclassing approach. It consists in defining a new class as a subclass of the class to be enhanced and injecting the new behaviour to that subclass instead.
This is a pretty straightforward approach and prevents the usage of a java agent since we don't care whether or not the initial class has already be loaded. Since we define a new class, the subclass, we're good to go no matter what happens with the initial class.

I have given an example of this approach in my previous article as described here.

In the case of boilerplate code generation such as done by Lombok, using an agent is pretty much the only way. The Value Objects or Java Beans enhanced this way can well be used later by the running program for ORM concerns or IoC concerns. These other frameworks, such as hibernate or spring, very often use the subclassing approach to inject their own behaviour.
If a programmer attempts to use the subclass trick to inject his own behaviour and then use his class with hibernate for instance, his changes will likely be ignored by hibernate or spring that will generate their own subclass (using CGLIB or Javassist) that can conflict the developer's own subclass. In this case enhancing the class itself is a way simpler approach.

Finally, using a java agent is a convenient way to avoid situations where the developer attempts to enhance a class that has already been loaded by the classloader. But that is not necessarily always required and the developer can well choose to implement his own control over the application lifecycle to ensure he has the chance to modify classes before the application attempts to load them.
But that is impossible when using annotations of course and hence frameworks such as Lombok using annotations extensively have no other choice than using a java agent.

4. BCG: a simple approach for generating boilerplate code using Javassist

Now back on the Javassist topic.

The purpose of this article is to give a second example of a sound Javassist use case: the generation of boilerplate code using bytecode manipulation, just as project Lombok is doing.
In fact, I will present here the BCG tool that mimic Lombok and re-implement two features of the Lombok feature set.

I am presenting here the few dozen lines of code on the BCG tool - BCG for Boilerplate Code Generator.
BCG is a simple tool that uses Javassist and implement a Java agent to key Lombok features:

toString() method generation
property getters and setters generation

Note that BCG is not a production tool or anything like it, it is really just a Javassist example and intended to demonstrate how straightforward, simple and efficient it would be to re-implement Lombok features using Javassist ... Should one want to do that, which is not likely since Lombok is working so cool and so easily extendable.

As mentioned above, we will only be mimicking project Lombok here using bytecode manipulation. We are not implementing these features the same way Lombok is doing. Lombok is working at compile-time using AST Transformation. We will be working at runtime using bytecode manipulation.

4.1 Principle

We want to be able to implement transformers that take care of performing one specific modification to target classes and activated by the presence of one specific annotation on these classes.

The key idea is to implement a Java Agent that analyze each and every class just before is loaded by the classloader and verifies if this class needs to be transformed.
We want to implement Transformers that recognize classes declaring a specific annotation and proceed with the transformation of these classes.
We want the system to be easily extendable with new transformers.

4.2 Design

The design of BCG is as follows:

Principal components are as follows:

EnhancerAgent : This is a JVM Agent that implements classes transformation from recognized annotations.
The EnhancerAgent is called before the application starts and registers a java.lang.instrument.ClassFileTransformer that enhances classes declaring specific annotations before they are loaded by classloader(s).
The java.lang.instrument.ClassFileTransformer here is a simple anonymous adapter.
ClassTransformer : This is an interface implemented by actual Class Transformers. A Class Transformer transforms Java classes declaring the recognized annotation(s) using Javassist.
AbstractTransformer : This is the base class for all Transformers. It provides commodity routines for ClassTransformers and simplifies registration API.

Then all actual transformers extend AbstractTransformer and simply declare the annotation they recognize.

4.3 Implementation

The source code of all classes and interfaces from the design above is given below.

(In all snippets of code from now on, I will be coloring relevant Javassist API calls in dark red)

4.3.1 The code of the Agent

[Class EnhancerAgent]

package ch.niceideas.common.enhancer;

import ch.niceideas.common.enhancer.impls.CountInstanceTransformer;
import ch.niceideas.common.enhancer.impls.DataTransformer;
import ch.niceideas.common.enhancer.impls.ToStringTransformer;
import javassist.CannotCompileException;
import javassist.ClassPool;
import javassist.CtClass;

import java.io.IOException;
import java.lang.annotation.Annotation;
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.security.ProtectionDomain;
import java.util.ArrayList;
import java.util.List;

/**
 * This is a JVM Agent that implements classes transformation from recognized
 * annotations.
 * <p />
 *
 * The EnhancerAgent is called before the application starts and registers a
 * <code>java.lang.instrument.ClassFileTransformer</code> that enhances
 * classes declaring specific annotations before they are loaded
 * by classloader(s).
 * <p />
 *
 * The <code>java.lang.instrument.ClassFileTransformer</code> here is a simple
 * anonymous adapter.
 */
public class EnhancerAgent {

    private static ClassTransformer[] transformers = null;

    // for now I don't have any better way than declaring all transformers here
    static {
        transformers = new ClassTransformer[] {
                new CountInstanceTransformer(),
                new ToStringTransformer(),
                new DataTransformer()
        };
    }

    // Java Agent API
    public static void premain(String agentArgs, Instrumentation inst) {
        agentmain (agentArgs, inst);
    }

    // API used when agent invoked after JVM Startup
    public static void agentmain(String agentArgs, Instrumentation inst) {

        inst.addTransformer(new ClassFileTransformer() {

            @Override
            public byte[] transform (
                    ClassLoader loader,
                    String className,
                    Class<?> classBeingRedefined,
                    ProtectionDomain protectionDomain,
                    byte[] classfileBuffer)
                    throws IllegalClassFormatException {

                // Can return null if no transformation is performed
                byte[] transformedClass = null;

                CtClass cl = null;
                ClassPool pool = ClassPool.getDefault();
                try {
                    cl = pool.makeClass(new java.io.ByteArrayInputStream(classfileBuffer));

                    for (ClassTransformer transformer : transformers) {

                        if (transformer.accepts (cl)) {
                            transformer.transform (cl);
                            System.out.println ("Transformed class " + cl.getName()
                                    + " with " + transformer.getClass().getSimpleName());
                        }
                    }

                    // Generate changed bytecode
                    transformedClass = cl.toBytecode();

                } catch (IOException | CannotCompileException e) {
                    e.printStackTrace();

                } finally {
                    if (cl != null) {
                        cl.detach();
                    }
                }

                return transformedClass;
            }
        });
    }
}

4.3.2 Interface ClassTransformer

Transformers implement this interface:

[Class ClassTransformer]

package ch.niceideas.common.enhancer;

import javassist.CtClass;

/**
 * A class Transformer transforms Java classes declaring the recognized
 * annotation(s) using Javassist.
 */
public interface ClassTransformer {

    /**
     * Used by the EnhancerAgent to know whether this class accepts the supported anotation
     *
     * @param cl the class to test
     * @return true if the passed annotation is accepted
     */
    boolean accepts(CtClass cl);

    /**
     * Proceed with the transformation of the javassist loaded class given as argument
     *
     * @param cl the javassist loaded class to be transformed
     */
    void transform(CtClass cl);

}

4.3.3 Common Abstraction

And an abstract class provides some binding commodities to transformers

[Class AbstractTransformer]

package ch.niceideas.common.enhancer.impls;

import ch.niceideas.common.enhancer.ClassTransformer;
import javassist.CtClass;

import java.lang.annotation.Annotation;

/**
 * Base class for ClassTransformers.
 * <br />
 * Provides commodity routines for ClassTransformers and simplifies registration API.
 */
public abstract class AbstractTransformer implements ClassTransformer {

    @Override
    public final boolean accepts(CtClass cl) {
        return cl.hasAnnotation(getAnnotationClass());
    }

    /**
     * Classes that wants to be transformed by this transformer needs to declare
     * this annotation.
     *
     * @return the type of the annotation accepted by this transformer.
     */
    protected abstract Class<? extends Annotation> getAnnotationClass();

    @Override
    public abstract void transform(CtClass cl);
}

4.3.4 The set of Class Transformers

A first Class Transformer : outputs the count of instances of classes declaring the @CountInstance annotation.

[Class CountInstanceTransformer]

package ch.niceideas.common.enhancer.impls;

import ch.niceideas.common.enhancer.ClassTransformer;
import ch.niceideas.common.enhancer.annotations.CountInstance;
import javassist.CannotCompileException;
import javassist.CtBehavior;
import javassist.CtClass;
import javassist.CtField;

import java.lang.annotation.Annotation;

/**
 * This transformer accepts classe declaring the "@CountInstance" annotation.
 * 

 * It enhances the class with an instanceCounter and outputs the value of ths
 * instanceCounter everytime an instance is built.
 */
public class CountInstanceTransformer extends AbstractTransformer 
        implements ClassTransformer {

    @Override
    protected Class<? extends Annotation> getAnnotationClass() {
        return CountInstance.class;
    }

    @Override
    public void transform(CtClass cl) {
        try {
            if (!cl.isInterface()) {

                // Add a static field in the class
                CtField field = CtField.make("private static long _instanceCount;", cl);
                cl.addField(field);

                CtBehavior[] constructors = cl.getDeclaredConstructors();
                for (int i = 0; i < constructors.length; i++) {

                    // Increment counter and output it
                    constructors[i].insertAfter("_instanceCount++;");
                    constructors[i].insertAfter("System.out.println(\""
                            + cl.getName() + " : \" + _instanceCount);");
                }
            }

        } catch (CannotCompileException e) {
            e.printStackTrace();
            throw new RuntimeException (e);
        }
    }
}

The CountInstanceTransformer accepts classes declaring the CountInstance annotation :

package ch.niceideas.common.enhancer.annotations;

/**
 * Classes declaring this annotation will have an instancecounter which value is
 * output everytime an instance is constructed
 */
public @interface CountInstance {
}

Second Class Transformer: generates getters and setters for classes declaring the @Data annotation.

[Class DataTransformer]

package ch.niceideas.common.enhancer.impls;

import ch.niceideas.common.enhancer.ClassTransformer;
import ch.niceideas.common.enhancer.annotations.Data;
import javassist.*;

import java.lang.annotation.Annotation;

/**
 * This transformer accepts classes declaring the "@Data" annotation.
 * <br />
 * It generates a getters and setters dynamically for every field of the class
 * if they do not already exist
 */
public class DataTransformer extends AbstractTransformer implements ClassTransformer {

    @Override
    protected Class<? extends Annotation> getAnnotationClass() {
        return Data.class;
    }

    @Override
    public void transform(CtClass cl) {
        try {
            if (!cl.isInterface()) {

                for (CtField field : cl.getDeclaredFields()) {

                    String camelCaseField = field.getName().substring(0, 1).toUpperCase()
                            + field.getName().substring(1);

                    if (!hasMethod("get" + camelCaseField, cl)) {
                        cl.addMethod(CtNewMethod.getter("get" + camelCaseField, field));
                    }

                    if (!hasMethod("set" + camelCaseField, cl)) {
                        cl.addMethod(CtNewMethod.setter("set" + camelCaseField, field));
                    }
                }
            }

        } catch (CannotCompileException e) {
            e.printStackTrace();
            throw new RuntimeException(e);
        }
    }

    /** javassist has unfortunately no hasMethod API */
    private static boolean hasMethod (String methodName, CtClass cl) {
        try {
            cl.getDeclaredMethod(methodName);
            return true;
        } catch (NotFoundException e) {
            return false;
        }
    }
}

The DataTransformer accepts classes declaring the Data annotation :

package ch.niceideas.common.enhancer.annotations;

/**
 * Classes declaring this annotation will have getters and setters automatically
 * generated
 */
public @interface Data {
}

Third Class Transformer : generates the toString method for classes declaring the @ToString annotation.

[Class DataTransformer]

package ch.niceideas.common.enhancer.impls;

import ch.niceideas.common.enhancer.ClassTransformer;
import ch.niceideas.common.enhancer.annotations.ToString;
import javassist.*;

import java.lang.annotation.Annotation;

/**
 * This transformer accepts classes declaring the "@ToString" annotation.
 * <br />
 * It generates a toString method dynamically. The toString method is generated using
 * bytecode manipulation and avoids reflection.
 */
public class ToStringTransformer extends AbstractTransformer implements ClassTransformer {

    @Override
    protected Class<? extends Annotation> getAnnotationClass() {
        return ToString.class;
    }

    @Override
    public void transform(CtClass cl) {
        try {
            if (!cl.isInterface()) {

                StringBuilder bb = new StringBuilder("{\n");
                bb.append("    StringBuilder sb = new StringBuilder(\"" 
                        + cl.getName() + "\");\n");
                bb.append("    sb.append(\"[\");\n");

                for (CtField field : cl.getDeclaredFields()) {
                    field.setModifiers(Modifier.PUBLIC); // hacky hack

                    bb.append("    sb.append(\"" + field.getName() + "\");\n");
                    bb.append("    sb.append(\"=\");\n");
                    bb.append("    sb.append(this." + field.getName() + ");\n");
                    bb.append("    sb.append(\" \");\n");
                }

                bb.append("    sb.append(\"]\");\n");
                bb.append("    return sb.toString();\n");
                bb.append("}");

                try {
                    CtMethod toStringMethod = cl.getDeclaredMethod("toString");
                    toStringMethod.setBody(bb.toString());

                } catch (NotFoundException e) {

                    CtMethod newMethod = CtNewMethod.make("public String toString() \n"
                            + bb.toString(), cl);
                    cl.addMethod(newMethod);
                }
            }

        } catch (CannotCompileException e) {
            e.printStackTrace();
            throw new RuntimeException (e);
        }
    }
}

The ToStringTransformer accepts classes declaring the ToString annotation :

package ch.niceideas.common.enhancer.annotations;

/**
 * Classes declaring this annotation will have a toString method generated
 * automagically
 */
public @interface ToString {
}

4.3.5 Test Class Example

A test class for the DataTransformer for instance, with the usage of a nice library to test agents: ElectronicArts AgentLoader.

EA Agent Loader is a collection of utilities for java agent developers. It allows programmers to write and test their java agents using dynamic agent loading (without using the -javaagent jvm parameter).

package ch.niceideas.common.enhancer;

import ch.niceideas.common.enhancer.testData.TestData;
import com.ea.agentloader.AgentLoader;
import junit.framework.TestCase;
import org.apache.log4j.Logger;
import org.junit.Before;
import org.junit.Test;

import java.lang.reflect.Method;

public class DataTest extends TestCase{

    private static final Logger logger = Logger.getLogger(DataTest.class);

    @Before
    public void setUp() throws Exception {
        AgentLoader.loadAgentClass(EnhancerAgent.class.getName(), "");
    }

    @Test
    public void testDataTransformer() throws Exception {

        TestData testData = new TestData();

        Method getI = TestData.class.getDeclaredMethod("getI");
        assertEquals (0, getI.invoke(testData));

        Method getMyString = TestData.class.getDeclaredMethod("getMyString");
        assertEquals ("abc", getMyString.invoke(testData));

        Method getValue = TestData.class.getDeclaredMethod("getValue");
        assertEquals (-1L, getValue.invoke(testData));

        Method setI = TestData.class.getDeclaredMethod("setI", int.class);
        setI.invoke(testData, 9);
        assertEquals (9, getI.invoke(testData));

        Method setMyString = TestData.class.getDeclaredMethod("setMyString", String.class);
        setMyString.invoke(testData, "xyz");
        assertEquals ("xyz", getMyString.invoke(testData));

        Method setValue = TestData.class.getDeclaredMethod("setValue", long.class);
        setValue.invoke(testData, 999L);
        assertEquals (999L, getValue.invoke(testData));
    }
}

This test case uses the following test data :

package ch.niceideas.common.enhancer.testData;

import ch.niceideas.common.enhancer.annotations.Data;

/**
 * a test Data for the DataTest test case
 */
@Data
public class TestData {

    private int i = 0;

    private String myString = "abc";

    private long value = -1;
}

5. Conclusion

Again. Bytecode manipulation opens a whole lot of new possibilities for the JVM and is key to address one of the biggest weakness of the JVM: the overwhelming verbosity of the language. The approaches and techniques presented above are extensively used by so many libraries and frameworks that have become the facto standard nowadays: AspectJ, Spring, Hibernate, Jprofiler, etc.
For that reason, and because it's really a lot of fun, one might find bytecode manipulation a pretty valuable mechanism to master.

In addition, even though Lombok uses for very good reasons a different technique (AST Transformation), I find it astonishing to see how bytecode manipulation enables a developer to mimic its features in so few lines of code.

One can use bytecode manipulation to perform many tasks that would be difficult or impossible to do otherwise, and once one learns it, the sky is the limit.

Javassist put this power in hands of every Java developer in a simple, intuitive and efficient way.

Part of this article is available as a slideshare presentation here: https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit.

Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code

Jerome Kehrli — Mon, 13 Feb 2017 15:30:33 -0500

Java bytecode is the form of instructions that the JVM executes.
A Java programmer, normally, does not need to be aware of how Java bytecode works.

Understanding the bytecode, however, is essential to the areas of tooling and program analysis, where the applications can modify the bytecode to adjust the behavior according to the application's domain. Profilers, mocking tools, AOP, ORM frameworks, IoC Containers, boilerplate code generators, etc. require to understand Java bytecode thoroughly and come up with means of manipulating it at runtime.
Each and every of these advanced features of what is nowadays standard approaches when programming with Java require a sound understanding of the Java bytecode, not to mention completely new languages running on the JVM such as Scala or Clojure.

Bytecode manipulation is not easy though ... except with Javassist.
Of all the libraries and tools providing advanced bytecode manipulation features, Javassist is the easiest to use and the quickest to master. It takes a few minutes to every initiated Java developer to understand and be able to use Javassist efficiently. And mastering bytecode manipulation, opens a whole new world of approaches and possibilities.

The goal of this article is to present Javassist in the light of a concrete use case: the implementation in a little more than 300 lines of code of a lightweight, simple but cute IoC Container: SCIF - Simple and Cute IoC Framework.

A new version of comet-tennis demo app with the SCIF framework integrated is available here.

Part of this article is available as a slideshare presentation here: https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit.

You might also want to have a look at the second article in this serie available here : Bytecode manipulation with Javassist for fun and profit part II: Generating toString and getter/setters using bytecode manipulation.

Summary

1. Introduction / Purpose
- 1.1 Runtime Reflection
- 1.2 Bytcode manipulation
2. Javassist
- 2.1 Javassist prupose and behaviour
- 2.2 A gentle example with Javassist
3. IoC
4.0 SCIF : Simple and Cute IoC Framework
5. Conclusion

1. Introduction / Purpose

Bytecode manipulation consists in modifying the classes - represented by bytecode - compiled by the Java compiler, at runtime. It is used extensively for instance by frameworks such as Spring (IoC) and Hibernate (ORM) to inject dynamic behaviour to Java objects at runtime.
But first let's look at a very summarized view of the Java toolchain to remind a few concepts:

Java source files are compiled to Java class files by the Java Compiler. These Java classes take the form of bytecode. This bytecode is loaded by the JVM to execute the Java program.
In principle the bytecode is read only and cannot be change once loaded. That is true, but:

The java classes bytecode can be modified before being loaded by the classloader through the usage of an agent
Classes bytecode can be modified at runtime without an agent as long as the class has not been loaded yet by a classloader.
Classes can be generated entirely dynamically at runtime using bytecode manipulation techniques

In this article we'll dig into the library Javassist which is a bytecode manipulation framework that can help in achieving all of the above mechanisms.
But before, let's describe three different, unrelated but complementary techniques: Introspection, Reflection and Bytecode Manipulation.

1.1 Runtime Reflection

Runtime reflection is the ability of a computer program to examine, introspect, and modify its own structure and behavior at runtime.

Reflection is commonly used by programs which require the ability to examine or modify the runtime behavior of applications running in the Java virtual machine.
Reflection is a powerful technique and can enable applications to perform operations which would otherwise be impossible.

The ability to examine and manipulate a Java class from within itself may not sound like very much, but in other programming languages this feature simply doesn't exist. For example, there is no way in a Pascal, C, or C++ program to obtain detailed information about the functions defined within that program.

In java there is no specific introspection API available natively. Introspection is performed as well using the Java Reflection API.

Still, conceptually, introspection and reflection are different things:

Type introspection is the ability of a program to examine the type or properties of an object at runtime.
This is short example of Type Introspection in Java, where we discover the fields of an object and show their values dynamically. Again, Introspection in Java is really done using the Reflection API

import java.lang.reflect.Field;

public class TestIntrospection {

    public static class TestData {
        private int i = 0;
        private String myString = "abc";
        private long value = -1;
    }

    // test Introspection
    public static void main (String[] args) {
        try {
            // Using Introspection, we really don't care of the actual type
            Object td = new TestData();

            // List fields of TestData and get their values
            for (Field field : td.getClass().getDeclaredFields()) {
                field.setAccessible(true); // just make private fields accessible

                System.out.println (field.getName() + "=" + field.get(td));
            }
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        }
    }
}

Runtime Reflection is a native feature in the Java programming language. It allows an executing Java program to examine or "introspect" upon itself, and manipulate internal properties of the program.
A short example could be as follows, where we change the value of a field using reflection:

import java.lang.reflect.Field;

public class TestReflection {

    public static class TestData {
        private int i = 0;
        private String myString = "abc";
        private long value = -1;
    }

    // test Reflection
    public static void main (String[] args) {
        try {
            // Using Reflection, we really don't care of the actual type
            Object td = new TestData();

            // Change the value of the field myString
            Field myStringField =  td.getClass().getDeclaredField("myString");
            myStringField.setAccessible(true); // just make private fields accessible
            myStringField.set(td, "xyz");

            System.out.println (myStringField.getName() + "=" + myStringField.get(td));

        } catch (NoSuchFieldException | IllegalAccessException e) {
            e.printStackTrace();
        }
    }
}

Runtime Reflection is a very powerful feature of the JVM.
I wrote a previous article on this very blog showing how to dynamically add values to an Enum Type in Java using Runtime Reflection.

Why are those important in the scope of bytecode manipulation and Javassist ?

Runtime Reflection is important in our context for two reasons:

First, because Javassist attempts to keep an API as close as possible to the Java Runtime Reflection API as a way to appear as natural as possible to Java developers.
Second, and this is maybe more important, because behaviour injected in Java Classes using bytecode manipulation is not known by the compiler. Thus, it is sometimes only available through runtime reflection.

1.2 Bytcode manipulation

Bytecode manipulation allows the developer to express instructions in a format that is directly understood by the Java Virtual Machine, without passing from source code to bytecode through compiler.
Bytecode is somewhat similar to assembly code directly interpretable by the CPU. But with Java the bytecode is, first, interpreted by a Virtual Machine, the JVM, and second much more understandable that assembly code.

One might wonder why one would want to get interested in bytecode manipulation and generation. As a matter of fact, every java developer has likely already been using bytecode manipulation all over the place without knowing it.
Since the JVM can modify bytecode and use new bytecode while it is running, this generates a whole new universe of languages and tools that by far surpasses the initial intent of the Java language.

Bytecode manipulation use cases

Some examples are:

ORM frameworks such as Hibernate use bytecode manipulation to inject, for instance, relationship management code (lazy loading, etc.) inside mapped entities.
FindBugs inspects bytecode for static code analysis
Languages like Groovy, Scala, Clojure generate bytecode from different source code.
IoC frameworks such as Spring use it to seamlessly weave your application lifecycle together
language extensions like AspectJ can augment the capabilities of Java by modifying the classes that the Java compiler generated
etc.

The Java platform provides you with many ways to work with bytecode, for instance:

One can write his own compiler for any kind of new and crazy language
One can generate on the fly sub-classes of already loaded classes and use them instead of original classes to get additional behaviour
One can write an instrumentation agent that plugs right into the JVM and modifies behaviour of classes before they are loaded by the classloader
etc.

With so many options available, one of these will certainly fit any experiment that one wants to play around with. With bytecode manipulation, one really gets the whole power of the JVM for free and the capacity to slot in any idea exactly where it's needed while reusing the rest of the Java platform.

From my perspective, this is what excites me the most, as a developer I can really focus on my crazy idea that is not supported by the Java language and I don't have to write an entire platform to make it come to life.
Certainly this has been one of the key areas why the Java community has constantly been experimenting with new ways to push the programming toolset further.

This article wont present the details of the Java bytecode any further. We'll focus instead on high level libraries aimed at manipulating the Java Bytecode.
Should you be interested in the low level details, I can only recommend that you read this excellent paper from ZeroTunaround, the guys behind JRebel.

Most common bytecode manipulation libraries

As a matter of fact, while Runtime Reflection is supported natively by the JVM, Bytecode manipulation, on the other hand, is fairly difficult to achieve without the usage of a specific library

The most common bytecode manipulation libraries in Java are as follows:

ASM s a project of the OW2 Consortium. It provides a simple API for decomposing, modifying, and recomposing binary Java classes. ASM exposes the internal aggregate components of a given Java class through its visitor oriented API. ASM also provides, on top of this visitor API, a tree API that represents classes as object constructs. Both APIs can be used for modifying the binary bytecode, as well as generating new bytecode
BCEL provides a simple library that exposes the internal aggregate components of a given Java class through its API as object constructs (as opposed to the disassembly of the lower-level opcodes). These objects also expose operations for modifying the binary bytecode, as well as generating new bytecode (via injection of new code into the existing code, or through generation of new classes altogether).
CGLIB is a powerful, high performance and quality Code Generation Library, it is used to extend JAVA classes and implements interfaces at runtime. CGLIB is really oriented towards implementing new classes at runtime, as opposed to modifying existing bytecode such as other libraries.
Javassist is a Java library providing a means to manipulate the Java bytecode of an application. In this sense Javassist provides the support for structural reflection, i.e. the ability to change the implementation of a class at run time.

Javassist is much easier to use that lower level libraries such as BCEL or ASM. It is also less limited and more powerful than CGLIB.

This chart shows in addition the AspectJ framework, as a way to have the user get an understanding of the level of abstraction provided by these tools:

Now, as you might have guessed from its title, this rest of this article will focus on Javassist.

2. Javassist

From the Javassist web site:

"Javassist (Java Programming Assistant) makes Java bytecode manipulation simple. It is a class library for editing bytecode in Java; it enables Java programs to define a new class at runtime and to modify a class file when the JVM loads it.
Unlike other similar bytecode editors, Javassist provides two levels of API: source level and bytecode level. If the users use the source-level API, they can edit a class file without knowledge of the specifications of the Java bytecode.
The whole API is designed with only the vocabulary of the Java language. You can even specify inserted bytecode in the form of source text; Javassist compiles it on the fly. On the other hand, the bytecode-level API allows the users to directly edit a class file as other editors."

The fact that Javassist is presented above as being able to modify classes at loading time is not a limitation of the Javassist framework itself, but rather a consequence from the linking system of the JVM. Once a class has already been loaded, changing it would result in a Linkage Error (unless the JVM is launched with the JPDA [Java Platform Debugger Architecture] enabled, which would make a class dynamically reloadable, but that is another story).
Interestingly, Javasssist is perfectly able to modify a class long after the application has started as long as that specific class has not been loaded.
This is just to emphasize that Javassist can perfectly be used to modify classes at runtime and not only at "pre-main" time by the usage of a JVM agent, even though this suffers from great constraints.

In my opinion, the great strength of Javassist over its competitors is that Javassist enables the user to generate bytecode on the fly from actual Java code given to it in the form of a string by calling the Java Compiler on the fly on such strings.
And that is freaking awesome.

2.1 Javassist purpose and behaviour

Javassist provides the developer with a high level API around classes, methods, fields, etc. aimed at making it as easy as possible to change the implementation of existing classes or even implement completely new classes, dynamically, at runtime, using bytecode manipulation.

(The following is explained in more details in the official Javassist tutorial.)

The most important elements of the Javassist API are presented on the schema below:

The class Javassist.CtClass is an abstract representation of a class file. A CtClass (compile-time class) object is a handle for dealing with a class file. The following program is a very simple example:

(In all examples of code from now on, I will be coloring relevant Javassist API calls in dark red)

ClassPool pool = ClassPool.getDefault();
CtClass cc = pool.get("test.Rectangle");
cc.setSuperclass(pool.get("test.Point"));
cc.writeFile();

ClassPool

This program first obtains a ClassPool object, which controls bytecode modification with Javassist. The ClassPool object is a container of CtClass object representing a class file. It reads a class file on demand for constructing a CtClass object and records the constructed object for responding later accesses.

To modify the definition of a class, the users must first obtain from a ClassPool object a reference to a CtClass object representing that class.get() in ClassPool is used for this purpose.
In the case of the program shown above, the CtClass object representing a class test.Rectangle is obtained from the ClassPool object and it is assigned to a variable cc. The ClassPool object returned by getDefault() searches the default system search path.

CtClass

The CtClass object obtained from a ClassPool object can be modified.
In the example above, it is modified so that the superclass of test.Rectangle is changed into a class test.Point. This change is reflected on the original class file when writeFile() in CtClass() is finally called.

writeFile() translates the CtClass object into a class file and writes it on a local disk. Javassist also provides a method for directly obtaining the modified bytecode. To obtain the bytecode, call toBytecode():

byte[] b = cc.toBytecode();

(Bear in mind that this is especially useful when implementing a Java agent)

You can directly load the CtClass as well:

Class clazz = cc.toClass();

A class can be returned to the pool, making it available to the classloader and hence the whole application:

pool.toClass(cc, Thread.currentThread().getContextClassLoader(), null);

Defining a new class

To define a new class from scratch, makeClass() must be called on a ClassPool.

ClassPool pool = ClassPool.getDefault();
CtClass cc = pool.makeClass("Circle");
cc.setSuperclass(pool.get("test.Point"));

This program defines a class Circle including no members except those inherited by the parent class Point. Member methods of Circle can afterwards be created with factory methods declared in CtNewMethod and appended to Circle with addMethod() in CtClass.
makeClass() cannot create a new interface; makeInterface() in ClassPool can do. Member methods in an interface can be created with abstractMethod() in CtNewMethod. Note that an interface method is an abstract method.

Implementing / Modifying a class

Methods are represented by CtMethod objects. CtMethod provides several methods for modifying the definition of the method. Note that if a method is inherited from a super class, then the same CtMethod object that represents the inherited method represents the method declared in that super class. A CtMethod object corresponds to every method declaration.
Constructors are represented by their very own type in Javassist : CtConstructor. Both CtMethod and CtConstructor extends the same base class and have a lot of their API in common.

Javassist does not allow to remove a method or field, but it allows to change the name. So if a method is not necessary any more, it should be renamed and changed to be a private method by calling setName() and setModifiers() declared in CtMethod, for instance to hide it. But beware of linkage errors at runtime if you mess with a method used by another class.

CtMethod and CtConstructor can be used to completely implement / rewrite a constructor or a method from scratch. They also provide methods insertBefore(), insertAfter(), and addCatch(). They are used for inserting a code fragment into the body of an existing method.

When implementing or rewriting completely a method from scratch, using CtNewMethod.make() is in my opinion the most convenient approach. It enables the developer to implement a method by providing Java Source Code syntax in a simple string.
For instance:

CtClass point = ClassPool.getDefault().get("Point");
CtMethod m = CtNewMethod.make(
        "public int xmove(int dx) { x += dx; }",
        point);
point.addMethod(m);

CtNewMethod provides a lot of high level methods for implementing getters, setters and other commodity methods directly, sometimes even without having to bother providing an implementation on your own.

Some pretty complete information in this regards is available on the Second official javassist tutorial.

2.2 A gentle example with Javassist

We'll see now a simple and yet complete example using Javassist. We'll implement the getters and setters for the fields of the class TestData introduced here using bytecode manipulation.
Then we'll test the getter and setter for the field myString. Since these getters and setters are injected using bytecode manipulation at runtime, we'll have to use runtime reflection to call them:

(Reminder, I am coloring relevant Javassist API calls in dark red)

import javassist.*;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

public class TestJavassist {

    public static class TestData {
        private int i = 0;
        private String myString = "abc";
        private long value = -1;
    }

    // test Javassist
    public static void main (String[] args) {
        try {

            ClassPool cp = ClassPool.getDefault();
            CtClass clazz = cp.get("ch.niceideas.common.utils.TestJavassist$TestData");

            for (CtField field : clazz.getDeclaredFields()) {
                String camelCaseField = field.getName().substring(0, 1).toUpperCase()
                        + field.getName().substring(1);

                // We don't need to mess with implementation here. CtnewMethod has a
                // commodity method to implement a getter directly
                CtMethod fieldGetter = CtNewMethod.getter("get" + camelCaseField, field);
                clazz.addMethod(fieldGetter);

                // Just for the sake of an example, we'll define the setter by actually
                // providing the implementation, not using the commodity method offered
                // by CtNewMethod
                CtMethod fieldSetter = CtNewMethod.make(
                            "public void set" + camelCaseField + " \n" + 
                            "    (" + field.getType().getName() + " param) { \n" +
                            "    this." + field.getName() + " = param; \n" +
                            "}",
                        clazz);
                clazz.addMethod(fieldSetter);
            }

            // Save class and make it available
            cp.toClass(clazz, Thread.currentThread().getContextClassLoader(), null);

            // Now instantiate a new TestData
            TestData td = new TestData();

            // Get the value of the field 'myString' using the newly defined getter
            Method getter =  td.getClass().getDeclaredMethod("getMyString");
            System.out.println (getter.invoke(td));

            // Change the value of field 'myString' using newly defined setter
            Method setter =  td.getClass().getDeclaredMethod("setMyString", String.class);
            setter.invoke(td, "xyz");

            // Get the value again
            System.out.println (getter.invoke(td));

        } catch (  NotFoundException | CannotCompileException | NoSuchMethodException
                 | IllegalAccessException | InvocationTargetException e) {
            e.printStackTrace();
        }
    }
}

3. IoC

OK. Since the example code I want to implement below with Javassist is a simple IoC container, I guess I should present what IoC actually is beforehand.

Inversion of Control is a design pattern related to lifecycle management of components in an application benefiting from a services architecture.
In such an application, business components are usually implemented in the form of various services, such as business services, business managers, DAOs, etc. The main class delegates specific business concerns to business services, which delegate finer aspects in their turn to managers, which further delegate various business of technical aspects to smaller managers, or DAOs, adapters. etc.

These various services need to know about each others to be able to call each others. Managing the construction and instantiation of these services is called components lifecycle management.

Very often, business services are stateless components, not keeping any state in instance variables or else. Traditionally, for a very long time, these stateless services have been implemented as singletons. For a very long time this was a very convenient approach since the main singleton simply needs to get the other singletons it was using, which in turn simply needed to get the other singletons they were using, and so on.
By separating the instantiation of singletons and their initialization in two stages, cycles could be handled easily and everyone was happy.
But with the rise of XP and unit testing, singleton-based applications were suffering from a very critical drawback:

Singletons were enforcing strict dependencies on other service implementation at compile time, making it pretty impossible to replace the dependencies by mock objects or stubs as required for efficiently unit testing a specific service.
Using singletons, testing a specific service often meant to be required to completely build and initialize the whole application, which can well turn into a nightmare.

Java EE was of course not an answer to this problem since it required to have a Java EE container, which is even more of a nightmare (well let's not get me started on Java EE, shall we ?).

Inversion of Control

Inversion of control is initially mostly an answer to this problem, as a way to increase the modularity of the application and making it more extensible, and more importantly testable in an easier way by removing the strict dependencies between components.
The key idea is to delegate the management of the lifecycle of the components and the injection of each component's dependencies to a framework, or rather a container, borrowing the term from Java EE, called here a lightweight container.

Instead of every component getting each other references on the singletons or specific instances by building them, the container takes care of instantiating the components, managing their lifecycle in the required scope, and injecting their dependencies at runtime.
Injecting the dependencies at runtime, with a configurable approach, using a configuration file, annotations or even a dedicated API, opens the possibility to inject a different implementation of a service depending on the context, as long as it respects the required interface.
For instance, injecting a mock object instead of the real deal for unit testing becomes straightforward.

Inversion of Control and Dependency Injection are two different things - and yet strongly related to each other - often confused in some documentation:

Inversion of Control - IoC : is the name of the design pattern, the approach. It is considered a design pattern, which, in my opinion, is wrong! IoC is an architecture pattern. But yeah that is really no big deal.
Dependency Injection - DI : is the name of a technique, a mechanism on which IoC often relies to take place. It consists in injecting the components required by a specific component at runtime, based on some configuration rules. DI is really just one aspect of IoC.

3.1 IoC history

Inversion of Control, as a term, was popularized in 1998 by the Apache Avalon team trying to engineer a "Java Apache Server Framework" for the growing set of server side Java components and tools at Apache.
To the Avalon team, it was clear that components receiving various aspects of component assembly, configuration and lifecycle was a superior design to those components going at getting the same themselves.

Later, the authors of the "Java Open Source Programming" book wrote XWork and WebWork2 to support their forthcoming book. Their concepts were very much like those from IoC/Avalon, but dependencies were passed into the component via setters. The need for those dependencies was declared in some accompanying XML.
That was actually the first IoC framework close to the form they have today.

In 2002, Rod Johnson, leader of the Spring Framework wrote the book "Expert One-on-One : J2EE Design and Development" that also discussed the concepts of setter injection, and introduced the codeline that ultimately became the Spring Framework at SourceForge in February of 2003.
I myself discovered the concept in 2003 when reading that book, following the advise of Mr. Parick Gras, which I take the opportunity to thank a lot for this here.

This whole history is presented in details on PicoContainer / Inversion of Control History and can be represented as follows:

(Source : PicoContainer / Inversion of Control History)

3.2 IoC Principle

As stated above, in a usual application, the lifecycle of components starts with a main component (or class) that either creates the other services it requires or get their singletons.
These other components, in their turn, create or get references on their own dependencies, and so on.

With IoC, a container, called lightweight container - as opposed to Java EE craps that are very heavy (and very bad) containers - takes care of instantiating and managing the lifecycle of the components as well as, more importantly, injecting the dependencies in every component.

3.3 Various frameworks

The most important IoC containers today are the following:

The Spring Framework is an application framework and inversion of control container for the Java platform. The core of spring is really about IoC and components management but nowadays there is a complete ecosystem of tools and side frameworks around spring core aimed at developing web application, ORM concerns, etc.
The Pico Container is a very lightweight IoC Container and only that. Unlike spring, it is designed to remain small and simple and targets only IoC concerns, nothing else. It is not heavily maintained.
Apache Tapestry is an open-source component-oriented Java web application framework conceptually similar to JavaServer Faces and Apache Wicket. It provides IoC concerns in addition to the web application framework.
Google Guice is an open source software framework for the Java platform released by Google. It provides support for dependency injection using annotations to configure Java objects.

4. SCIF : Simple and Cute IoC Framework

The rest of this article is dedicated to present the implementation of a very simple IoC framework using Javassist, rather as a way to illustrate how easy and straightforward that is with Javassist than for any other reason :-)
Implementing Dependency Injection is actually a state-of-the-art use case for Javassist and a nice way to present the possibilities and whereabouts of bytecode manipulation.
We'll see now how to use Javassist in the light of a concrete use case: the implementation in a little more than 300 lines of code of a lightweight, simple but cute IoC Container: SCIF - Simple and Cute IoC Framework.

4.1 Principle

SCIF - the system we want to build - is an MVP (Minimum Viable Product). We want it to implement Dependency Injection in its simplest form:

Services are managed by the framework and stored in a Service Registry
Services should declare the annotation @Service
to be discovered by the framework. The framework searches for services declaring this annotation in the classpath.
Dependencies are identified in services using the annotation @Resource. The framework analyze services to discover about their dependencies at runtime.
If @Resource is declared on a field, the framework injects the dependency directly, at build time.
If @Resource is declared on a getter, the framework uses bytecode manipulation to override the getter in a subclass and implement lazy loading of the dependency.

In case of getter (property) injection instead of field injection, SCIF is forced to generate a sub-class of the initial class and override the getter in that sub-class to implement lazy-loading.
This is a consequence of the usage of an annotation to identify the services to be enhanced: in order for the framework to be able to query the annotation on a class, that class needs unfortunately to be loaded by the Classloader. Javassist is not able to change a class once that class has been loaded (well at least not easily).
Changing a class that is already loaded leads unusually to a linkage error and that is forbidden without usage of very advanced techniques, too difficult for this simple framework.

Example

The code below presents a ServiceA having two dependencies: ServiceB and ServiceC.
The first dependency, ServiceB is declared by ServiceA on the field itself, using the annotation @Resource.
The second dependency, ServiceC is declared on the getter, indicating the will of the developer to benefit from lazy loading.

The example below illustrates in red the code or behaviour that should be injected at runtime by SCIF.

import ch.niceideas.common.service.Registry;
import ch.niceideas.common.service.Service;
import javax.annotation.Resource;

/** A Business Service  */
@Service
public class ServiceA {

    @Resource // here we inject the dependency on the field
    private ServiceB serviceB = service_injected_by_reflection;

    private ServiceC serviceC;

    public ServiceB getServiceB() { return serviceB; }
    public void setServiceB(ServiceB serviceB) { this.serviceB = serviceB; }

    @Resource // here we inject the dependency on the property (getter)
    public ServiceC getServiceC() { return serviceC; }
    public void setServiceC(ServiceC serviceC) { this.serviceC = serviceC; }

    // we want to use javassist to generate a sub-class on the fly, at runtime, to handle
    // the lazy loading of ServiceC at runtime
    public static class javassist_sub extends ServiceA {

        private Registry registry = injected_by_reflection;

        public ServiceC getServiceC() {
            ServiceC retObject = super.getServiceC();
            if (retObject == null) {
                retObject = (ServiceC) registry.getService(
                        ServiceC.class.getCanonicalName());
                super.setServiceC(retObject);
            }
            return retObject;
        }
    }
}

Now let's see what is the design of the SCIF framework enabling this behaviour.

4.2 Design

SCIF is implemented by the following fundamental classes:

Registry : a Registry is a service manager. It stores services following a specific scope passed to the storeService method.
Stored services can be retrieved regardless of the scope. They are searched in the smallest scope first and then in larger scopes.
StaticRegistry : a static registry stores services in a static map.
Only the APPLICATION scope is supported. Attempting to store a service in another scope results in an exception.
RegistryInitializer : the RegistryInitializer is the most important component of SCIF.
It is responsible for:
- Searching the classpath for classes declaring the @Service annotation
- Injecting dependencies in the various form supported by the IoC Framework:
  - Field injection. This is done using simply runtime reflection.
  - Method (getter) injection. This is done by generating dynamically a subclass that takes care of the Lazy Loading of the dependency.
@Service : this annotation identifies services to be searched for in the classpath.
@Resource : this annotation identifies dependencies to be injected either at field level or getter (property) level.

The design is as follows:

Rules regarding @Resource handling

The RegistryInitializer analyzes the classes annotated with @Service and handles @Resource annotations the following way:

If @Resource is declared on a field: in this case there is actually no need for bytecode manipulation. When the RegistryInitializer analyzes the services, it simply injects the reference to the field.
If @Resource is declared on a getter, and a corresponding setter is found: in this case, the system assumes he can use the setter to implement a cache for lazy loading. A subclass is created which overrides the getter to implement lazy loading. When calling the overriden getter, the latest will first call the original getter to see if the result service is already available. If that is the case, that service is returned. if it is not the case, the system will get the target service from the registry and use the corresponding setter to store it before returning it.
If @Resource is declared on arbitrary method or on a getter without a corresponding setter: in this case the system cannot assume he can use the underlying field (through the setter) as a cache, and simply returns the target service from the registry at every call.

The RegistryInitializer uses the following tools / libraries:

ReflectionUtils : a little helper class simplifying some operations related to Runtime Reflection.
Reflections (org.reflection.Reflections) : a mandatory package when it comes to analyzing the classpath looking for classes declaring a certain annotation.
Javassist : the bytecode manipulation library.

4.3 Some focus on code

We'll see below the most important pieces of code of the SCIF Framework.

(Reminder, I am coloring relevant Javassist API calls in dark red)

It all starts with the method RegistryInitializer.init() that takes care of the whole shebang:

/**
 * Initialize a registry from the root package (prefix) given as argument.
 * <br />
 * The RegistryInitializer will search for classes annotated with @Service and add them 
 * to the returned registry.
 *
 * @param rootPackage
 * @return A Registry containing all discovered services
 * @throws RegistryInitializationException in case of any error
 */
public static Registry init(String rootPackage) throws RegistryInitializationException {

    try {
        Reflections reflections = new Reflections(rootPackage);

        Set<Class<?>> annotated = reflections.getTypesAnnotatedWith(Service.class);

        // Create service wrappers
        Map<Class<?>, ServiceWrapper> wrappers = initServiceWrappers(annotated);

        // Build set of methods and fields to be analyzed
        analyzeWrappers(wrappers);

        // Now the complicated part, overwrite getters / methods using Javassist,
        // dynamically creating a subclass with bytecode manipulation
        enhancePropertyGetters(wrappers);

        StaticRegistry registry = new StaticRegistry();

        // Instantiate all the services and store them in Registry 
        // with class name as service name
        initializeRegistry(wrappers, registry);

        // Then do the easy thing : inject service on dependencies expressed on fields
        proceedFieldInjection(wrappers, registry);

        return registry;

    } catch (ReflectiveOperationException | NotFoundException | CannotCompileException e) {
        logger.error (e, e);
        throw new RegistryInitializationException (e.getMessage(), e);
    }
}

The really interesting call here is enhancePropertyGetters(wrappers);. This is where we use bytecode manipulation to generate the subclass dynamically and override the getter of the method declaring the @Resource annotation.
We won't present the other methods but we'll see the listing of this enhancePropertyGetters() method:

private static void enhancePropertyGetters(Map<Class<?>, ServiceWrapper> wrappers) 
        throws NotFoundException, CannotCompileException, ClassNotFoundException {
    ClassPool.doPruning = true;

    ClassPool pool = ClassPool.getDefault();
    pool.appendClassPath(new LoaderClassPath(
            Thread.currentThread().getContextClassLoader()));

    for (ServiceWrapper wrapper : wrappers.values()) {

        if (wrapper.getMethodsToEnhance().size() > 0) {

            CtClass superClazz = pool.get(wrapper.getServiceName());

            // Unfortunately I need to go with the sub-class approach
            // I cannot change the original class since it has already been loaded and
            // javassist cannot change a class that is already loaded (that would require
            // changing linking and javassist cannot do that)

            CtClass clazz = pool.makeClass(wrapper.getServiceName() + "$javassist_sub");
            clazz.stopPruning(true);

            clazz.setSuperclass(superClazz);
            clazz.setModifiers(Modifier.PUBLIC);

            ...

            // Add registry on class if not already one. The field might already have been
            // added on a parent class. If this is the case, don't add it again
            injectRegistryField(pool, clazz);

            // Proceed with method modification
            for (Method method : wrapper.getMethodsToEnhance()) {

                // Various cases :

                // 1. Method doesn't have the form of a getter
                if (!method.getName().startsWith("get")) {

                    // => Just override method so it returns the service from registry
                    overrideMethod(clazz, method);
                }

                // 2. Method is a getter
                else {

                    try {
                        Method setter = ReflectionUtils.getSetter(wrapper.getServiceClass(),
                                ReflectionUtils.getPropertyName(method),
                                method.getReturnType());

                        // 2.2 setter is found
                        // => Lazy loading : use underlying field as cache.
                        // If it it set, do nothing.
                        // If is is null, look for service and attach it
                        // Needs to override the getter for doing my business and
                        // delegate to the setter for setting the field
                        overrideGetter(clazz, method, setter);

                    } catch (NoSuchMethodException e) {

                        logger.debug (e, e);

                        // 2.1 No setter could be found
                        // => Just rewrite method so it returns the service 
                        // from the registry
                        overrideMethod(clazz, method);
                    }
                }
            }
            
            ...

            // make new subclass available to classloader
            pool.toClass(clazz, Thread.currentThread().getContextClassLoader(), null);
            clazz.stopPruning(false);

            // use the new subclass instead of the original class from now on
            Class<?> subClazz = Class.forName(clazz.getName());
            wrapper.overrideClass(subClazz);
        }
    }
}

In the code above, the interesting calls are overrideGetter(), overrideMethod() and injectRegistryField() since these are the methods where bytecode manipulation occurs.
Let's look at these methods:

private static void overrideGetter(CtClass clazz, Method getter, Method setter) 
        throws CannotCompileException {
    String targetService = getter.getReturnType().getCanonicalName();
    CtMethod newMethod = CtNewMethod.make(
                "public " + targetService + " " + getter.getName() + "() { \n" +
                "" +
                "    " + targetService + " retObject =  super." + getter.getName() + "(); "+
                "    if (retObject == null) {" +
                "         retObject =  (" + targetService + ") \n" + 
                "                 getRegistry().getService(\"" + targetService + "\"); " +
                "         super." + setter.getName() + "(retObject);" +
                "    }" +
                "    return retObject;" +
                "" +
                "}",
            clazz);
    clazz.addMethod(newMethod);
}

private static void overrideMethod(CtClass clazz, Method method) 
        throws CannotCompileException {
    String targetService = method.getReturnType().getCanonicalName();
    CtMethod newMethod = CtNewMethod.make(
                "public " + targetService + " " + method.getName() + "() { \n" +
                "" +
                "    " + targetService + " retObject =  \n" + 
                "            (" + targetService + ")\n" + 
                "            getRegistry().getService(\"" + targetService + "\"); " +
                "    return retObject;" +
                "" +
                "}",
            clazz);
    clazz.addMethod(newMethod);
}

/**
 * Inject a field to store the registry in the target clazz as well as a getter 
 * to retrieve that registry.
 *
 * @param pool the Javassist ClassPool to be used
 * @param clazz the class to be modified this way
 * @throws NotFoundException
 * @throws CannotCompileException
 */
public static void injectRegistryField(ClassPool pool, CtClass clazz) 
        throws NotFoundException, CannotCompileException {
    CtField registryField = null;
    try {
        registryField = clazz.getField("registry");
    } catch (NotFoundException e) {
        // ignored
    }
    if (registryField == null) {
        CtClass registryClass = pool.get(Registry.class.getName());
        registryField = new CtField(registryClass, "registry", clazz);
        registryField.setModifiers(Modifier.setPrivate(Modifier.STATIC));
        clazz.addField(registryField, "null");

        CtMethod registryGetter = CtNewMethod.getter("getRegistry", registryField);
        registryGetter.setModifiers(Modifier.PUBLIC);
        clazz.addMethod(registryGetter);

        CtMethod registrySetter = CtNewMethod.make(
                    "public static void setRegistry (" + Registry.class.getName() + 
                    "            holder) { " +
                    "    registry = holder; " +
                    "} ",
                clazz);
        clazz.addMethod(registrySetter);
    }
}

We've seen the most important pieces of code of the SCIF Framework above.
The framework itself is available for download in the next section.

4.4 DemoApp : Comet Tennis

I integrated the SCIF framework in the comet-tennis demo application. It's a small application I wrote initially here : comet-tennis.
This application uses a few services and the idea here is to use the SCIF framework to manage these services and inject their dependencies.

The new package with the SCIF framework integrated is available here.

5. Conclusion

Bytecode manipulation is a lot of fun and opens a whole new world of possibilities on the JVM. It's the only way to implement advanced tooling such as IoC Containers, ORM frameworks, boilerplate code generators, etc.
Normally, bytecode manipulation is something rather pretty difficult to achieve ... except with Javassist.
Javassist makes bytecode manipulation so easy and straightforward. The ability to write dynamically in simple strings actual java source code and add it on the fly as bytecode to classes being manipulated is striking. Javassist is in my opinion the simplest way to perform bytecode manipulation in Java.

I covered above some use cases for bytecode manipulation, there are many others, for instance tampering with licence checking systems of non-free software (Hush. I said nothing)

In my career, I have encountered many situations where I wish afterwards I have known Javassist since it would have been pretty helpful. Let me mention two:

Some 15 years ago, I was working on a pretty big J2EE Websphere application with a lot of EJBS. Tracking the user flow in the distributed system was a nightmare due to the complexity of the business processes and the business rules, so we ended up adding logging information each and every time a business method was entered and left, such as log.debug ("ENTER - business method") and log.debug ("LEAVE - business method").
In regards to troubleshooting, this may sound stupid but it ended up being not only pretty convenient but rather really our single and only way to figure what was going on in some situations in such a enormous software.
Adding these two lines of code (plus a few try { ... } finally { ... } statements to make sure the leaving trace was always output made us add thousands of lines of code to the application ... which could have been replaced by a few lines of code java agent and some javassist magic.
Some 10 years ago, I was working for a banking institution on a big Java application making an extensive use of Hibernate. The problem there is that we were trying to map a nice and meaningful business model to a legacy data model. With a lot of hibernate tricks we pretty much succeeded in achieving the mapping, using a lot of custom and pretty tricky code in hibernate session listeners to handle the relationships that hibernate as not able to handle natively.
There as well, we ended up writing thousands of lines of specific glue code in hibernate listeners which we could have replaced by a pretty simple Javassist base framework to complement the missing features of hibernate.

You might want to have a look at the second article in this serie available here : Bytecode manipulation with Javassist for fun and profit part II: Generating toString and getter/setters using bytecode manipulation.

Part of this article is available as a slideshare presentation here: https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit.

Java - Create enum instances dynamically

Jerome Kehrli — Sat, 13 Nov 2010 15:08:53 -0500

I remember the introduction of the brand new enum type in Java 5 (1.5) was a very exciting announce. However, when I finally switched from 1.4 to 1.5 and actually tried Java's flavoured enum types, I was a bit disappointed.

Before that, I was using Josh Bloch's "Typesafe enum" pattern (effective java) for quite a long time and I didn't really see what was so much better with the new Java native enum construction. Ok, fine, there was the ability to use enum instances in switch - case statements which seemed fine, but what else ?

Besides, what I used to find great with the "typesafe enum" pattern is that it could be tricked and changed the way I wanted, for instance to be able to dynamically (at runtime) add enum instances to a specific typesafe enum class. I found it very disappointing not to be able to do the very same thing easily with the native Java enum construction.

And now you might wonder "Why the hell could one ever need to dynamically add enum values ?!?". You do, right ? Well, let's imagine this scenario:

You have a specific column in a DB table which contains various codes as values. There are more than hundred different codes actually in use in this column. Related to this, you have a business logic which performs different operations on the rows coming from this table, the actual kind of operation applied on the row depends on the value of this code. So there are chance you end up with a lot of if - elseif statements checking the actual value of the code.
I myself am allergic to using string comparison in conditions so I want to be able to map the values from this column to an enum type in Java. This way I can compare enum values instead of strings in my conditions and reduce my dependency on the format of the string value.

Now when there are more than a hundred different possible codes in the DB I really don't have any intent to define them all manually in my enum type. I want to define only the few I am actually using the Java code and let the system add the other ones dynamically, at runtime, when it (the ORM system or whatever I am using for reading the DB rows) encounters a new value from the DB.

Hence my need for dynamically added enum values.

So recently I faced this need once again and took a few hours to build a little solution which enables one to dynamically add values to a Java enum type. The solution is the following :

(Complete code here : DynamicEnumTest.java)

void addEnum(Class<T> enumType, String enumName)

This method adds an enum instance to the enum type given as argument. The whole method code is presented below. We will first details its behaviour and then present every other method this one is using.



    /**

     * Add an enum instance to the enum class given as argument

     * 

     * @param <T> the type of the enum (implicit)

     * @param enumType the class of the enum to be modified

     * @param enumName the name of the new enum instance to be added to the class.

     */

    @SuppressWarnings("unchecked")

    public static <T extends Enum<?>> void addEnum(Class<T> enumType, String enumName) {



        // 0. Sanity checks

        if (!Enum.class.isAssignableFrom(enumType)) {

            throw new RuntimeException("class " + enumType + " is not an instance of Enum");

        }


        // 1. Lookup "$VALUES" holder in enum class and get previous enum instances

        Field valuesField = null;

        Field[] fields = enumType.getDeclaredFields();

        for (Field field : fields) {

            if (field.getName().contains("$VALUES")) {

                valuesField = field;

                break;

            }

        }

        AccessibleObject.setAccessible(new Field[] { valuesField }, true);



        try {



            // 2. Copy it            

            T[] previousValues = (T[]) valuesField.get(enumType);

            List<T> values = new ArrayList<T>(Arrays.asList(previousValues));



            // 3. build new enum

            T newValue = (T) makeEnum(enumType, // The target enum class 

                    enumName, // THE NEW ENUM INSTANCE TO BE DYNAMICALLY ADDED 

                    values.size(),

                    new Class<?>[] {}, // can be used to pass values to the enum constuctor

                    new Object[] {}); // can be used to pass values to the enum constuctor



            // 4. add new value

            values.add(newValue);



            // 5. Set new values field

            setFailsafeFieldValue(valuesField, null, 

                    values.toArray((T[]) Array.newInstance(enumType, 0)));



            // 6. Clean enum cache

            cleanEnumCache(enumType);



        } catch (Exception e) {

            e.printStackTrace();

            throw new RuntimeException(e.getMessage(), e);

        }

    }

The enum instances are stored in a static variable of the enum class named $VALUES. Here we search for this static variable, make it accessible and keep the reference for further usage.
The enum instances contained in the $VALUES static variable are copied (stored) in another (new) list.
Then the new enum instance is built using another method named makeEnum which will be discussed later.
The new enum instance is added to the new enum list.
The new enum instance list (containin the new enum instance) is set to the $VALUES field, overwriting the previous content.
The cleaning of the cache answers to something special and tricky. The problem is that the $VALUES is purely generated by the compiler. So the static java code located in java.lang.Class cannot be statically linked to it. Hence it needs to access $VALUES variable using runtime reflection which is a little costy (compared to static method call linking). Hence the code in java.lang.Class which needs an access to the enum values makes a copy of them upon first usage in a private instance variable, the enum cache
Thus, adding the new enum instance in the $VALUES list is not sufficient, one needs to make sure the java.lang.Class instance caches are cleared as well.

Object makeEnum(Class<?> enumClass, String value, int ordinal, Class<?>[] additionalTypes, Object[] additionalValues)

This method creates the new enum instance. It takes as argument the class for which to create a new enum instance and the name of the enum instance to create. The ordinal argument is the ordinal value which will be associated to the enum instance. The two last arrays are helpful if on want to add an enum instance to an enum type using constructor arguments to store values bound to the enum instances. The details of the method is as follows :


    private static Object makeEnum(Class<?> enumClass, String value, int ordinal, 

            Class<?>[] additionalTypes, Object[] additionalValues) throws Exception {

        Object[] parms = new Object[additionalValues.length + 2];

        parms[0] = value;

        parms[1] = Integer.valueOf(ordinal);

        System.arraycopy(additionalValues, 0, parms, 2, additionalValues.length);

        return enumClass.cast(getConstructorAccessor(enumClass, additionalTypes).newInstance(parms));

    }

The makeEnum method uses another method named getConstructorAccessor to obtain a reference on the constructor accessor allowing to actually create the new enum instance :


    private static ConstructorAccessor getConstructorAccessor(Class<?> enumClass, 

            Class<?>[] additionalParameterTypes) throws NoSuchMethodException {

        Class<?>[] parameterTypes = new Class[additionalParameterTypes.length + 2];

        parameterTypes[0] = String.class;

        parameterTypes[1] = int.class;

        System.arraycopy(additionalParameterTypes, 0, 

                parameterTypes, 2, additionalParameterTypes.length);

        return reflectionFactory.newConstructorAccessor(

                enumClass.getDeclaredConstructor(parameterTypes));

    }

setFailsafeFieldValue(Field field, Object target, Object value) throws NoSuchFieldException, IllegalAccessException

This method is a convenient method whose responsibility is to set the value given as argument to the field given as argument on the target given as argument. This sounds easy yet doing in failsafely is a little bit more tricky than one could guess :


    private static void setFailsafeFieldValue(Field field, Object target, Object value)

             throws NoSuchFieldException, IllegalAccessException {



        // let's make the field accessible

        field.setAccessible(true);



        // next we change the modifier in the Field instance to

        // not be final anymore, thus tricking reflection into

        // letting us modify the static final field

        Field modifiersField = Field.class.getDeclaredField("modifiers");

        modifiersField.setAccessible(true);

        int modifiers = modifiersField.getInt(field);



        // blank out the final bit in the modifiers int

        modifiers &= ~Modifier.FINAL;

        modifiersField.setInt(field, modifiers);



        FieldAccessor fa = reflectionFactory.newFieldAccessor(field, false);

        fa.set(target, value);

    }

cleanEnumCache(Class<?> enumClass) throws NoSuchFieldException, IllegalAccessException

This method clears the enum cache variable on two different JVM implementations.


    private static void cleanEnumCache(Class<?> enumClass) 

            throws NoSuchFieldException, IllegalAccessException {

        blankField(enumClass, "enumConstantDirectory"); // Sun (Oracle?!?) JDK 1.5/6

        blankField(enumClass, "enumConstants"); // IBM JDK

    }

It used the blankField method to actually blank the target field :


    private static void blankField(Class<?> enumClass, String fieldName) 

            throws NoSuchFieldException, IllegalAccessException {

        for (Field field : Class.class.getDeclaredFields()) {

            if (field.getName().contains(fieldName)) {

                AccessibleObject.setAccessible(new Field[] { field }, true);

                setFailsafeFieldValue(field, enumClass, null);

                break;

            }

        }

    }

On can test this dynamic enum instance adding system using the following little code snippet:


    private static enum TestEnum {

        a,

        b,

        c;

    };



    public static void main(String[] args) {



        // Dynamically add 3 new enum instances d, e, f to TestEnum

        addEnum(TestEnum.class , "d");

        addEnum(TestEnum.class , "e");

        addEnum(TestEnum.class , "f");



        // Run a few tests just to show it works OK. 

        System.out.println(Arrays.deepToString(TestEnum.values()));

        // Shows : [a, b, c, d, e, f] 

    }

The complete code presented in this article is available here : DynamicEnumTest.java

Interestingly, this is a situation where runtime reflection, as limited as it is, is yet sufficient for the few required manipulations involved in this dynamic enum creation feature. It would be interesting to see how a much more powerful mechanism such as javassist (bytecode manipulation) could make things easier. Maybe I'll dig into it someday ...

HTH

Java rocks !

Jerome Kehrli — Wed, 3 Nov 2010 03:40:36 -0400

I've been facing an interesting problem with string manipulation in Java lately at work. The requirement was the following :

We have a field on some screen where the user can type in a comment. The comment can have any length the user wants, absolutely any. Should he want to type in a comment of a million characters, he should be able to do so.

Now the right way to store this comment in a database is using a CLOB, a BLOB or a LONGVARCHAR or whatever feature the database natively provides to do so. Unfortunately that's not the way it was designed. Due to legacy integration needs, all these advance DB types are prohibited within our application. So the way we have to store the comment consists of using several rows with a single comment field of a maximum length of 500 characters. That means the long comment has to be split in several sub-strings of 500 characters and each of them is stored in a separate row in the DB table. The table has a counter as part of the primary key which is incremented for each new row belonging to the same comment. This way we can easily spot every row part of the same comment.

Now another problem we have is that under DB2 a field defined as VARCHAR(500) can contain 500 bytes max even though the strings are encoded in UTF-8 in the database. That means we might not be able to store 500 characters if the string contains one or more 2 bytes UTF-8 characters. Working in a french environment, this happens a lot.
So we had to write a little algorithm taking care of the splitting of the string in 500 bytes sub-strings.

The very first version of our algorithm was quite stupid and ended up in splitting the string in a quite naive way: we converted the string to a byte array following an UTF-8 encoding and split the byte array instead of the string. Then each of the 500 bytes arrays was converted back to a string before being inserted in the database.
Happily, we figured quite soon that this doesn't work as it ends up quite often splitting the string right in the middle of a 2 bytes character. The byte arrays being then converted back to strings, the split 2 bytes character was corrupted and could not be corrected any more.

Before writing as smarter version of the algorithm which would manually test the byte length of the character right at the position of the split, we took a leap backward and wondered : "Can it be that Java doesn't offer natively a simple way to do just that ?"

And the answer is yes of course.

The solution with Java is quite straightforward :


import java.nio.ByteBuffer;

import java.nio.CharBuffer;

import java.nio.charset.CharacterCodingException;

import java.nio.charset.Charset;

import java.nio.charset.CharsetDecoder;

import java.nio.charset.CharsetEncoder;

import java.nio.charset.CoderResult;

import java.util.ArrayList;

import java.util.List;



public class StringSplitter {



    public static List<String> splitString500(String value) throws Exception {

        List<String> retList = new ArrayList<String>();

        try {



            Charset utf8CSet = Charset.forName("UTF-8");

            CharsetEncoder enc = utf8CSet.newEncoder();

            CharsetDecoder decoder = utf8CSet.newDecoder();



            CharBuffer cBuffer = CharBuffer.wrap(value);

            ByteBuffer bBuffer = ByteBuffer.allocate(500);

            CoderResult cr = null;

            do {

                cr = enc.encode(cBuffer, bBuffer, false);

                retList.add(decoder.decode((ByteBuffer) bBuffer.flip()).toString());

                bBuffer.rewind();

            } while (cr != CoderResult.UNDERFLOW);



        } catch (CharacterCodingException e) {

            e.printStackTrace();

            throw new Exception(e);

        }

        return retList;

    }

}

This piece of code is interesting as it underlines two things :

Java is definitely a language making string manipulation as simple as possible. There are hundreds of classes aiming at simplifying the overwhelming complexity of string representation, encoding, decoding, etc. Once again Java rocks.
Despite working for almost 8 years everyday with Java technologies, I discover new classes and new stuff even in the standard Java libraries everyday, not speaking about the dozens of Java related technologies and libraries that flourish everyday. This is a good example as I was really about to code a piece of code on my own before actually finding out the JDK already provides such a feature.

Well that was sort of a good lesson of humility. Hackers tend to limit themselves to the set of things they know of the technology and consider everything else just doesn't exist and has to be done on their own while actually the very feature they are looking for might well already be implemented. One simply need to open his eyes.

CommunityBoard : a sample multi-module maven / glassfish / eclipse Java EE project.

Jerome Kehrli — Sun, 24 Oct 2010 16:29:13 -0400

CommunityBoard is a sample multi-module maven / glassfish / eclipse Java EE project.

It realizes is a little Forum / Note publishing application. Its main purpose it to act as an introducing laboratory to Java EE programming. As such the functionalities are rather limited. Yet it covers the most fundamental aspects or issues with Java EE programing in the way it show hows to :

write entity beans with bi-directional relationship;
use these Entity beans in EJBs (Stateless session beans);
use other EJBs in EJBs;
use EJBs in a servlet or a JSP located in a WAR (i.e. no processing of the @EJB annotation);
build a multi-module Java EE maven project with jars, wars, ears;
how to write JSPs with the JSTL (Ok I am not very proud of these JSPs yet they do the job) and
deploy a multi-module ear within Glassfish and use a container defined datasource

The project is available here : CommunityBoard.tar.gz

Build and installation instructions

1. This document

This document covers the few steps required to build and install the CommunityBoard EAR on a Glassfish 3.0 server and presents a very brief manual instruction. The document assumes a basic knowledge of maven command and the Glassfish 3.0 server.

As a sidenote, one should know the maven pom files provided here make usage of the maven-eclipse-plugin allowing a smooth project integration within eclipse simply by typing mvn eclipse:clean eclipse:eclipse at the root of the project. This command creates the eclipse project files (.project, .classpath, etc.).
This being said, the maven to eclipse project integration won't be covered any further in this document.

In the same way, this document assumes a basic knowledge of the usual maven commands and the basis of the Glassfish 3.0 Java EE application server on the reader side. We wont cover here the installation of both maven and glassfish and the way to use these tools.

2. Some technical information

2.1. The project structure

The project structure is as follows :

`CommunityBoard`	The aggregation module: this one aggregates the 3 other modules to produce a Java EE ear file containing the 3 other modules.
`CommunityBoard_data`	The entity module (data layer). This one is a JPA project which provides the Java EE entity beans used by the project. This modules is packaged as a java `.jar` and defined as a `jar` module within the aggregation ear.
`CommunityBoard_services`	The services module (business layer). This modules provides the various business services used by the CommunityBoard application. This modules is packaged as a java `.jar` file on its own and defined as an `ejb` module within the aggregation ear.
`CommunityBoard_presentation`	The Web module (presentation layer). This modules provides the front controller as well as various JSPs used to present the data and provides the user with ways to manipulate the data. This module is packaged as a java EE `.war` file on its own and defined as a `web` module within the aggregation ear.
`pom.xml`	The root maven POM file.
`docs`	A few files (mostly images) used by this README file.

Each individual module is very simple and only holds a few Java files or JSPs as well as its own maven POM file which defines the precise way to build it. This documentation won't cover much more into details the precise content of these modules and the author strongly encourage the reader to take a quick look at these modules before going any further with the reading of this document.

2.2. The project dependencies

The CommunityBoard project has only 2 dependencies. The snippet of the root maven POM file which defines the dependencies is:

        <dependency>
            <groupId>org.apache.derby</groupId>
            <artifactId>derbyclient</artifactId>
            <version>10.6.1.0</version>
        </dependency>
        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-api</artifactId>
            <version>6.0-SNAPSHOT</version>
            <scope>provided</scope>
        </dependency>

derbyclient is the JDBC client to the Glassfish embedded Derby database we will be using.
javaee-api is a required compile time only dependency for pretty much every Java EE project.

There are actually a few more dependencies such as taglibs for the JSP files. It is however not required to declare these taglibs in the maven POM file as they are neither required for the maven compilation (JSPs are compiled by the application server) nor are they required at deployment (both eclipse and glassfish provide them natively).

3. Building and installation

3.1 Build the application

Building the application, thanks to maven, couldn't be more straightforward. One only needs to cd into the root folder of the project and build the application with the usual maven command :

user@notebook:/data/.../workspace$ mvn clean install
[INFO] Scanning for projects...
[INFO] Reactor build order: 
[INFO]   Niceideas CommunityBoard project
[INFO]   Niceideas CommunityBoard - data layer
[INFO]   Niceideas CommunityBoard - business layer
[INFO]   Niceideas CommunityBoard - presentation layer
[INFO]   Niceideas CommunityBoard - Agregate EAR
[INFO] ------------------------------------------------------------------------
[INFO] Building Niceideas CommunityBoard project
[INFO]    task-segment: [clean, install]
[INFO] ------------------------------------------------------------------------
...
...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] ------------------------------------------------------------------------
[INFO] Niceideas CommunityBoard project ...................... SUCCESS [3.103s]
[INFO] Niceideas CommunityBoard - data layer ................. SUCCESS [6.652s]
[INFO] Niceideas CommunityBoard - business layer ............. SUCCESS [2.206s]
[INFO] Niceideas CommunityBoard - presentation layer ......... SUCCESS [4.780s]
[INFO] Niceideas CommunityBoard - Agregate EAR ............... SUCCESS [1.459s]
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18 seconds
[INFO] Finished at: Sun Oct 24 09:47:18 CEST 2010
[INFO] Final Memory: 28M/119M
[INFO] ------------------------------------------------------------------------

The above is what should be displayed on the console if the build is successful (really there's no reason for it not to be successful).

The build ends up with the project ear file located at ./CommunityBoard/target/CommunityBoard-1.0-SNAPSHOT.ear. This is the file we will be deploying within Glassfish.

3.2 Create the datasource

This section and the one below assume the glassfish application server is started and the user/reader has access to the administration console.
(Just as a sidenote, glassfish is typically started by : $ glassfish/bin/startserv domain2)

In addition to the glassfish application server, the glassfish embedded derby database server should be started as well. (Just as a sidenote, the embedded derby is typically started by : $ glassfish/bin/asadmin start-database --dbport 1527 --dbhome glassfish/databases)

3.2.1 Define the datasource

A datasource named jdbc/CommunityBoardDS in the JDNI tree is required for the CommunityBoard application to work properly. Under glassfish the datasource is easily defined through the admin console by using the 2 links under the Resource/jdbc menu on the left :

First, one should create a new Connection Pool named CommunityBoardPool for a Derby database. The type of the resource is javax.sql.Datasource :

On the next page, one should go to the bottom of the page and set the connection properties as follows :

The mandatory properties are :

DatabaseName	`CommunityBoard`
User	`admin`
Password	`whatever`

The other properties depend pretty much on your local glassfish installation. In general the embedded derby server is configured by default as accepting any string as password (hence the whatever above).

Then one should create a new JDBC resource named jdbc/CommunityBoardDS and using the provided Derby Pool :

This is it. The datasource is created an available to deployed applications.

3.2.2 Create the database.

Now the CommunityBoard database should be created. This involves creating the four tables used by the application and populating them with initial test data.
The project contains a DDL file located at CommunityBoard_data/src/main/ddl/create_db.sql and a little SQL file containing the test data located at CommunityBoard_data/src/main/ddl/insert_test_data.sql.

The reader is free to use pretty much any SQL client she is used to. With DBVisualizer for instance, the SQL statements from the SQL files can be easily executed (only one after the other when using the free version of the tool though).
Under DBVisualizer, the DB connection is defined as follows :

The derby JDBC client driver can be located in the maven local repository once the project has been built at least once.

Please notice that the DDL file should be edited a little bit before usage. It cannot be immediately executed as is, at least not without removing first the 4 first lines which contains drop statements in case the script is executed many times. These drop statements should be commented out or the script will most likely fail upon execution.

3.3 Deploy the ear

Deploying the ear file built by maven is a piece of cake with glassfish. We'll use the glassfish admin console here for the sake of simplicity.

The ear file to be deployed has been generated by maven in CommunityBoard/target/CommunityBoard.ear

One only needs to select the Applications item in the menu on the left and then click on Deploy.
On the next page, when one selects the ear we have built previously, the admin console fulfills automatically the rest of the page :

If everything goes fine, the admin console should display the following page which shows the CommunityBoard application successfully deployed :

The user can now reach the following URL with her favorite browser to start using the application : http://localhost:8080/CommunityBoard_presentation/controller2/login.do.

The test data insertion script should have created a few users, the first one is

Username	`user1`
Password	`pwd1`

which one simply need to type in on the following screen :

4. A few concerns ...

Just a few concerns :

The JSP are most likely too complicated and a JSP expert would surely be able to do the same or more in twice less lines of code using proper JSP tags.
Besides, there is the concept of backing bean which I couldn't dig very much into details.
Well, don't shoot me, I did my best in such a short time.
I am not yet 100% sure the ServiceLocator used in the web layer to grasp references on EJB's is required. It's just that I haven't been able to find a way to make the @EJB annotation work from within the servlet or the JSP pages once I put them and the EJBs in separated projects.
So I think glassfish just doesn't take into account a WAR project for annotation processing and hence the ServiceLocator is required.
(Please don't hesitate to contact me should you have a better understanding of this issue or a fix to submit for it)

DWR : A paradigm shift in web development

Jerome Kehrli — Thu, 29 Apr 2010 10:57:58 -0400

I discovered DWR recently and I believe it to be an amazing breakthrough in the world of HTTP client-server comunications.

First what is DWR ?

DWR stands for Direct Web Remoting - Easy Ajax for Java.

DWR is a Java library that enables Java on the server and JavaScript in a browser to interact and call each other as simply as possible.

Quoting the official website :

"DWR is a RPC library which makes it easy to call Java functions from JavaScript and to call JavaScript functions from Java (a.k.a Reverse Ajax)."
Read this : http://directwebremoting.org/dwr/introduction/index.html.

What's the point ?

Yes you can actually call your server-side Java code from within client-side Javascript code just as if it was in fact some other Javascript code located just aside and definitely not remote code in a foreign technology located on a server far away. Using DWR, you get the illusion your java environment on the server side and the Javascript environment on the client side suddenly and magically got merged together in a single client side Javascript environment.

Even better, using long-poll HTTP requests (recently renamed COMET or Piggy-back HTTP requests or even reverse-AJAX thanks to happy marketing efforts), you can get as close as possible from real server pushes where the Java code running on the server issues RPC-like calls to the client side Javascript code.
Pretty nifty, huh ?

And all of this of course is done (almost completely) transparently. The Javascript code is not aware that the Javascript stub it calls actually forwards the call to some sort of Java server somewhere, no. You can program your client-side Javascript application just as if the server-side business logic actually stands right there on the client-side.

Obviously real-life scenarii suffer from some limitations. For instance, for this to work efficiently, a thorough SOA design is required on both sides to minimize round-trips between the client and the server.

Why is it cool ?

It's cool ... because it's quite obviously cool. Let's look at the alternatives:

For ages we have been using CGI scripts, then PHP, then even JSP or lately JSF or any other crappy technology alike to build dynamic web applications. I'm sorry fanboys but all of this sucked a lot : complete mixture of concerns and technologies in the same file (some must be thinking this is some sort of good pattern), steady learning curve (got to know HTML, Javascript, CSS, PHP or JAVA, taglibs, etc.), hard to debug, huge network bandwidth consumption (90% of it being presentation stuff, no data), and many other reasons.

Then some smart boys came up with AJAX ... Well XMLHttpRequest actually is a good thing (although it's just an easier way to do just the same thing others were doing with hidden iframes since ages).
Anyway, it made (a)synchronous (or at least hidden) server calls easier and enabled client-side Web developers to build Web GUIs acting a little bit more like real GUIs : a single page is loaded, ït builds a GUI and then interacts with the server using artificial, hidden and (a-)synchronous method calls. This definitely was an improvement for two reasons: first it enables a much richer user experience with much more dynamic interfaces where only sub-parts can change and evolve depending on the user interactions. Second, the developer work is made easier as the GUI is built within a single technology, although different languages: Javascript, DOM (HTML) and CSS.
Most of the time, the client-side interface is built using high-level Javascript libraries such as Bindows or DOJO. With such libraries the building of a Web GUI becomes almost as easy as a rich-client GUI.

But let's face it, there is something with all of this still far from ideal. The problem comes with the XML in XMLHttpRequest. Why the hell should one bother to marshal / unmarshal XML in two different technologies (client-side Javascript and server-side PHP, java, etc.) ? There are cases where Web-services make sense, for instance when a .NET GUI needs to communicate with a Java backend. Both technologies offer several tools to simplify Web-services integration and make them an appropriate choice.
But guys, let's be honest, this very example really is the only one where Web-services make sense. In every other technology combination, especially within Web application where one side is written in Javascript, there is always a way better alternative to XML. In our case it's called: JSON.

JSON stands for Javascript Standard Object Notation. Doesn't it ring a bell ?
Happily, a lot of web developers understand this perfectly nowadays and JSON/RPC is becoming a de-facto standard for cutting-edge modern web developments. The server side backend usually relies on simple REST services and that is a huge simplification as well over the overwhelming complexity of Web-services.

And this is where DWR kicks in. DWR over-simplifies the complete development process for web application build around REST services and using JSON/RPC as the data exchange format. As I stated before, DWR offers such a high level of abstraction around those REST services that the Web developer really completely ignores every REST, RPC, marshalling, etc. concerns. She calls the services as if they were plain-old Javascript objects written in Javascript and standing at the client side.
Really cool.

And this is where I believe is the paradigm shift. DWR makes it feel like the client-server separation vanishes. It really feels like everything now stands in one single unified place at the client side.

How does it work ?

DWR consists of two main parts:

A Java Servlet running on the server that processes requests and sends responses back to the browser.
JavaScript running in the browser that sends requests and can dynamically update the webpage.

DWR works by dynamically generating Javascript based on Java classes. The code does some AJAX magic to make it feel like the execution is happening on the browser, but in reality the server is executing the code and DWR is marshalling the data back and forwards.
DWR used JSON to transmit the data over the network to the server. On the server sides it provides the completely JSON to Plain Old Java Object serialization system.

Every Java object which is intended to be available for remote method calls on the client application should be declared in the dwr.xml configuration file. DWR then generates the Javascript stubs taking care of the marshalling of the arguments, the remote over HTTP and the unmarshalling of the results. It can be used on a wide range of browser as it supports several way to actually implement the HTTP call: usually using an XMLHttpRequest object, but it can also use hidden iframes in case XMLHttpRequest is not available.

DWR can serialize a complex graph of objects from Javascript to Java and back as well as every sort of collections, exceptions, etc.

hibernate's [not-found="ignore"] is buggy as hell

Jerome Kehrli — Wed, 27 Jan 2010 13:52:36 -0500

I'm working on a java application which makes an extensive usage of hibernate's relation mapping system. The later offers several ways to define association mapping. We mostly use many-to-one relation declarations. The problem comes from the database. It's a pre-relational, pre-transactional, legacy database running on a prehistorical IBM zSeries host. The data on this database is very often dumb or corrupted. The lack of a proper referential integrity support and the foolish design make us end up quite often following non-existent relations.

Happily, hibernate provides a semantic which allow the application not to bother when a relation is missing, just as the legacy app does. This semantic is the not-found="ignore" parameter on the relation definition.

However, the usage of this semantic resumes to open very wide the doors to oblivion.

Hibernate's first level cache is very nifty for what is related to relation management. For instance, it allows great pre-fetching strategies. Should you load a hundred rows from some entity which holds references to another entity, all you have to do is define the relation with lazy="proxy" (which is the default), and when you follow one of these relations all the other (for the rest of the hundred rows) are loaded in more or less one shot.
This works great and is very efficient.

Problems arise when the same relation to the very same entity (db table) is mapped using different definitions. We will now illustrate this issue.

Let's imagine the following situation :

`Entity C`	represents the target of both relations
`Entity A`	holds a relation to Entity C. The relation is defined with `not-found="exception"` as in the following hibernate mapping file snippet: `<many-to-one name="associatedC" column="ID_OF_C" class="very.nifty.EntityC" lazy="proxy" not-found="exception"/>`
`Entity B`	holds a relation to Entity C. The relation is defined with `not-found="ignore"` as in the following hibernate mapping file snippet: `<many-to-one name="associatedC" column="ID_OF_C" class="very.nifty.EntityC" not-found="ignore"/>`

This means that both Entity A and Entity B hold a relation to Entity C. All instances of Entity A should mandatorily have a counterpart target in Entity C. If none is found following the relation, hibernate should throw an exception.
On the contrary when the target Entity C corresponding to Entity B is not found, hibernate should silently set null on the relation holder and not complain in any way.

We will now see that things can behave quite differently than what we can expect.

Case 1 : only Entity A instances are loaded.

When several instances of Entity A are loaded which reference the same instance of Entity C, everything works fine.
We have asked hibernate to manage the relation through a proxy. Hence, instances of Entity C associated to Entity A are not eagerly fetched. When an instance of Entity A is build, hibernate associates to it a proxy on the corresponding Entity C instance without actually fetching it from the DB. The proxy is obviously not null.
Only when the program actually tries to call a method on this proxy, hibernate puts the call on hold and tries to load the data from the database. At that stage, if the target Entity C instance does not exist in the database, hibernate does not have much of a choice but triggers an ObjectNotFoundException. Due to the proxy already associated to the relation in Entity A, returning null is not an option any more (obviously). So throwing an exception is pretty much anything hibernate can do.

This works as expected and is fine.

Case 2 : only Entity B instances are loaded.

The relation from Entity B to Entity C is defined as not-null="ignore". In order to support this, hibernate cannot use any proxing mechanism. It needs to check immediately the existence of the corresponding row in the target table in order to be able to set null on the relation holder within the source entity should the target row not exist. This prevents it from associating a proxy to the relation holder and fetching the row lazily.
You can perfectly define a relation set as not-null="ignore" with lazy="proxy" but this is just going to be ignored by hibernate. The not-null="ignore" primes and the relation is not managed with a proxy.

So what happens if one or several instance of Entity B are loaded which reference the same instance of Entity C ? Each and every time an Entity B is loaded, hibernate will try to find the matching row in Entity C in the DB. If it doesn't find it, it will set null to the relation holder in Entity B and return it to the calling program.

You might think "great, so this not-null="ignore" actually works" and you wouldn't be wrong. But there's a trick :

The problem is that the caching mechanisms within hibernate are quite stupid. They don't cache negative hits. Imagine you load a hundred rows from Entity B which all reference the very same row of Entity C which does not exist. When the first row is loaded, hibernate checks whether the target row in Entity C exist. It doesn't and hence hibernate sets null on the first entity relation holder. The the second row is loaded. At that stage hibernate does not remember it already tried to lookup that very same instance of Entity C and tries again, And again for the third row, and so on. This is quite stupid as there are very few chance that a row suddenly and magically appear between to SQL calls within the same transaction. This might depend of the transaction isolation level of course, but still, chances are very little, especially on very short time periods.
This is what I mean when stating that hibernate does not cache negative hits. It does not remember that it has already look for some data in a table within a single transaction and that it didn't find it and it tries again and again and again...Stupid.

However this doesn't affect the functional behaviour of the application, only the performance, and this is not the problem on which I want to focus here. So let's be happy with it. And as a sidenote it's no big deal to fix it on your own in hibernate's code. One can tweak the first level caching system or the 2nd level cache (a very little bit harder) to cache negative hits as well by using, for instance, a dummy static instance used to identify negative hits.

We'll know look into a third non-problematic case.

Case 3 : instances of both Entity A and Entity B are loaded which reference the same Entity C instance.Entity B (with not-found="ignore") loaded first

In this case we first load an instance of Entity B, where the relation is defined with not-found="ignore", and then an instance of Entity A where the relation is defined with not-found="exception" is loaded. Both these two instances reference the very same instance of Entity C.

This goes fine. When the instance of Entity B is loaded, hibernate searches the database for the target instance of Entity C. It doesn't exist so hibernate sets null on the relation holder. When the Entity A is loaded, a proxy for Entity C is created and associated to it. If one follows the relation on the java object, hibernate doesn't find the corresponding row and throws an exception, just as expected.

When further instances of Entity A or Entity B are loaded, hibernate still finds a way to behave as expected and associates null values or throws exception when the proxy is accessed, depending on the way the relation has been defined.

We will now look at the buggy case:

Case 4 : instances of both Entity A and Entity B are loaded which reference the same Entity C instance.Entity A (with not-found="exception") loaded first AND NOT USED.

Let's imagine now the following scenario :

We first load an instance of Entity A, where the relation is defined with not-found="exception", and then an instance of Entity B where the relation is defined with not-found="ignore". Again, both these two instances reference the very same instance of Entity C.

At the time the instance of Entity A is loaded, the relation is not followed and thus the proxy is not resolved. Hibernate doesn't know at that stage that the row does not exist.

The problem arises when the instance of Entity B is loaded. Just as usual, as we have seen before, hibernate should go to the database looking for the target row in order to set null on the relation holder if it doesn't exist. But for whatever reason, in this very specific case it does not. It gets screwed by the first level cache which keeps a reference on the proxy created for Entity A.
Instead of looking for the row in the DB and setting null on the relation holder, it finds the proxy in the first level cache and sets it on the relation holder, despite the relation being defined with not-found="ignore".
The corresponding instance of Entity C doesn't exist, but instead of having null on the relation holder, the Entity B instance has the proxy, which is not null at all !!!
When one tries to follow the relation from Entity B, hibernate doesn't have much of a choice but throws an exception, contradicting the not-found="ignore" definition.

This happens just always in this very situation. When two objects reference the same third object with different not-found behaviour, and the one which uses not-found="exception" is loaded first, the second gets the same proxy associated to its relation holder and gets screwed when the program tries to follow the relation. The program tests for nullity and gets the proxy, hence not null, and when it tries to call anything on the proxy, it gets an exception. Screwed.

The problem comes from the fact that the entity loading mechanism in hibernate doesn't keep track of the details of the relation definition when it looks in the first level cache for a relation target. If a proxy exist before any relation defined as not-found="ignore" is loaded, the relations will get the proxy from the first level cache, no matter whether the target row exists or not.

The funny thing here is that if any relation defined as not-found="ignore" has been loaded first, the problem is not triggered. No matter how the subsequent relations are defined, hibernate finds a way to behave correctly and doesn't reuse any proxy for relations defined with not-found="ignore". Everything works correctly.
But in the other case, again, is the proxy is created first before any other lookup for the row in the DB, every relation defined with not-found="ignore" will be screwed.

So how does one workaround this ?

Well there's no silver bullet here.

The very only solution for this is to make sure every relation which might target a table with missing rows or corrupted references must declare the relation with not-found="ignore" and check programmatically within the Business Object whether this is a problem or not. If it is, the business object throws an exception on his own.

I'm pretty sure this problem has not been experienced by many hibernate users so far as I haven't been able to find any discussion about it on the various hibernate forums or expert exchange or so.
This makes sense to me as I assume most hibernate users uses it on a modern Database with well defined referential integrity constraints and the whole valuable shebang modern databases can bring to a project.

But I'm working on a very big project where we have mapped hundred of db tables almost directly in our business object model using hibernate, each of these objects having a lot of relations together and most of them suffering from corrupter references or simply missing rows in the target table.

For us, this problem has been difficult to spot and we are still looking for a better workaround that the one describe above to handle it. If anyone has one, I'm a buyer.

As a sidenote, I cannot provide you here with any test case or program related to this issue as I am experiencing it at work and I have non-divulgation clause linking myself to my employer. And I don't really have the required time to reproduce it and a specific test case for the moment.
However I might sooner or later open a case for this in the hibernate bug tracking system, so if this problem interests you, you might want to keep looking there sometimes.