Sizeof for Java page1

来源：互联网发布：经济学书籍推荐知乎编辑：程序博客网时间：2024/06/11 03:03

Does Java have an operator like sizeof() in C?

A superficial answer is that Java does not provide anything like C's sizeof(). However, let's consider why a Java programmer might occasionally want it.

A C programmer manages most datastructure memory allocations himself, and sizeof() is indispensable for knowing memory block sizes to allocate. Additionally, C memory allocators like malloc() do almost nothing as far as object initialization is concerned: a programmer must set all object fields that are pointers to further objects. But when all is said and coded, C/C++ memory allocation is quite efficient.

By comparison, Java object allocation and construction are tied together (it is impossible to use an allocated but uninitialized object instance). If a Java class defines fields that are references to further objects, it is also common to set them at construction time. Allocating a Java object therefore frequently allocates numerous interconnected object instances: an object graph. Coupled with automatic garbage collection, this is all too convenient and can make you feel like you never have to worry about Java memory allocation details.

Of course, this works only for simple Java applications. Compared with C/C++, equivalent Java datastructures tend to occupy more physical memory. In enterprise software development, getting close to the maximum available virtual memory on today's 32-bit JVMs is a common scalability constraint. Thus, a Java programmer could benefit from sizeof() or something similar to keep an eye on whether his datastructures are getting too large or contain memory bottlenecks. Fortunately, Java reflection allows you to write such a tool quite easily.

Before proceeding, I will dispense with some frequent but incorrect answers to this article's question.

Fallacy: Sizeof() is not needed because Java basic types' sizes are fixed
Yes, a Java int is 32 bits in all JVMs and on all platforms, but this is only a language specification requirement for the programmer-perceivable width of this data type. Such an int is essentially an abstract data type and can be backed up by, say, a 64-bit physical memory word on a 64-bit machine. The same goes for nonprimitive types: the Java language specification says nothing about how class fields should be aligned in physical memory or that an array of booleans couldn't be implemented as a compact bitvector inside the JVM.

Fallacy: You can measure an object's size by serializing it into a byte stream and looking at the resulting stream length
The reason this does not work is because the serialization layout is only a remote reflection of the true in-memory layout. One easy way to see it is by looking at how Strings get serialized: in memory every char is at least 2 bytes, but in serialized form Strings are UTF-8 encoded and so any ASCII content takes half as much space.

Another working approach
You might recollect "Java Tip 130: Do You Know Your Data Size?" that described a technique based on creating a large number of identical class instances and carefully measuring the resulting increase in the JVM used heap size. When applicable, this idea works very well, and I will in fact use it to bootstrap the alternate approach in this article.

Note that Java Tip 130's Sizeof class requires a quiescent JVM (so that the heap activity is only due to object allocations and garbage collections requested by the measuring thread) and requires a large number of identical object instances. This does not work when you want to size a single large object (perhaps as part of a debug trace output) and especially when you want to examine what actually made it so large.

What is an object's size?
The discussion above highlights a philosophical point: given that you usually deal with object graphs, what is the definition of an object size? Is it just the size of the object instance you're examining or the size of the entire data graph rooted at the object instance? The latter is what usually matters more in practice. As you shall see, things are not always so clear-cut, but for starters you can follow this approach:

An object instance can be (approximately) sized by totaling all of its nonstatic data fields (including fields defined in superclasses)
Unlike, say, C++, class methods and their virtuality have no impact on the object size
Class superinterfaces have no impact on the object size (see the note at the end of this list)
The full object size can be obtained as a closure over the entire object graph rooted at the starting object

Note: Implementing any Java interface merely marks the class in question and does not add any data to its definition. In fact, the JVM does not even validate that an interface implementation provides all methods required by the interface: this is strictly the compiler's responsibility in the current specifications.

To bootstrap the process, for primitive data types I use physical sizes as measured by Java Tip 130's Sizeof class. As it turns out, for common 32-bit JVMs a plain java.lang.Object takes up 8 bytes, and the basic data types are usually of the least physical size that can accommodate the language requirements (except boolean takes up a whole byte):

// java.lang.Object shell size in bytes: public static final int OBJECT_SHELL_SIZE = 8; public static final int OBJREF_SIZE = 4; public static final int LONG_FIELD_SIZE = 8; public static final int INT_FIELD_SIZE = 4; public static final int SHORT_FIELD_SIZE = 2; public static final int CHAR_FIELD_SIZE = 2; public static final int BYTE_FIELD_SIZE = 1; public static final int BOOLEAN_FIELD_SIZE = 1; public static final int DOUBLE_FIELD_SIZE = 8; public static final int FLOAT_FIELD_SIZE = 4;

    // java.lang.Object shell size in bytes:
    public static final int OBJECT_SHELL_SIZE   = 8;

    public static final int OBJREF_SIZE         = 4;
    public static final int LONG_FIELD_SIZE     = 8;
    public static final int INT_FIELD_SIZE      = 4;
    public static final int SHORT_FIELD_SIZE    = 2;
    public static final int CHAR_FIELD_SIZE     = 2;
    public static final int BYTE_FIELD_SIZE     = 1;
    public static final int BOOLEAN_FIELD_SIZE  = 1;
    public static final int DOUBLE_FIELD_SIZE   = 8;
    public static final int FLOAT_FIELD_SIZE    = 4;

(It is important to realize that these constants are not hardcoded forever and must be independently measured for a given JVM.) Of course, naive totaling of object field sizes neglects memory alignment issues in the JVM. Memory alignment does matter (as shown, for example, for primitive array types in Java Tip 130), but I think it is unprofitable to chase after such low-level details. Not only are such details dependent on the JVM vendor, they are not under the programmer's control. Our objective is to obtain a good guess of the object's size and hopefully get a clue when a class field might be redundant; or when a field should be lazily populated; or when a more compact nested datastructure is necessary, etc. For absolute physical precision you can always go back to the Sizeof class in Java Tip 130.

To help profile what makes up an object instance, our tool will not just compute the size but will also build a helpful datastructure as a byproduct: a graph made up of IObjectProfileNodes:

interface IObjectProfileNode { Object object (); String name (); int size (); int refcount (); IObjectProfileNode parent (); IObjectProfileNode [] children (); IObjectProfileNode shell (); IObjectProfileNode [] path (); IObjectProfileNode root (); int pathlength (); boolean traverse (INodeFilter filter, INodeVisitor visitor); String dump (); } // End of interface

interface IObjectProfileNode
{
    Object object ();
    String name ();

    int size ();
    int refcount ();

    IObjectProfileNode parent ();
    IObjectProfileNode [] children ();
    IObjectProfileNode shell ();

    IObjectProfileNode [] path ();
    IObjectProfileNode root ();
    int pathlength ();

    boolean traverse (INodeFilter filter, INodeVisitor visitor);
    String dump ();

} // End of interface

IObjectProfileNodes are interconnected in almost exactly the same way as the original object graph, with IObjectProfileNode.object() returning the real object each node represents. IObjectProfileNode.size() returns the total size (in bytes) of the object subtree rooted at that node's object instance. If an object instance links to other objects via non-null instance fields or via references contained inside array fields, then IObjectProfileNode.children() will be a corresponding list of child graph nodes, sorted in decreasing size order. Conversely, for every node other than the starting one, IObjectProfileNode.parent() returns its parent. The entire collection of IObjectProfileNodes thus slices and dices the original object and shows how data storage is partitioned within it. Furthermore, the graph node names are derived from the class fields and examining a node's path within the graph (IObjectProfileNode.path()) allows you to trace the ownership links from the original object instance to any internal piece of data.

You might have noticed while reading the previous paragraph that the idea so far still has some ambiguity. If, while traversing the object graph, you encounter the same object instance more than once (i.e., more than one field somewhere in the graph is pointing to it), how do you assign its ownership (the parent pointer)? Consider this code snippet:

Object obj = new String [] {new String ("JavaWorld"), new String ("JavaWorld")};

Object obj = new String [] {new String ("JavaWorld"),
new String ("JavaWorld")};

Each java.lang.String instance has an internal field of type char[] that is the actual string content. The way the String copy constructor works in Java 2 Platform, Standard Edition (J2SE) 1.4, both String instances inside the above array will share the same char[] array containing the {'J', 'a', 'v', 'a', 'W', 'o', 'r', 'l', 'd'} character sequence. Both strings own this array equally, so what should you do in cases like this?

If I always want to assign a single parent to a graph node, then this problem has no universally perfect answer. However, in practice, many such object instances could be traced back to a single "natural" parent. Such a natural sequence of links is usually shorter than the other, more circuitous routes. Think about data pointed to by instance fields as belonging more to that instance than to anything else. Think about entries in an array as belonging more to that array itself. Thus, if an internal object instance can be reached via several paths, we choose the shortest path. If we have several paths of equal lengths, well, we just pick the first discovered one. In the worst case, this is as good a generic strategy as any.

Thinking about graph traversals and shortest paths should ring a bell at this point: breadth-first search is a graph traversal algorithm that guarantees to find the shortest path from the starting node to any other reachable graph node.

After all these preliminaries, here is a textbook implementation of such a graph traversal. (Some details and auxiliary methods have been omitted; see this article's download for full details.):

    public static IObjectProfileNode profile (Object obj)
    {
        final IdentityHashMap visited = new IdentityHashMap ();

        final ObjectProfileNode root = createProfileTree (obj, visited,
                                                          CLASS_METADATA_CACHE);
        finishProfileTree (root);

        return root;
    }

    private static ObjectProfileNode createProfileTree (Object obj,
                                                        IdentityHashMap visited,
                                                        Map metadataMap)
    {
        final ObjectProfileNode root = new ObjectProfileNode (null, obj, null);

        final LinkedList queue = new LinkedList ();

        queue.addFirst (root);
        visited.put (obj, root);

        final ClassAccessPrivilegedAction caAction =
            new ClassAccessPrivilegedAction ();
        final FieldAccessPrivilegedAction faAction =
            new FieldAccessPrivilegedAction ();

        while (! queue.isEmpty ())
        {
            final ObjectProfileNode node = (ObjectProfileNode) queue.removeFirst ();

            obj = node.m_obj;
            final Class objClass = obj.getClass ();

            if (objClass.isArray ())
            {
                final int arrayLength = Array.getLength (obj);
                final Class componentType = objClass.getComponentType ();

                // Add shell pseudo-node:
                final AbstractShellProfileNode shell =
                    new ArrayShellProfileNode (node, objClass, arrayLength);
                shell.m_size = sizeofArrayShell (arrayLength, componentType);

                node.m_shell = shell;
                node.addFieldRef (shell);

                if (! componentType.isPrimitive ())
                {
                    // Traverse each array slot:
                    for (int i = 0; i < arrayLength; ++ i)
                    {
                        final Object ref = Array.get (obj, i);

                        if (ref != null)
                        {
                            ObjectProfileNode child =
                                (ObjectProfileNode) visited.get (ref);
                            if (child != null)
                                ++ child.m_refcount;
                            else
                            {
                                child = new ObjectProfileNode (node, ref,
                                    new ArrayIndexLink (node.m_link, i));
                                node.addFieldRef (child);

                                queue.addLast (child);
                                visited.put (ref, child);
                            }
                        }
                    }
                }
            }
            else // the object is of a non-array type
            {
                final ClassMetadata metadata =
                    getClassMetadata (objClass, metadataMap, caAction, faAction);
                final Field [] fields = metadata.m_refFields;

                // Add shell pseudo-node:
                final AbstractShellProfileNode shell =
                    new ObjectShellProfileNode (node,
                                                metadata.m_primitiveFieldCount,
                                                metadata.m_refFields.length);
                shell.m_size = metadata.m_shellSize;

                node.m_shell = shell;
                node.addFieldRef (shell);

                // Traverse all non-null ref fields:
                for (int f = 0, fLimit = fields.length; f < fLimit; ++ f)
                {
                    final Field field = fields [f];

                    final Object ref;
                    try // to get the field value:
                    {
                        ref = field.get (obj);
                    }
                    catch (Exception e)
                    {
                        throw new RuntimeException ("cannot get field [" +
                            field.getName () + "] of class [" +
                            field.getDeclaringClass ().getName () +
                            "]: " + e.toString ());
                    }

                    if (ref != null)
                    {
                        ObjectProfileNode child =
                            (ObjectProfileNode) visited.get (ref);
                        if (child != null)
                            ++ child.m_refcount;
                        else
                        {
                            child = new ObjectProfileNode (node, ref,
                                new ClassFieldLink (field));
                            node.addFieldRef (child);

                            queue.addLast (child);
                            visited.put (ref, child);
                        }
                    }
                }
            }
        }

        return root;
    }

    private static void finishProfileTree (ObjectProfileNode node)
    {
        final LinkedList queue = new LinkedList ();
        IObjectProfileNode lastFinished = null;

        while (node != null)
        {
            // Note that an unfinished nonshell node has its child count
            // in m_size and m_children[0] is its shell node:

            if ((node.m_size == 1) || (lastFinished == node.m_children [1]))
            {
                node.finish ();
                lastFinished = node;
            }
            else
            {
                queue.addFirst (node);
                for (int i = 1; i < node.m_size; ++ i)
                {
                    final IObjectProfileNode child = node.m_children [i];
                    queue.addFirst (child);
                }
            }

            if (queue.isEmpty ())
                return;
            else
                node = (ObjectProfileNode) queue.removeFirst ();
        }
    }

This code is a distant relative of the clone-via-reflection implementation used in a past Java Q&A, "Attack of the Clones." As before, it caches reflection metadata to improve performance and uses an identity hashmap to mark visited objects. The profile() method starts by spanning the original object graph with a tree of IObjectProfileNodes in a breadth-first traversal and finishes with a quick post-order traversal that totals and assigns all node sizes. profile() returns a IObjectProfileNode that is the generated spanning tree's root, and its size() is the entire graph's size.

Of course, profile()'s output is useful only if I have a good way to explore it. To this end, every IObjectProfileNode supports examination by a node visitor together with a node filter:

interface IObjectProfileNode { interface INodeFilter { boolean accept (IObjectProfileNode node); } // End of nested interface interface INodeVisitor { /** * Pre-order visit. */ void previsit (IObjectProfileNode node); /** * Post-order visit. */ void postvisit (IObjectProfileNode node); } // End of nested interface boolean traverse (INodeFilter filter, INodeVisitor visitor); ... } // End of interface

interface IObjectProfileNode
{
    interface INodeFilter
    {
        boolean accept (IObjectProfileNode node);

    } // End of nested interface

    interface INodeVisitor
    {
        /**
         * Pre-order visit.
         */
        void previsit (IObjectProfileNode node);

        /**
         * Post-order visit.
         */
        void postvisit (IObjectProfileNode node);

    } // End of nested interface

    boolean traverse (INodeFilter filter, INodeVisitor visitor);

    ...

} // End of interface

A node visitor gets a shot at doing something with a tree node only if the accompanying filter is null or if the filter accepts the node. For simplicity, the node's children are examined only if the node itself has been examined. Both pre- and post-order visits are supported. The size contributions from the java.lang.Object shell plus all primitive data fields are lumped together in a pseudo-node attached to every "real" node representing an object instance. Such shell nodes are accessible via IObjectProfileNode.shell() and also show up in the IObjectProfileNode.children() list: the idea is to be able to write data filters and visitors that consider primitive data overhead on equal footing with instantiable data types.

It is up to you how to implement filters and visitors. As a starting point, the ObjectProfileFilters class (see this article's download) offers several useful stock filters that help prune large object trees based on node size, node size relative to its parent's size, node size relative to the root object, and so on. The ObjectProfilerVisitors class contains the default visitor used by IObjectProfileNode.dump(), as well as a visitor that can create an XML dump for more sophisticated object browsing. It is also easy to turn a profile into a Swing TreeModel.

As an illustration, let's do a full dump of the two-string array object mentioned above:

public class Main { public static void main (String [] args) { Object obj = new String [] {new String ("JavaWorld"), new String ("JavaWorld")}; IObjectProfileNode profile = ObjectProfiler.profile (obj); System.out.println ("obj size = " + profile.size () + " bytes"); System.out.println (profile.dump ()); } } // End of class

public class Main
{
    public static void main (String [] args)
    {
        Object obj = new String [] {new String ("JavaWorld"),
                                    new String ("JavaWorld")};

        IObjectProfileNode profile = ObjectProfiler.profile (obj);

        System.out.println ("obj size = " + profile.size () + " bytes");
        System.out.println (profile.dump ());
    }

} // End of class

This code produces:

obj size = 106 bytes 106 -> <INPUT> : String[] 58 (54.7%) -> <INPUT>[0] : String 34 (32.1%) -> String#value : char[], refcount=2 34 (32.1%) -> <shell: char[], length=9> 24 (22.6%) -> <shell: 3 prim/1 ref fields> 24 (22.6%) -> <shell: String[], length=2> 24 (22.6%) -> <INPUT>[1] : String 24 (22.6%) -> <shell: 3 prim/1 ref fields>

obj size = 106 bytes
  106 -> <INPUT> : String[]
    58 (54.7%) -> <INPUT>[0] : String
      34 (32.1%) -> String#value : char[], refcount=2
        34 (32.1%) -> <shell: char[], length=9>
      24 (22.6%) -> <shell: 3 prim/1 ref fields>
    24 (22.6%) -> <shell: String[], length=2>
    24 (22.6%) -> <INPUT>[1] : String
      24 (22.6%) -> <shell: 3 prim/1 ref fields>

Indeed, as explained earlier, the internal character array (referenced by java.lang.String#value) is shared between both strings. Even though ObjectProfiler.profile() assigns the ownership of this array to the first discovered string, it notices that the array is shared (shown by refcount=2 next to it).