VJET for ExtJS


ExtJS 4 is a great framework for creating rich internet application. However there is a big problem: it’s a huge productivity drain. Modern development tools are complete with features to help make the code writing experience as productive as possible. In particular, code completion. This post is about a project to add ExtJS code completion to VJET a plugin for Eclipse which provides a means to add type information to Javascript.


Productivity

Because it’s a dynamic language, Javascript editors are not too helpful at the best of times because its tough to compute reliable type inference which is require to present code completion proposals. However the structure of ExtJS code means hard becomes pretty much impossible. At least when writing ‘normal’ JS an editor can offer a list of defined functions and instantiated objects. Because ExtJS does not use conventional means to:

define: Ext.define(‘My.Class.Name’, {…});

and instantiate: Ext.create(‘My.Class.Name’, {…});

objects, editors have nothing reasonable to present to the programmer. Moreover, classes are referenced using strings. Spket offers some release because it knows something about ExtJS but the functionality it offers is nothing like that which is available for typed languages like C# and Java or even just Javascript whether one uses Eclipse or Visua Studio. Ditto Aptana.

Because there is no meaningful code completion for EtJS an eidetic memory is really handy. Either that or its necessary to reference the documentation constantly. This is time consuming and productivity sapping. It makes using an IDE little better than using a text editor.

Architect

Sencha is trying to address some of the productivity problem by releasing versions of Architect. This tool is a great asset as it allows the programmer to see an application’s layout translated to good ExtJS code. However it offers almost no code support and, not surprisingly, there’s still a lot of code to write once the design is completed.

VJET

This is an interesting tool. It’s an add-in to Eclipse that reads type information added to JS code by a programmer and is able to use that information to present useful code completion proposals. VJET type information can added in-line or as type libraries. The current VJET package includes type libraries for JS libraries like JQuery.

VJET is not alone in this regard. Visual Studio also allows information to be added to JS code which is used by VS to present code complete information. However neither has mechanisms that allow code completion for ExtJS to be presented because of the way ExtJS code is written. The advantage of VJET is that its an open source add-in so amenable to change. That is, changed so it can be modified to support ExtJS.

VJET and Sencha

At a recent event in San Fransisco Sencha made public its intention to create and extension or modify VJET to support ExtJS. The announced timescale is “3rd Quarter”. My expectation is that this will come to nothing soon. Sencha is notoriously unable to hit any deadline. Either this project has not yet started or is not being done by Sencha. If it is started, it is not being done in public so it’s not possible to guage how it’s progressing.

Also, addressing coding productivity is not in Sencha’s current sweet spot. It’s clear their emphasis is on mobile development right now not enterprise development. It’s unlikely that much coding goodness will be added to the Architect, not be cause there’s no will but because the Architect is a Qt application so does not have the infrastructure built-in to support code parsing, type inference and code completion.

VJET for ExtJS

My plan is to modify VJET to support ExtJS. As mentioned above this is more than providing a type library (though providing one is an important part of the project). Modifications to the VJET code are required to recognize the use of specific ExtJS functions and react so that correct type information is available. For example, when a programmer defines a class:

Ext.define('My.class.name', {});

I want the class My.class.name to be added to the type cache automatically and for a corresponding configuration type to be added for the config parameter. In this way, in the list of code completion proposals generated for Eclipse to present will include the class name. Also, the class properties and functions will appear as proposals when it is appropriate.

When a programmer creates a class instance:

Ext.create('My.class.name', {});

I want it to be possible that relevant classes are proposed and that on selection of a class both the class name string is generated and the configuration object is validated.

Where it will become more complicated is validating the config object. The list of valid fields in the config object varies by class being created. But more than this, the config object will reference potentially many sub-classes. For example, when a programmer enters ‘xtype’ and requests code completion proposals, Eclipse will show a list of relevant classes and add a text string for the selected class. When the programmer has selected a class it will be ideal that the subsequent fields are interpreted as being valid for the class selected not, say, the outer object(s). It will also be ideal that when a user enters the name of a field like ‘layout’ (and assuming it is used in an appropriate context) the programmer is prompted with a list the appropriate valid alternative layouts.

How does VJET hang together

These are notes for my own benefit. They are my understanding of the subject so are likely to be incorrect or, at the very least, incomplete.

VJET is relies the Dynamic Languages Toolkit (DLTK – http://www.eclipse.org/dltk/) which, in turn is based on the Plugin Development Environment (PDE – http://www.eclipse.org/pde/). These two projects provide a huge range of services on which VJET relies but, even so, VJET provides a lot of functionality.

When a programmer changes code, Eclipse and the PDE is responsible for grabbing the code and calling methods on defined interfaces of installed plugins registered to handle a given file type. The install process registers VJET and, so, the DLTK as handler for .js files. In this way the PDE is able to rely on interfaces implemented by the DLTK which in turn uses VJET classes to get the necessary work done. Eclipse and the PDE is also responsible for initiating the required action when a programmer pressed the ‘.’ character to request a list of code completion proposals.

When a plugin is created the author is able to override methods of base PDE classes to create specific visual artifacts or, as they are called in PDE lingo ‘views’. Code editors are views, a workspace navigator is a view, a code outliner is a view, etc. Views request data to display and respond to events. For example the code editor responds to keyboard events. It displays the code and any ‘markers’ to show relevent errors and warning. The ‘registered class’ is VjetUIPlugin which provides links to all the VJET code and defines the file patterns for which VJET code will react.

The PDE provides base classes which can be sub-classed by a plugin developer so they are able to display the information the plugin author wants to display. By way of an example, here’s the class hierarchy of the VJET editor view:

AbstractDecoratedTextEditor (PDE)<br />
        ScriptEditor (DLTK)<br />
                JavaScriptEditor (DLTK)<br />
                        VjoEditor (VJET)

Not surprisingly, the code editor is an important view. Obviously it displays code but it’s real importance is that the AbstractDecoratedTextEditor implements the PDEs Damager/Repairer mechanism which is used by al Eclipse code editors to inspect code in the editor and return a list of any problems. This is overridden in the DLTKs ScriptEditor so the ‘reconciler’ mechanism exposed by the DLTK can be used to handle the vagaries of dynamic languages. ScriptEditor also provides support for code folding, semantic highlighting, code folding, etc. The DLTK provides a specific editor for JavaScript but this is a minimal implementation to apply, for example, JavaScript specific configuration settings. VJET uses and expands on the DLTK JavaScript class to add context menus, hold editing state and so on.

The PDE provides the damager/repairer classes and VJET overrides these by implementing a ‘reconciler’ and registering it by calling VjoSourceViewerConfiguration.getPresentationReconciler(). This method follows the PDE rules to define a ‘partitioner’ and provide parsers to scan different sections of a code file (code, single line comments, multi-line comments, etc). These parsers and the reconcilers is where the heavy lifting is done.

So what is VJET reconciling? The DLTK JavaScript implementation provides a JavaScript parser which takes the source and converts it into an abstract syntax tree (AST) graph. From this the DLTK is able to validate the code, generate a list of problems to provide code highlights on-screen. But this is not enough for VJET. It must also scan single and mult-line comments in the code to extract type information. Then, crucially, it must reconcile the AST nodes with the type information to create a graph of typed nodes. These nodes mirror AST nodes but are based on the class BaseJstNode.

When code is edited the reconciler starts and runs though the process: parse (to AST) translate (to grab comment and type information), reconcile (to generate a final set of type

  1. parse: Create an AST
  2. translate: Grab comments and type information to create an initial type structure
  3. reconcile: Generate a final set of types which accommodate scoping, assignments and other functional implications
  4. validate: Generated a list of errors and warnings implied by the reconciled types

The most immediately interesting class to me is JstExpressionTypeLinker. It’s in this class that many of the Jst nodes are created which, of course, means adding type information to the type cache. It is the location calls to Ext.define() or Ext.create() can be identified, analyzed and appropriate types constructed.

When requesting code completion proposals the process starts with the same three steps. However instead of validating, the process evaluates the types generated to create and present a list of proposals.

Installing the source code

I have found the notes on the VJET web site (http://www.ebayopensource.org/wiki/display/VJET/Building+from+Source) to be accurate. The Maven build framework allows you to generate an installable package while using the Eclipse option does allow you to debug the code. My experience is that many of the tests do not work so I found it necessary to remove from the Maven builds by removing references to the test projects from the respective pom.xml file and explicitly not importing them into an Eclipse workspace.

Debugging

My biggest hurdle was learning Eclipse at the same time as installing the source code. For example, I was not sure which of all the classes and packages should be the ‘debug’ project. As it turns out, none of them. Once all the projects are added to a workspace and a debug configuration is configured, you are go to go.

There are a *lot* of projects and, so, a huge number of class but precious little documentation about the structure of the code. Learning about the PDE and the DLTK is a good start because VJET has to extend and integrate with these projects. That is, learning how a plugin is created and structured and learning how to go about creating a new language editor. There are good tutorials about both topics available from the respective project’s home page.

But these are just the price of entry. As far as I am concerned, there are two principal blocks of code: parsing the source to generate types; and generating code completion proposals.

Much of the key code is to be found in the org.ebayopensource.vjet.core.jstojava project. This project includes packages for the parser, translator and resolver. The validation rules used to generate errors and warnings are in the org.ebayopensource.vjet.core.jsgenshared project. Classes in these projects and packages are called from classes in other projects. For example the process initiated when code is changed is mediated by VjoParserToJstAndIType.parse() in the package org.ebayopensource.vjet.eclipse.core.parser in the project org.ebayopensource.vjet.eclipse.core. If you want to see what’s going on when the programmer changes code you could start by setting a break point in the parse method.

If you want to see what’s going on when the programmer requests code completion proposals you could do worse that setting a break point here:

org.ebayopensource.vjet.eclipse.internal.ui.text.completion.VjoTypeCompletionProposalComputerBaseJst.computeScriptCompletionProposals()

To begin looking at what occurs as the nodes and types are reconciled, a break point here is not bad:

org.ebayopensource.dsf.jstojava.controller.JstExpressionTypeLinkerTraversal<T>.accept()

This method establishes a scope for the type (if needed) and links node and type. As far as I can see, this is where the magic happens. For example, its the place to catch Ext.define and process it. This method will call ‘postVisit’ which will determine the node type of each AST node and, in the case of Ext.define dispatch it to be handled by:

org.ebayopensource.dsf.jstojava.controller.JstExpressionTypeLinker.postVisitMtdInvocationExpr(MtdInvocationExpr)

It is in this method that the type for the input method expression node graph is resolved and where a new type might be created and added to the type cache for use by code validation, syntax highlighting and code completion routines.

Adding types

The code below illustrates how the classname referenced in an Ext.define() function instance might be added to the type cache.

if (mtdKey.equals("Ext") && mtd.getName().getName().equals("define"))<br />
{<br />
        List<IExpr> args = mie.getArgs();<br />
        String typeName = args.get(0).toExprText();<br />
        // Check this is a string.  May need to follow a class back to a string<br />
        typeName = typeName.substring(1, typeName.length()-1);<br />
        typeName = mtdKey + "." + typeName;</p>
<p>     // Get the type space manager<br />
        JstTypeSpaceMgr m_tsMgr = m_resolver.getController().getJstTypeSpaceMgr();<br />
        org.ebayopensource.dsf.ts.ITypeSpace<IJstType,IJstNode> m_ts = m_tsMgr.getTypeSpace();</p>
<p>     String groupName = mtd.getRootType().getPackage().getGroupName();<br />
        IJstType oldType = m_ts.getType(new TypeName(groupName, typeName));<br />
        if (oldType == null)<br />
        {<br />
                // Look for predefined types 'void' and 'String'<br />
                IJstType voidtype = m_ts.getType(new TypeName(JstTypeSpaceMgr.JAVA_PRIMITIVE_GRP, "void"));<br />
                IJstType stringtype = m_ts.getType(new TypeName(JstTypeSpaceMgr.JS_NATIVE_GRP, "String"));</p>
<p>             JstArg arg1 = null;</p>
<p>             JstType t =JstFactory.getInstance().createJstType(typeName, true);<br />
                t.setImpliedImport(true);<br />
                JstMethod c = new JstConstructor();<br />
                c.setDoc(new JstDoc("This is a comment about the default constructor"));<br />
                c.setParent(t);<br />
                c.getModifiers().merge(JstModifiers.PUBLIC);<br />
                t.setConstructor(c);</p>
<p>             // Here's an example of including another constructor<br />
                c.addOverloaded(c);</p>
<p>             JstMethod co = new JstConstructor(new JstModifiers(JstModifiers.PUBLIC),<br />
                                new JstArg(stringtype, "arga", false)<br />
                );<br />
                co.setDoc(new JstDoc("This is a comment about the alternate constructor"));</p>
<p>             c.addOverloaded(co);</p>
<p>             // The method addType is not part of the standard VJET release.<br />
                // Ideally TypeDependencyMgr.addType() would be called but this<br />
                // type is package private.  So instead method addType has been<br />
                // added to JstTypeSpaceMgr.  This method calls TypeDependencyMgr.addType();<br />
                m_tsMgr.addType(new TypeName(groupName, typeName), t);<br />
        }<br />
}

Information and Links

Join the fray by commenting, tracking what others have to say, or linking to it from your blog.


Other Posts

Reader Comments

Sorry, comments are closed.