The More Things Slow Down, The More They Heat Up

Well, it's that time of year again. The big push to get CDT 5.0 out the door is essentially over. Ganymede bits are being uploaded to the servers by the various projects. Things are looking good. It seems like it's just about time to finally sit out on a patio somewhere with a nice frosty beverage and relax.

Or is it?

Firstly, I'm busy prepping to participate again this year in the Ride for Sight, which is the longest running motorcycle charity in Canada. It's going to be a full day this coming Saturday, with many hours of riding, and I have to make sure all my gear is in order. And also, it's the final stretch of fundraising, so I'm busily annoying all my friends. (Shameless plug... to find out how you can donate, go to my donation page ). Hopefully this weekend I won't forget my hat like I did at the Port Dover Friday the 13th Rally last weekend and get sunburnt...

But more on topic, my team still has a lot of work going on right now. Right now most of us are working on Remote Development Tools, which is not on the Ganymede train. We just contributed our first initial drop of our C/C++ remote indexing tools to Bugzilla for consideration, and are still working hard to get more done so that we can get to the point where people might actually start using this stuff.

So... what is this stuff?

Essentially, we are trying to create a development environment where you can run your IDE on your local workstation, but the actual code resides on another target machine. Maybe this machine is a mainframe with no graphical windowing capabilities, maybe it's a gigantic supercomputer that you don't physically have on your desk (or if you did, you'd need a REALLY big desk...). In any case, the code you're actually working on resides somewhere that is not local.

Most of the most exciting value adds provided by Eclipse compared to other development environments require knowledge of the structure of the user's source code. Features such as source code navigation, content assist, intelligent search, call hierarchy view, type hierarchy view, the include browser, refactoring, and other features all require parsing the user's source code and producing an index which allows for name based lookups of source code elements.

Parsing and indexing are both very CPU and memory intensive operations, and good performance is a key requirement if these features are to be used by the average user. The remote scenario provides for some unique, additional challenges which have to be overcome in order for the features to work quickly and correctly.

Some important points to consider:

• Network mounting the files and operating on them "locally" has been proven to be slow, even on fast (100 megabit) connections with very few intermediate hops.
• Downloading the entire set of remote files (both project files and included system headers, which are not generally found on the local machine) is similarly slow.
• Sometimes the remote machine uses a different text codepage encoding than the local machine. This means that not only must the source files be transferred, but they may have to undergo a codepage conversion process, which slows things down even further.
• Abstract Syntax Trees (ASTs) and indices are typically much larger than the original source code from which they are produced, because they store much more information. I.e., they store a certain amount of syntactic and/or semantic understanding, which is inherently more information than is imparted by the raw bytes that correspond to the source text. As such, it's even more impractical to transfer ASTs or indices than it is to just transfer the original source.
• The way a file needs to be parsed in order to be interpreted correctly is often dependent upon the manner in which the file is compiled. E.g., macros and include paths may be defined/redefined on the command line of any individual compilation of any individual file. A successful parse requires that those same macros and include paths be provided to the parser when it runs.
• Often the remote machine has greater CPU power than the local machine, so it can often complete parsing and indexing tasks more quickly than the local machine.
• Remote machines are often accessed at geographically separated locations. The intermediate topology of the network can often be complicated, with many hops, and slow links. As such, in order to maintain performance it's important for as little data as possible be transferred back and forth between the local machine and the remote machine.

As such, we feel that if the Remote Development Tools are to be successful, then they must provide remote services that allow the user to do all of the parsing and indexing on the remote machine. The local machine can query the remote machine for data it is interested in, and only this data gets transferred over the remote connection.

So, that's the motivation. We just contributed a framework and reference implementation that implements the following features for C/C++:

• A New Remote C/C++ Project wizard that allows you to create remote projects and configure your service model (files are served by EFS)
  
• Integrated index lifecycle management
• Automatic source code delta handling (the index is automatically updated when files in your project are added/removed/changed)
• Remote Search
• Remote Call Hierarchy
• Remote Navigation (e.g. Go To Declaration)

Other planned services include:

• Content Assist
• Type Hierarchy
• Include browser
• Model builder

But that's not all. Currently I'm working on a remote make builder, which will essentially let you farm out the build commands for your project over the protocol of your choice (e.g. SSH), scan the build output for errors (like we already do in CDT), and map those back to the EFS resources in your project so that you can do standard things like click on errors in the Problems View from your remote compile and be whisked away to the corresponding source location.

The builder in fact is probably the most important feature to have. Really the "I" in IDE isn't there if you can't build anything. Source introspection tools are nice, but if your tool can't build my source, chances are I'm not going to use it.

At any rate... it's looking like it's going to be a busy summer...

e4: Now With More Openness!

Well I haven't been blogging too much since EclipseCon. I came home from EclipseCon with a nasty bout of pneumonia which kept me on the bench for a bit, and ever since I've been back in the game I've been working flat out on the Remote Development Tools (RDT) initiative. (So flat out in fact that the wiki page that link goes to is woefully out of date. ) A lot of late hours and many Red Bulls later, we've got a first proof of concept demo up and running on the new RDT framework in time for our upcoming deadline, so I can finally take a bit of time again for things like attending meetings and blogging about them.

I spent Thursday and Friday last week at the e4 Summit. I won't bore everyone with all the technical details, as those can be found on the wiki page, but I wanted to make a point to express some serious kudos to everyone that made this meeting happen, especially given my past criticisms of openness in the platform.

The platform is entering a new era of openness. Seeing thirty people from thirteen different companies/organizations all working together under the scrutiny of the public eye to help design and build the next generation of the Eclipse Platform is a wonderful sight to behold. All the discussion and decisions are being made in the open where anyone can participate, whether it's at the Summit, on the mailing lists, or on the upcoming project conference calls. All the people who attended the summit are going to receive commit rights (if they want them) under the aegis of the new incubator project (which is not the old incubator project).

So, massive props to the Platform Team and everyone else involved. You are doing a great thing right now, and you deserve to be recognized for it. Good on ya!

We've got a lot of momentum going right now on e4. Great opportunities are opening up for everyone to participate in something special here, whether it's on the new browser-based-Eclipse stuff, or working or a new resource model, or something else. If you think there is something that should be a part of e4 that isn't, now is your time to speak up and contribute.

I want to emphasize the word contribute as well. People have harped on the Platform for years about it being hard to contribute to. Now that things are changing in that respect, the onus is on the rest of the community to put their code where their mouth is. Being open is not the silver bullet to build a thriving, diverse committer community. Ultimately it takes people writing code, and you could be one of those people. If you want to see something happen, pick up a keyboard and lend a hand, any way you can. The Platform folks are doing their part in opening things up, now it's the community's turn to take advantage of the opportunity. Don't waste it.

Sneak Peek: Eclipse In The Browser

The whole e4 thing about being able to run Eclipse in a browser with AJAX is really exciting. What most people don't know is that there is another super secret aspect of e4 that's being worked on right now that will allow people to bring Eclipse to just about everywhere, which I think is even more exciting than AJAX. That work is the new HTML port of SWT.

Some screenshots are in order:

Here's a shot of an SWT example, running natively:


Here's a shot of the same example running under the HTML port (click for a larger view with more detail):


The awesome thing about the HTML port is it can be run in ANY browser, even on mobile or other embedded devices. The secret, if you zoom in really really far, is the following:


That's right. The graphics are actually being rendered as HTML text, and use the colour settings of the font to help render the graphics. The beauty of this is that this theoretically works with any display capable of colour text. This makes it possible to run RCP apps on a whole host of new places that previously didn't have the graphical horsepower to run SWT or eSWT. Perhaps we'll even see a text mode port of SWT eventually that won't even require a browser!

Truly this is an exciting time for Eclipse!

EclipseCon Poker Shameless Tactic

Alright, this is pretty shameless but...

I am collecting the BEA cards in Eclipse Poker. Will trade a Jack or King of hearts. Comment below as to what you've got that you're willing to trade (BEA only please).

e4 Is News To Me Too

Just read about the entire e4 debacle (thanks Doug Schaefer). I don't have a whole lot new to say other than I completely agree with Doug Gaff and Doug S on this one (see the thread on the planning council mailing list for details). The fact that that this has been done totally below the radar really irks me to put it mildly.

An interesting point for me to make though, is that those of us in IBM aren't really any more in the loop than the rest of you. Other than some vague "we're thinking of doing an Eclipse 4.0 and thinking about what that might look like" comments (which really weren't at all surprising and didn't really tell me anything that I would consider super-secret advanced knowledge), I haven't heard diddly about any of this e4 stuff. I'm just as blindsided as the rest of you, and I actually interact with folks on the platform occasionally. I have no more clue about e4 than what was in the component announcement.

For the record, the people on the platform are good people (some of them are even my friends). They certainly don't mean ill to anyone, and that certainly includes the Eclipse community. They do not sit in their Evil Eclipse Platform Lair, wringing their hands and cackling maniacally while they try to figure out the next way to thwart openness and diversity. They want to be open and diverse, they really do. Some of them really try very hard to be, and some of them are doing a good job of it (kudos to the platform debug team particularly!).

But, I have to be critical here. A lot of them just don't seem to get it when it comes to this whole open/transparent/diverse thing. I think it's good that a call for participation is being made now, but with all due respect, this shouldn't be the first time we're hearing about all this. What is this e4 thing? Where were the discussions that led up to this? Who participated? What requirements went into it? How were requirements solicited? Did you do a requirements doc? A design doc? Were those documents reviewed with community? Any meetings? Conference calls? Could people join those calls? Where are the minutes? These are all things I've ranted about on this blog before, but I haven't seen a whole lot of progress.

The perception is that once again the IBM skunkworks went off and did its thing, their reasons known only to themselves, and are now just presenting us e4 as a fait accompli. I don't believe that to actually be true, and I very much believe that they're open to participation going forward. And to be honest, I'm going to make a bit of a leap of faith here and assume that whatever e4 is, it's not going to annihilate our Eclipse universe as we know it and doom us all. They platform people are very smart, and they value the community greatly.

However, people need to take some ownership here and acknowledge that there is a PROBLEM. The entire process that led us up to this point was closed. THAT is the problem that we need to solve. Until the Platform starts opening up their process, the Platform will never truly be open. And that's a problem for all of us, whether you work for Big Blue or not. The lack of openness hurts people in IBM like me just as much as the rest of you.

At any rate, I hope the powers that be use this as a learning experience.

Security And The Eclipse Board Elections

So, I just voted in the Eclipse Board elections. I encourage everyone entitled to a vote to do the same.

I'm not feeling the greatest about the security of the whole thing though. My password to login to the voting system was sent via email, which is not a very secure protocol given that the contents are sent out in the clear. Why couldn't I just login using my SSH credentials, which are relatively secure? Granted, the likelihood that someone is sniffing committers' email in an attempt to fraudulently login and rig the board elections is pretty low, but it's the principle of the thing.

Even worse, was that after I voted, the system told me to expect a confirmation email. Ok, great. Except, the confirmation email contains a listing of how I voted for all the candidates. So much for secret ballots. Oy.