Chris Recoskie's Blog

Do You Hate It When Resources Are Out Of Sync?

2011-02-23T11:00:00.003-05:00

Do messages like this annoy you?

Do you wish Eclipse would just fix the problem, refresh the file, and let you edit it?

If the answers to these questions are "YES" then I invite you to comment/vote on a bunch of Bugzilla entries that we are discussing these issues on.

Bug 14867 - Don't force me to refresh all the time
Bug 303517 - Update IResource sync state in a timely manner when discovered out of sync
Bug 337446 - Text search should handle out-of-sync resources

e4.1 is Fast

2010-10-27T14:26:00.000-04:00

I spent the morning with Eric Moffat from the Eclipse Platform Team, trying out e4.1 with CDT and RDT. We had heard a lot of complaints in the past about e4 being slow, and the Platform Team had assured us that 4.1 was doing a lot to fix that. So, we decided to give it a spin.

Eric was kind enough to sit with us and help us get up and running, which actually wasn't that hard. The compatibility layer in e4 really shines. All the JDT, PDE, and Team stuff that I use on a daily basis just worked. I added some RSE plugins to my workbench via the Indigo update site, checked out the CDT and RDT code from CVS, built it, launched it as a runtime workbench under the debugger, and I was off to the races. I was editing, compiling, and debugging C/C++ projects, both locally on my Windows box using just CDT, and remotely over the network on a Linux/PPC box using RDT.

"So, how was the speed" you ask?

In short, pretty damned fast. I never got around to trying out e4.0, so I can't comment on whether the performance there was lacking or not, but I can tell you that e4.1 feels pretty snappy. Snappier in fact than 3.6 or 3.7. I didn't put a stopwatch on it, but everything seems faster to me. Significantly faster. Startup time seems better. GUI operations all seem faster, whether it's progress monitor updates, drawing large trees full of data, scrolling windows full of content, or resizing areas in the workbench. I was skeptical going in, given the negative reviews of performance I had heard about e4.0, but I have to say that I'm pretty impressed.

The speedups make a lot of sense too once Eric pointed out some subtle things about how the UI has been redone. The workbench now uses far less SWT Composites, and less OS level widgets than it used to. That means less things to process, as well as less memory and less OS level resource consumption. They've redone a lot of other things under the hood that I'm sure are helping too. They're not done yet either. They've got quite a lot of time between now and June to make further improvements.

Now, I don't want to falsely give the impression that there were no problems whatsoever. There was the odd glitch here and there. This stuff is pretty bleeding edge though, and that's rather to be expected. We found a few issues here and there with NullPointerExceptions that didn't used to happen, for example, because the new code lacked some null checks. Eric is taking that stuff back to the team to get fixed though, and we were able to work around them.

So, is it perfect? No. But the future is looking bright, and the speed issue may have just given everyone that was taking the "wait and see" approach to e4 a reason to start looking at it again. Not everyone might care about style sheets, dependency injection, and other goodies, but we all certainly care about speed. Speed is probably the number one thing that those of us working on Eclipse products get beaten up about. The speed of e4.1 might just be the thing that makes it take off.

The More Things Slow Down, The More They Heat Up

2008-06-18T10:11:00.001-04:00

Well, it's that time of year again. The big push to get CDT 5.0 out the door is essentially over. Ganymede bits are being uploaded to the servers by the various projects. Things are looking good. It seems like it's just about time to finally sit out on a patio somewhere with a nice frosty beverage and relax.

Or is it?

Firstly, I'm busy prepping to participate again this year in the Ride for Sight, which is the longest running motorcycle charity in Canada. It's going to be a full day this coming Saturday, with many hours of riding, and I have to make sure all my gear is in order. And also, it's the final stretch of fundraising, so I'm busily annoying all my friends. (Shameless plug... to find out how you can donate, go to my donation page ). Hopefully this weekend I won't forget my hat like I did at the Port Dover Friday the 13th Rally last weekend and get sunburnt...

But more on topic, my team still has a lot of work going on right now. Right now most of us are working on Remote Development Tools, which is not on the Ganymede train. We just contributed our first initial drop of our C/C++ remote indexing tools to Bugzilla for consideration, and are still working hard to get more done so that we can get to the point where people might actually start using this stuff.

So... what is this stuff?

Essentially, we are trying to create a development environment where you can run your IDE on your local workstation, but the actual code resides on another target machine. Maybe this machine is a mainframe with no graphical windowing capabilities, maybe it's a gigantic supercomputer that you don't physically have on your desk (or if you did, you'd need a REALLY big desk...). In any case, the code you're actually working on resides somewhere that is not local.

Most of the most exciting value adds provided by Eclipse compared to other development environments require knowledge of the structure of the user's source code. Features such as source code navigation, content assist, intelligent search, call hierarchy view, type hierarchy view, the include browser, refactoring, and other features all require parsing the user's source code and producing an index which allows for name based lookups of source code elements.

Parsing and indexing are both very CPU and memory intensive operations, and good performance is a key requirement if these features are to be used by the average user. The remote scenario provides for some unique, additional challenges which have to be overcome in order for the features to work quickly and correctly.

Some important points to consider:

Network mounting the files and operating on them "locally" has been proven to be slow, even on fast (100 megabit) connections with very few intermediate hops.
Downloading the entire set of remote files (both project files and included system headers, which are not generally found on the local machine) is similarly slow.
Sometimes the remote machine uses a different text codepage encoding than the local machine. This means that not only must the source files be transferred, but they may have to undergo a codepage conversion process, which slows things down even further.
Abstract Syntax Trees (ASTs) and indices are typically much larger than the original source code from which they are produced, because they store much more information. I.e., they store a certain amount of syntactic and/or semantic understanding, which is inherently more information than is imparted by the raw bytes that correspond to the source text. As such, it's even more impractical to transfer ASTs or indices than it is to just transfer the original source.
The way a file needs to be parsed in order to be interpreted correctly is often dependent upon the manner in which the file is compiled. E.g., macros and include paths may be defined/redefined on the command line of any individual compilation of any individual file. A successful parse requires that those same macros and include paths be provided to the parser when it runs.
Often the remote machine has greater CPU power than the local machine, so it can often complete parsing and indexing tasks more quickly than the local machine.
Remote machines are often accessed at geographically separated locations. The intermediate topology of the network can often be complicated, with many hops, and slow links. As such, in order to maintain performance it's important for as little data as possible be transferred back and forth between the local machine and the remote machine.

As such, we feel that if the Remote Development Tools are to be successful, then they must provide remote services that allow the user to do all of the parsing and indexing on the remote machine. The local machine can query the remote machine for data it is interested in, and only this data gets transferred over the remote connection.

So, that's the motivation. We just contributed a framework and reference implementation that implements the following features for C/C++:

A New Remote C/C++ Project wizard that allows you to create remote projects and configure your service model (files are served by EFS)
Integrated index lifecycle management
Automatic source code delta handling (the index is automatically updated when files in your project are added/removed/changed)
Remote Search
Remote Call Hierarchy
Remote Navigation (e.g. Go To Declaration)

Other planned services include:

Content Assist
Type Hierarchy
Include browser
Model builder

But that's not all. Currently I'm working on a remote make builder, which will essentially let you farm out the build commands for your project over the protocol of your choice (e.g. SSH), scan the build output for errors (like we already do in CDT), and map those back to the EFS resources in your project so that you can do standard things like click on errors in the Problems View from your remote compile and be whisked away to the corresponding source location.

The builder in fact is probably the most important feature to have. Really the "I" in IDE isn't there if you can't build anything. Source introspection tools are nice, but if your tool can't build my source, chances are I'm not going to use it.

At any rate... it's looking like it's going to be a busy summer...

e4: Now With More Openness!

2008-05-26T14:18:00.000-04:00

Well I haven't been blogging too much since EclipseCon. I came home from EclipseCon with a nasty bout of pneumonia which kept me on the bench for a bit, and ever since I've been back in the game I've been working flat out on the Remote Development Tools (RDT) initiative. (So flat out in fact that the wiki page that link goes to is woefully out of date. ) A lot of late hours and many Red Bulls later, we've got a first proof of concept demo up and running on the new RDT framework in time for our upcoming deadline, so I can finally take a bit of time again for things like attending meetings and blogging about them.

I spent Thursday and Friday last week at the e4 Summit. I won't bore everyone with all the technical details, as those can be found on the wiki page, but I wanted to make a point to express some serious kudos to everyone that made this meeting happen, especially given my past criticisms of openness in the platform.

The platform is entering a new era of openness. Seeing thirty people from thirteen different companies/organizations all working together under the scrutiny of the public eye to help design and build the next generation of the Eclipse Platform is a wonderful sight to behold. All the discussion and decisions are being made in the open where anyone can participate, whether it's at the Summit, on the mailing lists, or on the upcoming project conference calls. All the people who attended the summit are going to receive commit rights (if they want them) under the aegis of the new incubator project (which is not the old incubator project).

So, massive props to the Platform Team and everyone else involved. You are doing a great thing right now, and you deserve to be recognized for it. Good on ya!

We've got a lot of momentum going right now on e4. Great opportunities are opening up for everyone to participate in something special here, whether it's on the new browser-based-Eclipse stuff, or working or a new resource model, or something else. If you think there is something that should be a part of e4 that isn't, now is your time to speak up and contribute.

I want to emphasize the word contribute as well. People have harped on the Platform for years about it being hard to contribute to. Now that things are changing in that respect, the onus is on the rest of the community to put their code where their mouth is. Being open is not the silver bullet to build a thriving, diverse committer community. Ultimately it takes people writing code, and you could be one of those people. If you want to see something happen, pick up a keyboard and lend a hand, any way you can. The Platform folks are doing their part in opening things up, now it's the community's turn to take advantage of the opportunity. Don't waste it.

Sneak Peek: Eclipse In The Browser

2008-04-01T14:59:00.000-04:00

The whole e4 thing about being able to run Eclipse in a browser with AJAX is really exciting. What most people don't know is that there is another super secret aspect of e4 that's being worked on right now that will allow people to bring Eclipse to just about everywhere, which I think is even more exciting than AJAX. That work is the new HTML port of SWT.

Some screenshots are in order:

Here's a shot of an SWT example, running natively:

Here's a shot of the same example running under the HTML port (click for a larger view with more detail):

The awesome thing about the HTML port is it can be run in ANY browser, even on mobile or other embedded devices. The secret, if you zoom in really really far, is the following:

That's right. The graphics are actually being rendered as HTML text, and use the colour settings of the font to help render the graphics. The beauty of this is that this theoretically works with any display capable of colour text. This makes it possible to run RCP apps on a whole host of new places that previously didn't have the graphical horsepower to run SWT or eSWT. Perhaps we'll even see a text mode port of SWT eventually that won't even require a browser!

Truly this is an exciting time for Eclipse!

EclipseCon Poker Shameless Tactic

2008-03-20T12:00:00.000-04:00

Alright, this is pretty shameless but...

I am collecting the BEA cards in Eclipse Poker. Will trade a Jack or King of hearts. Comment below as to what you've got that you're willing to trade (BEA only please).

e4 Is News To Me Too

2008-03-07T14:00:00.000-05:00

Just read about the entire e4 debacle (thanks Doug Schaefer). I don't have a whole lot new to say other than I completely agree with Doug Gaff and Doug S on this one (see the thread on the planning council mailing list for details). The fact that that this has been done totally below the radar really irks me to put it mildly.

An interesting point for me to make though, is that those of us in IBM aren't really any more in the loop than the rest of you. Other than some vague "we're thinking of doing an Eclipse 4.0 and thinking about what that might look like" comments (which really weren't at all surprising and didn't really tell me anything that I would consider super-secret advanced knowledge), I haven't heard diddly about any of this e4 stuff. I'm just as blindsided as the rest of you, and I actually interact with folks on the platform occasionally. I have no more clue about e4 than what was in the component announcement.

For the record, the people on the platform are good people (some of them are even my friends). They certainly don't mean ill to anyone, and that certainly includes the Eclipse community. They do not sit in their Evil Eclipse Platform Lair, wringing their hands and cackling maniacally while they try to figure out the next way to thwart openness and diversity. They want to be open and diverse, they really do. Some of them really try very hard to be, and some of them are doing a good job of it (kudos to the platform debug team particularly!).

But, I have to be critical here. A lot of them just don't seem to get it when it comes to this whole open/transparent/diverse thing. I think it's good that a call for participation is being made now, but with all due respect, this shouldn't be the first time we're hearing about all this. What is this e4 thing? Where were the discussions that led up to this? Who participated? What requirements went into it? How were requirements solicited? Did you do a requirements doc? A design doc? Were those documents reviewed with community? Any meetings? Conference calls? Could people join those calls? Where are the minutes? These are all things I've ranted about on this blog before, but I haven't seen a whole lot of progress.

The perception is that once again the IBM skunkworks went off and did its thing, their reasons known only to themselves, and are now just presenting us e4 as a fait accompli. I don't believe that to actually be true, and I very much believe that they're open to participation going forward. And to be honest, I'm going to make a bit of a leap of faith here and assume that whatever e4 is, it's not going to annihilate our Eclipse universe as we know it and doom us all. They platform people are very smart, and they value the community greatly.

However, people need to take some ownership here and acknowledge that there is a PROBLEM. The entire process that led us up to this point was closed. THAT is the problem that we need to solve. Until the Platform starts opening up their process, the Platform will never truly be open. And that's a problem for all of us, whether you work for Big Blue or not. The lack of openness hurts people in IBM like me just as much as the rest of you.

At any rate, I hope the powers that be use this as a learning experience.

Security And The Eclipse Board Elections

2008-02-27T13:30:00.000-05:00

So, I just voted in the Eclipse Board elections. I encourage everyone entitled to a vote to do the same.

I'm not feeling the greatest about the security of the whole thing though. My password to login to the voting system was sent via email, which is not a very secure protocol given that the contents are sent out in the clear. Why couldn't I just login using my SSH credentials, which are relatively secure? Granted, the likelihood that someone is sniffing committers' email in an attempt to fraudulently login and rig the board elections is pretty low, but it's the principle of the thing.

Even worse, was that after I voted, the system told me to expect a confirmation email. Ok, great. Except, the confirmation email contains a listing of how I voted for all the candidates. So much for secret ballots. Oy.

Toronto DemoCamp tonight!

2007-11-22T15:00:00.000-05:00

Just a reminder that the Toronto DemoCamp is tonight. Come out to see some cool Eclipse demos, talk to other Eclipse enthusiasts (and committers!), and have some free refreshments.

And yes, the weather is a bit crappy today, but we're still on! The snow is supposed to stop this afternoon.

Hope to see you at DemoCamp!

One Week Till Toronto Demo Camp

2007-11-15T14:55:00.000-05:00

We have some cool demos lined up:

Marcelo Paternostro, EMF: Building an Ecore model and seeing what EMF can do with it
Mike Kucera, CDT Using Parser Generators To Build CDT Parsers
Shaun Smith, Eclipse Persistence Services (EclipseLink) & Dali JPA Tools--Object/XML and Object/Relational mapping in Java
Eugene Kuleshov, Task management and team collaboration with Eclipse Mylyn (Mylyn)

There is still limited room for attendees and presenters, so sign up now and see some cool Eclipse stuff, have a few frosty beverages on the Eclipse Foundation, and hang out with some committers!

Paintball, The "Other" Planet Eclipse, and A Humble Request

2007-10-18T20:00:00.000-04:00

The other day a bunch of us here in the lab went out paintballing as a sort of team-building event. Little did the poor saps know that I used to play every single weekend, have all my own gear, etc. Needless to say the other team was frequently punished by the steady rain of purple paint from my tricked out Classic Automag .68 hammering down upon them. Muwahahah.

Oddly enough, going out with my co-workers has bitten me with the paintball bug again, so when waiting on a build I was checking around online to see how much some of the new markers I like cost. While looking around, I saw for sale a "Planet Eclipse 07 Ego" paintball marker. "There's an Eclipse paintball marker? COOL!!!" Intrigued, I Googled further, and it seems that http://www.planeteclipse.com is the website for the paintball company called (you guessed it) Eclipse that makes this particular marker.

They even have a marker that comes in a cool purplish-blue colour that sort of fits with the theme of our favourite IDE...

So instead of giving away Eclipse mountain bikes this year, how about some cool Eclipse paintball swag? This is way cooler than a mountain bike...

Here's hoping anyway :-)

BeerClipse Meetings?

2007-10-05T17:00:00.000-04:00

I just read Ian's blog about demo camps.

An interesting thought.

In addition to this sort of one-time thing, perhaps we could do something like the 2600 meetings. Set a standard time, like the second Friday of the month or something like that, to be the standard day for meetings. Then, people could locally organize their own meetings at that time at their local foodcourt/pub/whatever.

The 2600 meetings range from organized events with actual agendas and presentations, to informal "let's show up and shoot the breeze about hacking" meetings. The nature of the meeting is up to the organizers.

I think you could do something similar for Eclipse and have it be pretty successful. If nothing else. the promise that committers will be there will definitely draw in the user and contributer community because they'll want to corner us and ask questions. And if they don't show, there would always be stuff that we committer folk could talk about amongst ourselves over some frosty beverages.

What do people think about this idea? Would you participate on a regular basis?

Java Makes It Hard to Find Good Help These Days...

2007-09-25T19:10:00.000-04:00

From the Google-alerts-are-cool department:

Leif Frenzel has posted an interesting blog entry on the difficulties of creating tools for non-Java languages in Eclipse. He argues that in order to have tools that rival JDT, you have to have complicated tools that have intimate knowledge of the syntactic structure and semantics of the user's code, and this is hampered by the inability to code plug-ins easily in anything but Java. It's hard to find for example a parser generator that can handle Haskell, unless you're willing to use a parser generator written in Haskell itself. Go read it. Interesting food for thought.

Leif argues that this makes it hard for the community to contribute, because inherently the people that are most interested in seeing support for the language in question are people that are heavy users of that language, which also means that it's likely that their primary language of choice is not Java.

This makes sense to me because I've seen it first hand when trying to hire new members of my team that works on CDT. It's difficult to find people that have both good Java skills (needed for writing plug-ins), and a good knowledge of C and C++ (needed to know what the heck our plug-ins should be doing).

Would we find it easier to get help on CDT (both in terms of new hires, and in terms of help from the community) if people could write plug-ins in C++? Hmm... maybe...

Food for thought anyway. Personally I prefer writing my plug-ins in Java, thanks.

Blog to Blog: Diversity

2007-09-10T13:55:00.000-04:00

I was going to post a comment to Bjorn's blog entry about diversity, but I thought my response was food for thought enough (and controversial enough!) to merit its own blog entry. These thoughts have been simmering for a while.

I think diversity is good. A lot of projects need more diversity. However, I think that until the Platform has a non-IBM committer that is doing more than just contributing builds of SWT for a non-primary platform, then diversity at Eclipse is somewhat of a farce. The Platform, if anyone, should be a shining paragon of diversity and openness to the rest of the projects on Eclipse.org, but it's the number one project that people complain about with respect to diversity and openness.

It is very hard to get contributions in to the platform (often with good reason, we don't want crap in there obviously...), and as a result, it is hard to build up enough of a reputation to become a platform committer. While the platform team are good at posting plans and information, the processes that go into them are essentially closed. The day to day discussions about design and development issues do not seem to happen in an open forum. There are no public conference calls, and the mailing lists don't have significant traffic about the actual development going on in the platform. Instead they usually amount to status reports on builds and testing, peppered with the odd user question (which usually ought to have been posted to the newsgroups instead). And no folks, Bugzilla on its own does not count... not everyone has time to keep hitting refresh on Bugzilla to see what bugs are appearing and how they are going to be fixed. People at least need a heads up that "Hey, we're discussing $BIG_ISSUE over on Bugzilla #whatever, go there if you're interested in the discussion." Also, I would say that Bugzilla is not the greatest forum for discussing broader issues and project plans, etc.

In this sort of environment, it's hard to build a sense of camaraderie with the team. There is a definite "US and THEM" atmosphere. Not that the Platform people are adversarial, don't get me wrong. But, you definitely don't get the feeling they are actively trying to get newcomers into the fold. It's difficult to get enough information about what's going on in order to try to get up to speed to the point where you're on somewhat of an equal footing.

Now to be clear. I'm not trying to slag the Platform or the people that work on it. They have done a ton of awesome work over the years and we all owe a lot to them. Theirs is no easy job at the best of times. But, I do think there are things that they and other projects could be doing better, and so I'm trying to criticize constructively. Hopefully everyone takes it as such.

Now, let's contrast the status quo with the Platform project with my experience with becoming a committer on CDT. The company I was working for at the time was looking to move their IDE to the Eclipse/CDT platform. As I started coding up our integration, I started hanging on out the cdt-dev mailing list, reading and eventually participating in the in-depth requirements gathering and technical discussions that went on there. CDT conference calls were (and still are) open to the public, so I just started showing up. At first I didn't have a lot of useful things to say other than introducing myself and communicating that our company was starting to use CDT and was looking to contribute, but over time as I got ramped up, I had more meaningful things to say.

After fixing a few bugs on the 2.0 release, CDT 3.0 came along, and there were some features that I needed implemented that the commiters (of which I wasn't one yet) indicated that they didn't have time to work on. So what did I do? Well I hopped on the planning conference call and told them that I was committing to deliver patches for those features for the release. "Great!" they said, and put my items on the plan with my name next to them. I did what I promised and after the release went out I was rewarded with a nomination and subsequent election to commit rights.

The difference was the CDT gang went out of my way to make me feel like I was part of the team, even if strictly speaking I didn't have commit rights yet. Discussions and information were open enough that I could participate as nearly a first class citizen. Sure, I still had to submit patches for anything I wanted to change, and convince someone that those patches were worth committing, but that's much easier to do when you've already been collaborating as near-equals for a while. The open and collaborative nature of the project allowed me to build up an important thing: trust.

Anyway, back to the point.

I think really that the focus of the diversity rules and enforcement should not be to stop projects from starting that are not yet diverse, but instead should focus on opening up projects that already exist but are not diverse. I don't think diversity rules should get in the way of contributions. It's better to have a non-diverse project than no project at all, because it's better to have imperfect code that does something for someone, rather than have no code, which does nothing for everyone.

That being said, if a project exists and people can't contribute even when they want to, then that is a definite problem which we should be trying to fix.

Admittedly the devil is in the details, especially with a project as large and as widely consumed as the Platform, but without trying to sound heavy-handed, I think we should be looking at ways to open up it and other projects. Such changes would have to be practical. Maybe, for instance, you can't have a 100% open Platform Conference Call where anyone and everyone that dials in can say whatever they want, because you could get far too much signal to noise. But, maybe you can have a moderated call, where some people who are known and trusted (e.g. Eclipse committers) can speak freely, and others can flag questions for the moderators attention. This way information still flows, but hopefully supporting that flow is not onerous for the committers.

At any rate, I'm curious to see what's going to happen with this new diversity push. The Platform Debug team has been setting a good example as of late, and is actively trying to mentor in some new committers on the Debug sub-project. Kudos to them, and hopefully we will start seeing more things like this.

Staffing the #eclipse IRC channel

2007-08-15T09:36:00.000-04:00

For a long time our wonderful one of our beloved committer reps has been harassing us committers (and rightly so) in the hopes that more of us will hang out on the #eclipse channel on irc.freenode.net and help out with user questions about our various components.

This is arguably a noble goal, but in the past I had always felt that I just plain didn't have the time to devote to this. There are a lots of new user type questions that pop up on the newsgroups and mailing lists (to the point where a couple of years ago Webmaster had to tuck the mailing lists away in a corner on the website because the lists were becoming a free-for-all). This is especially true of the platform newsgroups and mailing lists. The fact that there are so many of these types of posts is a real testament to the popularity of Eclipse though, so it's not something to complain about either. But, in essence, I was afraid that there would be a bad signal to noise ratio for me, i.e. lots of basic questions about the platform and not a lot about CDT. Given that I have other things to do with the bulk of my time, I didn't figure I could keep up with the thread of conversation in a timely fashion, because it's not like I'd have the window up in front of me while I was working (sorry zx, I haven't tried ECF yet).

Lo and behold, I found something neat a few weeks ago which changed my mind. The Chatzilla plug-in for Firefox has this cool feature called "stalk words."

My previous experience with IRC was circa the late nineties, back when I was hacking away on and doing the writing for a video game total conversion that ended up, sadly, dying (but more relevantly, I was hanging out in IRC all day and night when I should have been doing my CS assignments). Back then IRC was not really state of the art, and if you wanted to do anything cool like this you had to write mIRC scripts yourself.

Chatzilla however, makes it pretty dirt simple. Basically, you can tell Chatzilla that there are certain phrases that you are particularly interested in, and when someone types one of them, Chatzilla can be set to alert you by playing a sound and/or flashing in the task bar. Particularly useful for us committers if you want to known when people are asking questions about the Eclipse project you work on.

This has been working out pretty well for me, and I've been fielding questions about CDT for a few weeks now. If you're a committer and your project is underrepresented in IRC right now, you might want to try it out too.

CDT Webinar

2007-07-12T09:19:00.000-04:00

Just a reminder to everyone that Doug and I are presenting a webinar on CDT today. Here's the blurb that Lynn sent out recently:

==========================
CDT 4.0 – Reaching for Uberness
July 12, 2007 at 8:00 am PDT / 11:00 am EDT / 3:00 pm GMT/UTC
Presented by Doug Schaefer and Chris Recoskie
To register email webinar-cdt at eclipse dot org

C/C++ Development Tooling (CDT) 4.0 is our most exciting release yet bringing C/C++ developers a load of new features, quality, and performance improvements. This webinar presented by Doug Schaefer, the CDT Project Lead, will walk through all of CDT’s features from new project creation, code editing, and source navigation, to build and debug with a special focus on what’s new in CDT 4.0. New users will gain an understanding of how to use the CDT with their projects and all users will see how the improvements in CDT 4.0 make it a world class C/C++ development environment.

For more information on this and other Eclipse webinars visit http://live.eclipse.org/. Special thanks to Adobe for contributing their Adobe Acrobat Connect product to host the webinar.
==========================

Hope to see you all there!

Edit: In case you missed it live, here is the link to the recorded webinar: http://live.eclipse.org/node/293

Eclipse & CDT In The Scientific Community

2007-06-12T14:08:00.000-04:00

After putting myself at the mercy of an extensive background check, I spent a week recently at Oak Ridge National Laboratory in Tennessee for a workshop centred around the Eclipse Parallel Tools Project and its usage for development of high performance, scientific applications. We had about thirty or so attendees from industry, academia, and the various U.S. national laboratories.

The workshop was largely strategic in nature (although Beth Tibbitts, Greg Watson, and Craig Rasmussen did give a packed tutorial on Eclipse, PTP, CDT, and Photran at the lab the day before the workshop). Basically we split up into groups and tried to come up with the challenges and deficiencies that we have right now with respect to Eclipse, PTP, performance analysis, debugging, and language support and services (such as the CDT).

That generated a giant list of action items, which everyone then got to vote on and hopefully sway the future development plans of all us tools developers in the room.

Greg has posted the meeting report, but in summary, two major themes won out.

1. We need remote tools.

These developers are creating massively parallel programs on big iron. They don't have these machines sitting on their desks, they are typically in another building, or for some, in another city. They often can't run Eclipse directly on these machines due to lack of Eclipse support (and often the OS lacks any graphical UI whatsoever). Even if they can run Eclipse on it, for some the slowdowns due to all the network traffic required inhibit their productivity significantly. Generally since these huge machines are a scarce commodity their utilization rate is high and this can mean a lot of people using the machine at the same time.

2. We need performance tools.

These applications are massively parallel for a reason. It's because they are doing huge simulations and/or crunching huge amounts of data. Performance is key because even in a best case scenario it often takes days to do a run. You don't want those days to turn into weeks, and so a large amount of development time goes into tweaking performance. Unfortunately, there isn't really much in the way of performance tools integrated into CDT or Photran right now. These people either need support for existing performance tools, or some new ones provided somewhere by Eclipse.org.

I created a survey asking the participants various questions about their targeted platforms, what development tools they use, how they use those tools, and more, which I'll be discussing in today's PTP monthly conference call. The results were quite interesting. I'll post about them in a follow-up soon.

I also have lots of great pictures from the tour of the lab that they gave us that I'll be sharing soon. I just have to remember to upload them when I'm at home sometime.

The Europa Train - Blessing or Curse?

2007-05-15T10:43:00.000-04:00

A bit of a controversial topic today...

I've always had mixed feelings about the simultaneous releases of Eclipse. It's definitely great to be able to go to one place and download a family pack of nuggets of Eclipse goodness (mmm.... nuggets...), and know that they all work together. Each year the release makes a big splash in the media too, which is great for Eclipse and everyone involved. That kind of publicity will likely make Eclipse.org the victim of its own self generated Denial Of Service attack as the download servers are brought to their knees by the throngs of downloads. That is definitely the kind of popularity and success that we all want.

One thing I've never liked about it though is all the peer pressure to make sure the train is on time. Don't get me wrong, I love having the train on time just as much as everyone else. It's nice to be able to know when the train is going to arrive so that you can plan your ride. Your boss is not going to be happy if you tell him you have absolutely no clue when you will be in to work because you don't know when the train is coming. But, if you knew that parts of the train were broken, wheels were missing, etc., wouldn't you want the mechanics to take the time to fix it, rather than it breaking down mid trip, forcing you to whip out your multi-tool and a pack of gum and MacGuyvering a solution? (BTW, I hereby declare MacGuyvering to be a word.)

My point is, the train sometime feels less like a train and more like a juggernaut. Last year for Callisto, we shipped a CDT 3.1.0 that had a lot of problems with it. CDT 3.1 was a big release that had a lot of new functionality, and it's inevitable that when you deliver a lot of new content there are going to be lots of problems, through absolutely no fault of the committers and contributors. Even good code inevitably has a certain amount of bug, and so it follows to at least a certain extent that the more new code you put in, the more bugs there will be. As some for-instances, there were a lot of problems with the indexer, and search was pretty much broken at the time. Yet, we shipped anyway, because the train had to be on time. It's often said to ISVs that consume CDT that they shouldn't take the dot-zero because it will be buggy, instead take the dot-one. I hate to say it, but really IMHO CDT 3.1.0 should have been baked for a while longer.

I don't think this is the greatest situation. Individual projects ought to be able to hold up the train if they need to. However, along the lines of a recent post by Doug Schaefer, I think Eclipse gets itself sometimes into a situation where it's a victim of its own hype. The date for the release is picked a year in advance. We spend so much time in that year hyping the release that by the time we start getting into the bug fixing cycle, there is already so much pressure to release on time that we couldn't hold things up if we wanted to. We shouldn't be releasing things that we know for a fact that people shouldn't be using. Speaking in practical terms, a dot-zero release is never going to be flawless, but if you're shipping with major functionality broken, or with crippling bugs that preclude widespread adoption, then I think the purpose for the release has been somehow lost sight of. A release that can't really be used is somewhat pointless.

Now, I'm sure someone is going to reply to this and say something along the lines of "the release is at the whims of the committers", and that we really have the power to hold things up. Technically it may be true by the letter of the process, but if you really believe that I suggest you try it and see what happens. Short of the Platform or JDT being horribly busted, I'm pretty sure you will get voted down.

I think that what needs to happen is that the release needs to be bug count driven. This process is not flawless either, but the idea is that you do what we do on the CDT milestones, and we don't do the build until all the bugs targeted for that milestone are fixed. When we reach Zarro Boogs, then it's Go Time. Sure, nefarious people can play games still by spuriously marking bugs RESOLVED - INVALID, or by playing games with severeties and target milestones or what have you, but I think that on the whole the idea works. This way, the date of the release is driven from the bottom up by the committers, and is not imposed on them from above for them to deal with after the fact. Sure, you still need to give a rough estimate to people as to when they can expect something (e.g. "Summer 2008"), and the committers shouldn't be given license to delay as long as they please without reason, but at least then there is some flexibility built into the plan.

Don't misread what I am saying here either. CDT 4.0, which is coming out on the Europa train, is shaping up to be both the most feature rich and yet most robust version of CDT yet. I'm not currently anticipating a recurrance of what happened on 3.1.0, and I would definitely recommend to users of previous versions of CDT that they move up to 4.0 and take advantage of the scores of bug fixes and new features it includes. And, I also think having a release train is on the whole a good thing. But, I think there are some things we can do so make sure everyone gets a say in how the train operates.

I'm curious to see how next year's release will unfold...

Language Extensibility In CDT 4.0

2007-05-07T09:33:00.000-04:00

This screenshot is really cool. But why?

We'll get back to the screenshot I promise.

Well, CDT 4.0 RC0 just went out the door last week, marking our first feature complete build for Europa. Confusingly enough our next build this week is going to be marked as M7, so we have the odd situation where we have a milestone build after our first RC, but the team felt it was important to keep the naming convention constistent with the Europa build of which we will be a part of, and we didn't want users getting confused about which build of CDT to use with Europa M7.

There are a few cool new features that my team here at IBM have been working on that a number of ISVs and other language tools authors are going to hopefully find useful. It's always been fairly easy to add support to CDT for compiling different languages via CDT's Managed Build and Standard Make projects, but we've been working recently to make it easier to integrate new C-like languages into the Core so all those cool features like search, open declaration, and content assist all work.

For a while now, it's been possible to contribute definitions for new languages into the CDT core. Circa CDT 3.1, we added an extension point to CDT to allow you to contribute new languages via the ILanguage interface, and to map those ILanguages to an Eclipse content type. Each ILanguage has methods it must provide that let you parse a file and get an Abstract Syntax Tree (AST) out of it as result. Once you have an AST all those cool features I mentioned eariler start working, provided you use CDT's DOM AST APIs.

This worked great for clients such as the Photran project (who do the Fortran language IDE integration for Eclipse), but it was a bit problematic if you actually wanted to override what the language was for C and C++ files. CDT would look for extensions to the extension point, but would stop looking once it found the first one for any given content type (I'm simplifying things but this is how it would appear from the user's point of view). Hence, there was no deterministic way to make sure that your language was the one that got used.

In CDT 4.0 we've now added the concept of language mappings to the workbench preferences and the project properties. What this means is that the user can go in and change which language is mapped to which content type, even down to the level of the individual file.

The language mapping feature is great for those that compile the same project on multiple platforms with different compilers that all support slightly different variations of C or C++. Now if you have a build configuration for each platform you can set the language mappings on each configuration individually, and your code will be parsed properly in each configuration (provided that you have an ILanguage to handle those scenarios). It's also great for embedded vendors, as most of them tend to have slightly different variations on the C programming language to enable you to do some cool things like handle interrupts, etc. This way they can define their own ILanguage which can handle these differences.

Another big thing we've been working on is making it easier to create the parsers for those language variants. The most frequently encountered use case for this stuff are the use cases belonging to people like those embedded vendors I mentioned. For the most part the language they are implementing is nearly identical to C or C++, and they just need to add a couple of keywords or a few new constructs. Up until now they've pretty much had to write a whole new parser for that from the ground up. The GNU C and C++ parsers that are bundled with CDT are lean, mean, hardcoded state machines, and they are pretty difficult to get your head around if you are brave enough to crack open the code; difficult enough that most people that want to integrate a new language variant into CDT pretty much gave up right there. Don't get me wrong, those parsers are great at what they do (and without them we'd have been parserless for years), but they were designed with peformance in mind, and not readability or maintainability. If you tried to extend from the concrete classes in order to modify the behaviour of the parser you'd end up overridding big gnarly functions that do most of the work, and so if we ever fixed a bug in the original parser it probably wouldn't trickle down to your code unless you looked for it and cut & paste it into the parser you created.

Enter the new parsers that my team has been working on. One of our core requirements from our internal customers here at IBM was support for new language variants. Since we knew we were going to have several variants to support over the next few years, it seemed like a worthwhile investment to create some kind of extensible parser framework. To keep things "simple" we started with C. The goal was to create a basic C parser based on the ISO C99 standard, and to make it reuseable to support other language variants. In theory then language implementers would get C parsing for "free" and could concentrate on just defining the delta of their language compared to the base language.

It seemed natural for us to to use a parser generator for this. Parser generators take as input a grammar which specifies the rules of a language, and from that grammar it generates a parser that can handle that language. Just having a grammar will let you recognize whether a set of input abides by the rules of the language, but generally you want to do more than that. Typically as well you would define semantic actions in your grammar that do interesting things, which in our case was build up an AST with CDT's DOM AST APIs, so that once the language was parsed all those cool tools I mentioned earlier could recognize the structure of the code and do Cool Things(TM) with it.

So, what we did was create a C99 parser using the LPG parser generator, which has semantic actions in it to build up an AST for CDT. LPG is a parser generator built by some folks at IBM Research, which is being used for the parser in Eclipse's JDT, as well as for the SAFARI IDE Generator. The cool thing about LPG is that it has a notion of language inheritance. What this means is that if you take our C99 grammar file, you can do the equivalent of a #include in your own grammar for your own language to pull in our grammar. You can then add new rules or overrride our rules as you see fit, i.e. you get C parsing "for free".

The results of this were pretty amazing. One of our requirements which we got from the Eclipse Parallel Tools Platform people was to support a new programming language coming out, Unified Parallel C, which is a variant of C for massively parallel applications. The language adds new keywords and constructs which allow you to control the parallelization and synchronization of your program. By including the C99 grammar in our UPC grammar, we were able to get UPC working in a matter of days. Time to go back to our screenshot of the CDT editor, with a UPC file open:

There's a whole lot of cool stuff going on there:

Syntax highlighting of new keywords (upc_forall)
Outline view works
Content assist is finding constructs in the code
Content assist is working on constructs that are not normally legal C!!! It's a subtle point, but take a look at where the caret is in the upc_forall statement. This construct takes four expressions, not the usual three that your plain old ordinary for loop takes. Yet, content assist in that fourth expression just plain works!

Doing all this with the old parser would have taken a long time and been very error prone. I would definitely say that thusfar this effort has been a resounding success.

After CDT 4.0 is out the door we're going to start looking at doing some more interesting things with parsers.

Firstly, we want to write a GNU C language variant on top of our C99 parser and see how that stacks up against the existing GNU parser in CDT in terms of correctness and performance. We're already re-using all of the parser JUnits on our parser, so I already have a warm fuzzy about correctness. If the speed is good enough then I would love to replace the old parser with one based on ours because then it will be a lot easier to maintain.

Secondly, we're going to start tackling C++. Parsing C++ properly is a very difficult problem, given all the ambiguities in the language itself. I know personally of teams of people using bottom-up parsing techniques to parse C++ so I know it can be done (LPG is bottom-up too), but we have to figure out how feasible this is to do with LPG. Luckily we have a good line of communication with the LPG authors, and they are keen to see LPG being used successfully on C++, so if we encounter any roadblocks hopefully we can work together to smash them down.

The future for language support in CDT is looking very bright :-)

Ride For Sight 2007

2007-04-23T15:02:00.001-04:00

I am proud to support Ride for Sight and The Foundation Fighting Blindness' plight to discover treatments and cures for blindness. Help join the fight against blindness today! My personal goal is to raise $500.

You can help support me by making a secure online donation using your credit card. Click on the link below:

https://secure.e2rm.com/registrant/StartUp.aspx?SID=1444606

For more information on how YOU can participate in Ride for Sight, please visit us at http://www.rideforsight.com .

Thanks for your support!

Chris