(So, in reality, D probably isn’t all that little, but here’s to it making it up the mountain.)

A good while ago I wrote a post about my hope for the D programming language as an up-and-comer in systems and embedded programming. D promises to bring a number of modern language concepts and constructs to low-level systems while yet producing a fairly efficient runtime. It’s 3 years on since my original post, but I’m still hopeful.

Just a few days ago a comment came in on my original post. My friend Mark VanderVoord responded in that comment thread. I thought I’d make the content of the thread a bit more visible (below) as it’s concerned with using D in the real world.

There’s been respectable progress toward D becoming viable in the mainstream. I’ll venture a guess that it’s going to be several more years until it truly is.

Update (7/1/2010): We don’t often attract a great many comments on our posts. But we have here. I point it out because there’s good resources, background, and perspective on D in the comments. Take a read. The original comment thread that sparked this post can still be found beneath this update.

1. Steve Klabnik pointed out a new kernel project written in D (Steve is one of the authors)
2. Kevin Bowling recommends the book The D Programming Language written by another of our commenters Andrei Alexandrescu
3. Descent is an Eclipse plugin for D
4. Several have compared/contrasted D to the new Go programming language by Google

Update (7/6/2010): Commenter devdanke suggested Apple as a D corporate backer. Google seems to be pursuing Go; why not Apple getting behind D? Apple certainly has a lot invested in LLVM of late. Apple may just be up to something—though I’m doubtful it’s D. Time will tell…

slide Says:
June 25th, 2010 at 07:35 PM

Is there any active development that you know of to get D running on embedded, bare-metal type platforms?

Mark Says:
June 25th, 2010 at 11:01 PM

slide:

This is the current state of things as far as I know:

The official D versions (D1 and D2) by Digital Mars (DMD) are only x86 so far

GDC – this is a D front-end with a gcc backend. Theoretically this means you can target anything gcc can (which is almost everything). The work I’ve done with it (mostly ARM) has been pretty tricky to get working, though.

LDC – this is a D front-end to LLVM. Again, this means you should be able to target anything LLVM can. LLVM has a lot of momentum lately… and this port seems to be pretty active too. I have done some small experiments with getting it to target an ARM9. It seemed to work well. I hope to have more time to work on that in the future.

For either LDC or GDC, you’re getting D version 1, which is missing some of the new (in-progress) features. But, they’re more stable, so that’s nice.

D links easily to C, so bringing in libraries, RTOS’s etc is pretty easy.

I hope this helps.

Mark

For the last several weeks I have been creating a tool that accepts a set of configuration files and builds a number of charts based on information in those configuration files. For a few reasons I chose to use Ruby and the WIN32OLE support in Excel for staging the data and creating the charts. There may be better Windows solutions to creating charts, but Excel isn’t bad at it and the OLE automation in Ruby makes it fairly easy to drive Excel.

One of the things I find myself yearning for in a lot of programming languages is a powerful pattern matching system. I wrote one for ruby, but ruby's syntax just wasn't flexible enough to make something as elegant as I'd like. When I started using clojure, it seemed like a great little project for getting to know clojure's macro facilities as well as clojure itself. I set out to build a kickass pattern matching library for clojure that fit in with the language's way of doing things and was at least as expressive than any other pattern matching facility I've used.

The outcome of this effort, is matchure (github, clojars), a pattern matching library for clojure featuring

• equality checks,
• sequence destructuring,
• map destructuring,
• regexp matches,
• variable binding,
• "instance of" checking,
• arbitrary clojure expressions,
• and boolean operators (and, or, not).

All of which compile down to high performance clojure.

Introduction

Clojure, like most lisps, has a built-in facility which gets you part of the way there: destructuring in most variable-binding contexts. For example,

1
2
3

(let [a 1]
  a) 
; returns 1

binds 1 to the variable a. You can just as easily grab the first value out of a sequence:

1
2
3

(let [[fst & rst] (list 1 2 3)]
  [fst rst])
; returns [1 (2 3)]

This facility is even more powerful. You can also destructure maps as well. There's just one problem: let doesn't work well as a way of testing values.

1
2
3

(let [[fst & rst] (list)]
  [fst rst])
; returns [nil nil]

This is, in part, why I wrote matchure. Matchure provides a pattern matching facility that makes it really easy to perform complex tests against values and bind parts of them to variables.

Syntax and Examples

Like any lisp, clojure is homoiconic - clojure code is represented by clojure data structures. Unlike other lisps, however, clojure has a rich set of data structures with literal representations. '(1 2 3) is a linked list, [1 2 3] is a vector, with fast random access, and {:a 1, :b 2} is a hash map. Furthermore, these syntaxes have special places within clojure itself. Lists are clojure function or macro calls and vectors are used anywhere variables are bound or sequences are destructured. Aside from being a literal representation in code, the hash syntax is used for destructuring maps in let bindings.

Matchure takes this rich set of literal representations and uses it as a natural way to match patterns. At the moment, there are three main forms, if-match, when-match, and cond-match. They work similarly to the clojure built-ins, but match values.

At the most basic level, matchure can be used to test for equality.

1
2
3
4
5

(if-match [nil nil] true) ;=> true
(if-match [1 1] true) ;=> true
(if-match ["asdf" "asdf"] true) ;=> true
(let [s "asdf"]
  (if-match ["asdf" s] true)) ;=> true

Regular expression literals test for a match

(if-match [#"hello" "hello world"] true) ;=> true

and fully qualified class names test for instance? relationships.

1
2

(if-match [java.lang.String "foo"] true) ;=> true
(if-match [java.lang.Comparable "foo"] true) ;=> true

Matchure supports _ and ? as wildcards, so both of these match anything

1
2

(if-match [_ "foo"] true) ;=> true
(if-match [? "foo"] true) ;=> true

_ is idiomatic in clojure when you want to ignore a value. In matchure, ? has special meaning. ? can be thought of as "the thing this part of the pattern is matching against". As such, ? always matches successfully. It is also used in binding variables. ?foo matches successfully and stores the matched value in the variable foo.

(if-match [?foo "bar"] foo) ;=> "bar"

just like regular clojure, list literals represent clojure code. Here, too, the special meaning of ? comes into play. You can perform arbitrary tests this way using ? to represent the matched against value:

(if-match [(odd? ?) 1] true) ;=> true

Just like with let, you can destructure sequences

(if-match [[?fst & ?rst] [1 2 3]] [fst rst]) ;=> [1 (2 3)]

and maps

(if-match [{:foo (even? ?)} {:foo 2}] true) ;=> true

Finally, unlike any other pattern matching facility I've seen, matchure has support for boolean operators.

1
2

(if-match [[(and ?fst (even? ?)) & ?rst] [2 3 4]] [fst rst]) ;=> [2 (3 4)]
(if-match [[(and ?fst (even? ?)) & ?rst] [1 2 3]] [fst rst] :failed-match) ;=>:failed-match

or and not are also supported.

You can get the code from github, or use it in your clojure project by grabbing it from clojars.

Apple introduced closure support with the release of Snow Leopard. Plausible Labs has since created a framework and a drop-in runtime that provides closure support for both MacOS X 10.5 and the iPhone SDK. Closure support is a fundamental and important change to the Objective-C language. Functions are now first-class citizens in the runtime environment; they can be saved, copied, and passed around just like any other object.

That being said, much of the existing API in both MacOS X and the iPhone SDK rely on callbacks, selectors, and delegate patterns to cope with lack of first-class functions. But don’t let that stop you from using closures. It is relatively easy to create small intermediary wrappers that bridge the gap between callbacks and closures.

I recently needed to update some Java code that was frequently making unecessary calls to an external web service. Depending on the particular UI component, or the item being displayed in that component, a large set of the data being retrieved ended up not being displayed. My task was to update the code so that it only made external calls to the web service when the data was going to be displayed.

My first thought was to pass along a flag from the view layer that would indicate whether or not to retrieve the data in question. This would require updating all of the UI classes that displayed the data, as well as passing the new flag through several additional method calls to prevent the data from being collected when it wasn’t needed.

I did not care for this solution because it meant changing several classes so that details of the view could be passed down into the data retrieval logic. Instead, I came up with an alternative solution, the crux of which is a generic, thread-safe, delayed evaluation Map.

Instead of the business layer service returning a fully populated Map of data, it returns a Map that will contain that data, if it is accessed. If no user interface components request the data for display, the data will never be retrieved. This meant that I did not have to change any code outside of wrapping the logic for populating the Map with a new LazyImmutableMap. Here is an example of how it is used:

1
2
3
4
5
6
7
8
9
10
11

public Map<String, String> getData(final Integer id) {
  return new LazyImmutableMap<String, String>(
    
    new Callable<Map<String, String>() {

      public Map<String, String> call() throws Exception {      
        return externalService.getSomeData(id);
      }
    
  });
}

The call to externalService.getSomeData() will only be invoked if something tries to access an element in the LazyImmutableMap. Implementing the class itself turned out to be extremely simple with the help of Google Collections’ ForwardingMap (see this previous post for more on Google Collections) and the standard Java concurrency class FutureTask.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

public class LazyImmutableMap<K,V> extends ForwardingMap<K,V> {
  public static class AccessException extends RuntimeException {
    public AccessException(Throwable cause) {
      super(cause);
    }
  }

  private final FutureTask<Map<K, V>> task;

  public LazyImmutableMap(final Callable<Map<K,V>> eval) {    
    task = new FutureTask<Map<K,V>>(new Callable<Map<K,V>>() {
      public Map<K, V> call() throws Exception {
        return ImmutableMap.copyOf(eval.call());
      }
    });
  }

  @Override
  protected Map<K, V> delegate() {
    task.run();
    try {
      return task.get();
    } catch (InterruptedException e) {
      throw new AccessException(e);
    } catch (ExecutionException e) {
      throw new AccessException(e.getCause());
    }    
  }
}

The constructor creates a FutureTask that when accessed will return an immutable copy of the Map provided by the passed in Callable.

Extending ForwardingMap means that I only needed to implement the delegate method to get a fully compliant Map class. The FutureTask takes care of preventing the passed in Callable from being executed more than once, and it’s thread-safe, so any number of threads can access an instance of the map simultaneously.

Unfortunately Java’s checked exceptions add quite a bit of clutter to an otherwise very clean and simple class. The AccessException is needed because the normal Map interface does not allow for checked exceptions to be thrown. The get() method on a FutureTask can throw two different types of exceptions. They need to be handled differently, so they are both caught and properly wrapped in an AccessException (a subclass of RuntimeException) and re-thrown.

Using this technique I was able to retrieve the external data if and only if it was going to be displayed to the user. And I was able to do it without breaking the separation between the view and the business logic.

Version 1.0 of the Google Collections library was officially released December 30, 2009. I have been using the library for the past 6 months or so on a variety of Java projects, with great success. Today I gave a brown-bag talk to share with some interested Atoms.

I have included below the sample code I used as presentation material, in case others are interested.

Recently, a colleague and I were experimenting with SWFUpload, a popular flash based file upload tool. We quickly realized that SWFUpload cannot forward cookies; and since our rails application utilizes cookie based sessions we were unable to upload a file. After a little bit of research we stumbled across a blost post that offered a solution using rack middleware.

In short, it has you add the required cookies as form parameters and then insert a rack middleware object before the rails session store. This object is responsible for extracting the form parameters and then inserting them into the “HTTP_COOKIE” header before rails is invoked. After following the instructions on the blog post we were able to successfully upload a file using SWFUpload.

This particular problem inspired me to write a lightweight rack middleware library called Cookie Monster. It solves the very problem that was described above and is wrapped up in an easy to use gem (or rails plugin).

 gem install rack-cookie-monster

or

script/plugin install -f git://github.com/dewind/rack-cookie-monster.git

Configuring it for rails is easy:

1
2
3
4
5
6
7
8
9

# In rails initializer
  Rack::CookieMonster.configure do |c|
    c.eat :_session_id
    c.eat :user_credentials
    c.share_with /^(Adobe|Shockwave) Flash/
    c.share_with "Burt"
  end

  Rack::CookieMonster.configure_for_rails

I recently attended JRubyConf in San Francisco. The conference definitely had a strong web presence, but it wasn’t without a few “web neutral” presentations. The JRuby State of the Union, The JRuby Testing Story, and some of the lightning talks were interesting and fun.

Phil Hagelberg gave a brief presentation on Clojure’s Software Transactional Memory and how it can be used within JRuby. He wrote a small library that exposes and wraps the necessary STM components in the Clojure Java implementation. I briefly experimented with library, and it appears to work quite well and could provide immediate value to multi-threaded applications.

Last weekend I attended JRubyConf 2009. Overall I enjoyed the opening JRubyConf State of the Union, the follow-up JRuby on Rails, and the concluding JRuby Core Team Panel sessions the best. Despite browsing the JRuby mailing list as part of my daily routine, JRuby has a rapid development cycle; these sessions helped me get a clearer view of the JRuby team's current development goals.

I am curious when and how JRuby will take advantage of the invokedynamic and method handle technologies coming with the next Java platform release. After reading John Rose's paper Bytecodes meet Combinators: invokedynamic on the JVM, it sounds like language implementors will need to make a significant development investment if they want to fully utilize the new features. Charles clearly has a strong grasp on the new features. He's already tried out and given feedback on them, so I'm sure he'll be on the forefront of an implementation changeover. Still, I wonder what kind of timeframe we're looking at to go from the simulated method dispatching we have now to a fully fleshed out invokedynamic-backed implementation.

This style is great for testing because it clearly establishes intent for each test. This point is invaluable when another developer either inherits your code, or you revisit it at some point in the future. The main problem with this style is the need to create stubs for all the calls in the describe's before block, or resort to using stub_everything for all your mock objects. These are both fine solutions for simple code, but as soon as the logic encounters a condition you may be setting yourself up for potential testing bugs, and difficulty when you add new logic.

One way to avoid this problem is to leverage the power of rspec, and use nested describe blocks for each condition statement. This helps limit the before block for each describe to only define the stubs for that condition block. This is helpful because it prevents the tester from accidentally defining a stubbed expectation for something that should not happen. It also creates a nice placeholder for adding logic into the code in the future.

The following example shows how nesting rspec describes works. It is important to note that the branch that contains the most logic should be encapsulated in the sub-describe.

Example Source Code:

1
2
3
4
5
6
7
8
9
10
11
12
13

class Example 
  def run(director, runner)
    runner.prepare
    if director.go?
      runner.run
      if runner.complete?
        director.stop
      end
      runner.stop
    end
    runner.finish
  end
end

Example Test Code with nesting describes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

describe Example do
  before do
    @target = Example.new
  end
  
  describe '#run' do
    before do
      @director = mock
      @runner = mock
      
      @runner.stubs(:prepare)
      @director.stubs(:go?).returns false
      @runner.stubs(:finish)
    end
    
    it "should prepare the runner" do
      @runner.expects(:prepare)
      @target.run @director, @runner
    end
    
    it "should check the director" do
      @director.expects(:go?).returns false
      @target.run @director, @runner
    end
    
    describe 'director returns true for go' do
      before do
        @director.stubs(:go?).returns true # redefine
        @runner.stubs(:run) 
        @runner.stubs(:complete?).returns false
        @runner.stubs(:stop)
      end
      
      it "should run the runner" do
        @runner.expects(:run)
        @target.run @director, @runner
      end
      
      it "should check if the runner is complete" do
        @runner.expects(:complete?).returns false
        @target.run @director, @runner
      end
      
      describe 'runner returns true for complete' do
        before do
          @runner.stubs(:complete?).returns true # redefine
        end
        
        it "should stop the director" do
          @director.expects(:stop)
          @target.run @director, @runner
        end
      end
      
      it "should stop the runner" do
        @runner.expects(:stop)
        @target.run @director, @runner
      end
    end
    
    it "should finish the runner" do
      @runner.expects(:finish)
      @target.run @director, @runner
    end
    
  end
end

I realize the test code gets a bit long, but it is very easy to understand and extend. Also, nesting on conditional breaks creates an easy framework to follow when writing test code.

Just today someone told us he heard you can’t do real development in Ruby. Funny – the AGI Goldratt Institute paid us a whole bunch of money for nothing then. It must be that their brand new, multi-platform, JRuby-based desktop simulation app doesn’t exist. Pity.

• Desktop Application Development in JRuby
• Rolling a JRuby desktop application

Now that the AGI app we couldn’t name in those previous posts is done, we can talk much more about how we put it together. The most exciting point is that Ruby can be used successfully for large-scale desktop development.

UPDATE (February 4, 2009): Added demo movie goodness provided by AGI.

Atomic Object has been providing development and consulting services for a company in the healthcare industry. We’ll refer to this company as DoctorsInc hereinafter. Our primary focus was to bring agile project management and TDD practices to DoctorsInc’s development process. The group that we worked with at DoctorsInc uses a powerful ETL tool called Ensemble.

Ensemble is a product developed by a company called InterSystems that sits on top of another product called Cache. Cache is an integrated development platform and OODB that provides a powerful set of tools for development, database management, and much more.

I had the opportunity to attend Apple's World Wide Developer Conference this past June. I attended the conference to learn about and gain experience in iPhone development. Most of my time was spent in the iPhone Lab and iPhone development presentations and sessions. I thought most the presentations were informative and well presented and I came away with a lot.

My only complaint is that Apple would not talk about or even acknowledge the more complex areas of sample code in their presentations. It was obvious Apple was focusing on making iPhone development look very easy through and through. And compared to other mobile platforms it is easier. However I wish they had spent some time going into the more complex, non-apple abstracted code that actually takes some time and thought.

The development experience is a good one. The language (Objective C), iPhone SDK and the development tools are well done. Objective C is a powerful object oriented language and the iPhone SDK is well abstracted and easy to use. If you need access to the more low-level API's they are readily available. Interface Builder is spectacular and it puts other interface/view building applications to shame. Xcode is still way behind other IDEs but it is very usable nonetheless. Their performance and debugging tools are excellent. Even when your code is well formed and well tested you have the potential to have memory leaks, and their memory leak detection tool is great for that kind of problem.

From a professional development standpoint the platform is ready to go. A developer can test drive their code using OCMock and OCUnit. If you prefer a different testing platform I recommend looking into rbiphonetest.

I was able to successfully create an iPhone application using Atomic Object's Presenter First style of development. However, bootstrapping the application becomes problematic when the application uses a great deal of composition. It would be nice if a dependency injection framework existed and was available for iPhone development. However, it is possible to use Interface Builder to instantiate all of the singletons in the system and inject them, which could be an acceptable alternative.

Or, How to test your Presenter classes

Our first choice was NMock, primarily because we had experience using it on previous projects and we knew it could get the job done. Before too long, though, we switched to using Rhino Mocks. The primary reason for the change (if I am remembering correctly) was because of its superior event handling capabilities, and we grew to prefer its explicit use of the record-playback-verify mocking model.

Rhino Mocks was one of the first libraries to directly support event registration and event raising by mocks (at least, it was the first we had used). This was a big advantage over having to add a SubscribeEvent() method to our view and model interfaces and having to use syntactically obscure paradigms to capture and fire the event raisers. With Rhino Mocks we could add public events directly to our interfaces and (fairly) explicitly capture the event raisers.

Read the rest of this entry

Have you ever needed to display Unicode characters in a Win32 MessageBox from a Ruby script? My pair and I needed to do just that and so I thought I would share what we found.

There are a number of ways to access Win32 calls from a Ruby script. The code we were working with did the following:

1
2
3
4
5
6
7
8
9

require 'dl'
def show_message_box(message, title)
  mb_ok = 0
  mb_iconexclamation = 48
  
  user32 = DL.dlopen("user32")
  message_box = user32['MessageBoxA', 'ILSSI']
  message_box.call(0, message, title, mb_ok | mb_iconexclamation)
end

That works well until you need to display Unicode characters in the MessageBox. It turns out the MessageBoxA version of the function is for ASCII characters. There is another version of the API call, MessageBoxW, that can handle Unicode, or wide characters. So the issue becomes converting your Ruby string into a wide string so it can be passed to MessageBoxW. The MultiByteToWideChar Win32 call can do this for you. And the windows-pr gem (from win32utils) adds a nice ruby wrapper around the function.

gem install windows-pr

1
2
3
4
5

require 'windows/unicode' 
include Windows::Unicode 
$KCODE='UTF8'

str = multi_to_wide("This is a test")

This seemed to work quite well for the most part. However, from time to time we would see garbage text showing up at the end of our messages. It could easily be reproduced if the message was very short.

Having done enough C/C++ coding to recognize a string that was not being null terminated, we experimented with adding null characters to the end of the string. It turns out a wide null terminator (”\0\0”) is needed. The following code will properly display Unicode characters in a Win32 MessageBox:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

require 'dl'
require 'windows/unicode' 
include Windows::Unicode 
$KCODE='UTF8'

def win32_wide(text)
  multi_to_wide(text) + "\0\0"
end

def show_message_box(message, title)
  mb_ok = 0
  mb_iconexclamation = 48
  
  user32 = DL.dlopen("user32")
  message_box = user32['MessageBoxW', 'ILSSI']
  message_box.call(0, win32_wide(message), win32_wide(title), 
                             mb_ok | mb_iconexclamation)
end

show_message_box "MessageBox displayed from Ρουμπίνι", "¡Alert!"