Nov 16, 2009

S’Zockerpapier het abgschlage

Don’t quote me on this, colleagues at home, readers, or anybody out there … because I do not even know the source of this Swiss-German saying, but the meaning is the following:

  • something is not as sweet as it has been once

You can apply this to this years PDC give-aways:

DSCN2773

The traditional bag is missing some potency and this years drinking bottle was made from recyclable plastic only. And yes, the T-Shirt is even missing …

Let’s hope this is not foreboding of the upcoming technologies, products and services from Microsoft ;-)

Windows HPC Server 2008 and how to run your jobs

A twelve hours flight to Los Angeles is a very long time that can be used in a couple of way. One is sleeping, using the on board entertainment system (see on the right; so not to sleep) or reading.

One job you come across every couple of years (or even month as did I lately) as a Windows developer is running computational work that last for a couple of minutes, hours or event up to days. Microsoft Windows HPC Server may look promising to replace a lot of your distributed runtime infrastructure code. It contains as a major component a job scheduler I will make some notes about here.

Overview

HPC 2008 brings high performance computing in a cluster environment to the Windows platform. A so-called job may contain several tasks, which basically are single executables (a simple sequential program or an already parallel one) you plan to run. There are a lot of ways to (1.) submit your job:

  • Job management console
  • CLI & PowerShell
  • SOA APIs (WCF)
  • COM APIs
  • .NET APIs or
  • WS-HPC Basic Profile (Web service)

Job scheduling (2.) can get quite sophisticated using First-come-first-serve, exclusive scheduling, resource optimizing. Job execution (3.) goes through the states Queued, Running, Finished, Failed or Cancelled. The scheduler decides according to defined requirements which job to run. Failed jobs can get automatically re-run. Jobs can run in parallel on several nodes or on several local CPUs. Tasks are usually not designed to communicate (I may come back to this in another post). Jobs may even get preempted (killed or slowed down to let other jobs be run before).

The head note (which can also be a compute node) acts as the central job management point. Jobs are stored in an underlying SQL-Server (Express?) database and execute under the submitting users account. There exists only one queue for a cluster.

(source: Microsoft)

Job Submission

Jobs get priorities, allocated resources (nodes, CPU and run time), dependency information or node exclusivity upon submission. A job can be submitted from the command line like this:

job new [options] [/f:jobdescriptionxmlfile] [/scheduler:host]

The two primary client user interface tools to use are the Job Management Console and the Administration Console. Jobs defined with the new job wizard can be saved to a job description XML file (that is intended to be used or even automatically generated). Message-Passing-Interface (MPI) executables must be prepended with mpiexec.

(source: Microsoft)

Job input (data) and output (results) can be on the local node or on a file share. Large and static data should be copied to the nodes whereas small and dynamically updated data should be placed centrally. Data should be cut into pieces (fragmented) and run in so-called “sweep”, which is some kind of (for int i = 0; i < 100; i++) – loop index.

Jobs can also be submitted using the .NET API and the Microsoft.Hpc.Server namespace classes Scheduler (to connect and submit), ISchedulerJob job = scheduler.CreateJob(), job.AddTask(t), etc.

Another way is to call the scheduler using Web Services and the HPCPBClient class (hpcbp.CreateAcitivity(jdsl), hpcbp.GetActivityStatuses(), etc.).

Job Scheduling

The most simple scheduling is a FIFO queue. “Backfill” is a method where time windows are used for small jobs to run according to the definitions of the large submitted jobs. The “resource matchmaking” method is done by the scheduler according to compute, network and application requirements. “Job templates” are administrator or system owner predefined jobs that can be used by high performance computing client users.

Microsoft seems to have done a good job when it comes down to security. As mentioned above, jobs are executed under the submitting users account. Credentials are passed during submission and safely (encrypted, etc.) stored to be used when the job needs to run. Credentials are passed by secured .NET remoting channels to the nodes and deleted after the job run.

Picture: The fail-over topology from Microsoft

Please note the HPC 2008 Server supports the MPI Standard by it’s own MS-MPI implementation.

HPC 2008 Server looks reasonable priced at around 500$ per node (as I found out after a quick search). You are getting some Quality-of-Services you probably cannot code yourself for this money. So have a look yourself next time defining IJob, IJobScheduling interfaces in your project.

Nov 6, 2009

IE8 vs. Google Codesearch

Google codesearch (www.google.com/codesearch) is great to search for source code. Since installing Internet Explorer 8 recently there is an error on the page. Who’s the culprit:

image

Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30729)
Timestamp: Fri, 6 Nov 2009 14:45:24 UTC

Message: Not implemented

Line: 339
Char: 257
Code: 0
URI:
http://www.google.com/codesearch/js/CachedFile/F1A2CB189D0FCB1FF201C42BF6A5447C.cache.js

I found out IE8 compatibility view does the trick:

image

Google states they are working on it http://groups.google.com/group/google-code-search/browse_thread/thread/587e5f2e521b384d

Oct 27, 2009

F# – sharp what?


I’m really happy with C# as an expressive, powerful and readable language (after recently being somewhat unfaithful with VB.NET). For a customer project doing heavy mathematics I had a look on the suitability of F# for concurrent table- and matrix-oriented math.

(logo from whattofix.com; not found one from Microsoft!?)

Overview

On the Microsoft F# Developer Center I got a first overview (e.g. F# will be part of Visual Studio 2010).

F# is a functional, compiled but object-oriented language. An interactive console allows for initial coding:

image

There is no autocompletion, syntax highlighting or intellisense. Errors are quite understandable however.

Or you can create a new Visual Studio project:

image

There is useful little video on channel9 that gives a first idea of the language. Since Cobol/Lisp I have not felt so “strange” with a new language …

Downloads

Installation

The installation for VS2008 took some minutes:

image

Learning

In the samples there is a solution called Samples101 that you can build and run:

image

Concurrency

On the development center site there a book chapter about asynchronous and concurrent programming (Apress). Unfortunately this book is based on an older version. I found some slides from Matthew Podwysocki.

The technique I am interested in is called “asynchronous workflows” (a powerful set of techniques for structuring asynchronous programs in a natural way).

// mleder.blogspot.com : my.net

// my first f# concurrent program

 

#light

 

// import or open namespaces

open System.IO

open Microsoft.FSharp.Control.CommonExtensions

 

// define array of key/value pairs

let people = ["Markus", "F# is cool";

               "Jonas", "eff what?";

               "Aaron", "(doing something different)";

               "Elias", "bla bla bla"]

 

// define a function to be executed asynchonously                      

let myAsync(name:string, comment:string) =

    async {

        System.Console.WriteLine("Waiting {0} ... {1} / {2}", System.Threading.Thread.CurrentThread.ManagedThreadId, name, comment)

        System.Threading.Thread.Sleep 1000       

        System.Console.WriteLine("Waited! {0}", name)

    }

 

// change the number of threads in the pool

// let ok = System.Threading.ThreadPool.SetMinThreads(10,10)

 

// start function on thread pool threads

for nm,url in people do

    Async.Start(myAsync(nm,url))

 

// wait for termination (using the "pass-forward" or "pipe-forward" operator to chain calls)

System.Console.ReadKey() |> ignore

Which seems to run on 2 pool threads (instead of 2 per CPU which would mean 4) !?:

image

Uncommenting the SetMinThread line above gives a different behavior

image

F# Summary

  • a language you have to get used to
  • steep learning curve (for a C# guy like me)
  • not widely used yet. Check maintainability and people skills.
  • suited for certain problem domains (like mathematics, analytics, simulation, …); to be preferred to other CLR languages
  • building on existing .NET framework assets
  • well integrated into Visual Studio (even if my often used “go to definition”/F12 does not work)
  • concurrent and asynchronous operation is an integrated part of the language

Oct 14, 2009

ASP.NET MVC 2 Resources

I’ve been asked about MVC tutorials, blogs, how to learn. Here are some useful links to get started for all of you:

ASP.NET MVC

Have fun!

Sep 23, 2009

Unit Testing – A Book you have been waiting for

Does your project have low test coverage, unreadable test code, high investments in maintaining unit tests? Do you feel writing unit test being awkward and time consuming, questioning the overall benefit, not trusting some of the unit tests? Ever had discussions about fake/stubs/mocks, what is better Record-Replay or Arrange/Act/Assert (triple A) syntax, where to set up your fakes, differences between unit and integration tests, testing best practices, different tools and test frameworks?

No? Working as a hair dresser or what? Then you can easily forget about the following …

I just finished reading Roy Osherove’s (300 pages) book “The Art of Unit Testing” which is an easy but MUST-READ book, I’ve been waiting for a long time now. You do not have to agree with all of his views, but definitively opens up your mind concerning unit testing.

Go an get it. Or if you are a team lead or manager go and get a stack of it. Your investment will pay back soon!

Sep 22, 2009

VS MSTEST Test Run Deployment Item Error – DLL not trusted

I remember having this error for a couple of times already during the last few years – so I write a post of my own about the solution to the following issue:

Running Visual Studio 2008 (MSTEST) unit tests fails with:

Failed to queue test run '2009-09-22 17:11:11': Test Run deployment issue: The location of the file or directory 'sqlite3.dll' is not trusted.

image

There were some hints about using the caspol tool to add full trust to the assembly which DID NOT WORK FOR ME:

C:\Program Files\Microsoft Visual Studio 9.0\VC>caspol -m -ag 1.2 -url \\localhost\c$\project\tools\* FullTrust

Microsoft (R) .NET Framework CasPol 2.0.50727.4016
Copyright (c) Microsoft Corporation.  All rights reserved.

The operation you are performing will alter security policy.
Are you sure you want to perform this operation? (yes/no)
yes
Added union code group with "-url" membership condition to the Machine level.
Success

A better approach was:

1. Remove the reference to the dll from VS
2. Shut down VS
3. Right click the assembly (here the sqlite3.dll) und click the UNBLOCK button.
4. Restart Visual Studio
5. Re-add the reference to the file

image

(thanks to markrob35)

Sep 15, 2009

WLW Syntax Highlighting with CopyAsHtml

Some of my fellow blogger colleagues at my company hinted using SyntaxHighlighter to post source code. As I am using CopyAsHtml (Visual Studio plugin) for quite some time now and found out about the HTML view in Windows Live Writer

image

I keep sticking to it. Looks much better again:

   21 private void timer1_Tick(object sender, EventArgs e)

   22 {

   23     Color c = ColorUnderCursor.Get();

   24     this.label1.Text = "Color under cursor = " + c.ToArgb().ToString("x8") + " [hex]";

   25     this.label2.Text = c.ToArgb().ToString() + " [int]";

   26 }

Sep 14, 2009

Color under Cursor in C#

To easily “steal” some styles and colors from a given web app, I wrote the following little application to find out the color under the mouse pointer.

image

And here’s the code for the major parts:

using System;
using System.Drawing;
using System.Runtime.InteropServices;

namespace ColorUnderPixel
{
    public class ColorUnderCursor
    {
        [DllImport("gdi32")]
        public static extern uint GetPixel(IntPtr hDC, int XPos, int YPos);

        [DllImport("user32.dll", CharSet = CharSet.Auto)]
        public static extern bool GetCursorPos(out POINT pt);

        [DllImport("User32.dll", CharSet = CharSet.Auto)]
        public static extern IntPtr GetWindowDC(IntPtr hWnd);

        /// <summary>
        /// Gets the System.Drawing.Color from under the mouse cursor.
        /// </summary>
        /// <returns>The color value.</returns>
        public static Color Get()
        {
            IntPtr dc = GetWindowDC(IntPtr.Zero);

            POINT p;
            GetCursorPos(out p);

            long color = GetPixel(dc, p.X, p.Y);
            return Color.FromArgb((int)color);
        }
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct POINT
    {
        public int X;
        public int Y;
        public POINT(int x, int y)
        {
            X = x;
            Y = y;
        }
    }
}

[Update] Please note that the hex values must be reversed to represent a RGB value like in a CSS

background-color: #3b5998;

Human Interaction Workflow

Windows Workflow Foundation 4.0 does not provide explicit framework classes or tools for workflow human interaction (e.g. waiting for a real actor of a workflow to make a decision how to continue).

Approach

Therefore we gave the following approach a try:

  1. Create a workflow with a SQL persistence extension that represent an order confirmation process
  2. Send email to actor, including a link to a ASP.NET application, a URL with request argument (the workflow instance id)
  3. Add a pick activity with two pickbranches, each including a aec.CreateNamedBookmark(<bm>)
  4. Create an ASP.NET application page to confirm a process, loading the workflow instance and calling ResumeNamedBookmark(<bm>)

Here a some details.

Bookmarks

To block the workflow and wait for human interaction we used the concept of “Bookmarks”. A bookmark is a named point in your workflow to unload and later resume. To create a bookmark and unload we created a simple custom activity:

public class CreateBookmark : NativeActivity
{
    public InArgument<string> Bookmark { get; set; }

    protected override void Execute(ActivityExecutionContext context)
    {
        context.CreateNamedBookmark(Bookmark.Get(context), new BookmarkCallback(BookmarkCallback));
    }

    private void BookmarkCallback(ActivityExecutionContext executionContext, Bookmark bookmark, object value)
    {
        executionContext.RemoveAllBookmarks();
    }
}

Sending Email

[Credits for this code to Alex]

Before we can unload our workflow we need to send an URL to the human. The link requests a page that resumes our workflow later on.

public class SendMailActivity : CodeActivity
{
    public InArgument<string> From { get; set; }
    public InArgument<string> To { get; set; }
    public InArgument<string> Subject { get; set; }
    public InArgument<string> Body { get; set; }

    protected override void Execute(CodeActivityContext context)
    {
        var fromAddress = new MailAddress(From.Get(context));
        var toAddress = new MailAddress(To.Get(context));

        var addresses = new MailAddressCollection();
        addresses.Add(toAddress);

        using (var message = new MailMessage(fromAddress, toAddress))
        {
            message.Subject = Subject.Get(context);
            message.Body = Body.Get(context);

            var mail = new SmtpClient("localhost", 25);
            mail.Send(message);
        }
    }
}

Get Workflow Instance Id

We did not find a simple way to get the workflow instance identifier in our workflow. To work around this we created another simple activity:

public class WorkflowIdActivity : CodeActivity<Guid>
{
    protected override void Execute(CodeActivityContext context)
    {
       Result.Set(context, context.WorkflowInstanceId);
    }
}

Pick and Pickbranch

To summarize: we have a workflow that sends a link to a human, persists and unloads. The question is now (the answer has cost me a beer) what WF4 construct to use to continue our workflow; resuming on two parallel branches. We used the pick and pickbranch activities and included a CreateBookmark trigger activity:

<p:Pick>
  <p:PickBranch>
    <p:PickBranch.Trigger>
      <w:CreateBookmark Bookmark="[‘Accepted’]" />
    </p:PickBranch.Trigger>
    <e:SendMailActivity Body="[‘Your order has been accepted’]" From="[‘goodboss@email.com’]" Subject="[‘Laptop order’]" To="[‘goodboss@email.com’]" />
  </p:PickBranch>
  <p:PickBranch>
    <p:PickBranch.Trigger>
      <w:CreateBookmark Bookmark="[‘Rejected’]" />
    </p:PickBranch.Trigger>
    <e:SendMailActivity Body="[‘Your order has been rejected’]" From="[‘badboss@email.com’]" Subject="[‘Laptop order’]" To="[‘badboss@email.com’]" />
  </p:PickBranch>
</p:Pick>

The Web Page

Last step is to build a simple web page, including a green and a red button. Note how we parsed the URL argument to get the workflow identifier.

public partial class OrderConfirmationForm : System.Web.UI.Page
{
    static SqlPersistenceProviderFactory persistenceProviderFactory;
    static AutoResetEvent instanceUnloaded = new AutoResetEvent(false);
    private Guid workflowId;

    protected void Page_Load(object sender, EventArgs e)
    {
       string id = this.Context.Request.Params["id"];
         workflowId = new Guid(id);
    }

    protected void btnAccept_Click(object sender, EventArgs e)
    {
        ResumeWorkflow("Accepted");
    }

   protected void btnDecline_Click(object sender, EventArgs e)
    {
        ResumeWorkflow("Rejected");
    }

    private void ResumeWorkflow(string bookmark)
    {
        SetupPersistence();

        PersistenceProvider persistenceProvider = persistenceProviderFactory.CreateProvider(workflowId);
        WorkflowInstance instance = WorkflowInstance.Load(new OrderProcessing(), persistenceProvider);
        instance.Extensions.Add(persistenceProvider);

        instance.OnUnloaded = () => instanceUnloaded.Set();

        instance.ResumeBookmark(bookmark, null);
        instanceUnloaded.WaitOne();

        ClosePersistence();
    }

    static void SetupPersistence()
    {
        persistenceProviderFactory = new SqlPersistenceProviderFactory(@"Database=Instances;Integrated Security=True", false, false, TimeSpan.FromSeconds(60));
        persistenceProviderFactory.Open();
    }

    static void ClosePersistence()
    {
        persistenceProviderFactory.Close();
    }
}