Monday, May 24, 2010 9:54:13 AM (Pacific Standard Time, UTC-08:00)
Posted in NoSQL | Open Source | Talks
Thursday, April 22, 2010 1:01:01 PM (Pacific Standard Time, UTC-08:00)
Maybe you’ve heard people talking about ditching their SQL Servers and other RDBMS
entirely. There is a movement out in the software development world called
the "No
SQL" movement and it’s taking the web application
world by storm.
“Insanity!” you may cry, “for where will people put their data if not in a database?
Flat files? Tell me we aren’t going back to flat files.”
No, but in the relational model, something does has to give. The NoSQL movement
is about re-evaluating the constraints and scalability of data storage systems in
the light of the way modern web applications generate and consume data.
The outcry about flat files above is meant to highlight an assumption developers
often have about building data-driven applications: Data goes in the database (SQL
Server, Oracle, or MySql). Just maybe, if we are really cutting-edge, we might consider
storing our data in the cloud, but the choices generally stop there.
The NoSQL movement asks the question:
“Is the relational database (RDBMS) always the right tool for data storage and data
access?”
Starting from an RDBMS is virtually an
axiom of software development. However, those of us who are excited about
NoSQL believe that relational databases are not always the answer. I think this
highlights one of the reasons this NoSQL thing is called a movement. People are
realizing they have a choice where they thought they had none.
The converse is, of course, also true. The NoSQL databases are also not always the
right choice either. If you look carefully however, you will find that they are
a good choice much of the time. Don’t take my word on it. Ask Facebook, Twitter,
Digg, SourceForge, WebEx, Reddit and a bunch of other companies
here and
here
that are using NoSQL databases.
This move towards NoSQL is driven by pressure from two angles in the web application
world:
- Ease-of-use and deployment
- Performance - especially when there are many writers as compared to the number of
readers (think Twitter or Facebook).
|
Choosing NoSQL for Ease-of-Use and Deployment
I cover the programming model in detail as well as introduce the actual database
server below. For some vague motivation, let me just give you a quick look at how
you define the data model and maintain it.
- Define your classes in C# (largely) without regard to putting them in a database.
Related classes? Easy - one has a collection of the others.
- Create a simple DataContext-like class which exposes each top-level type that is
to be stored in the database. This is only a few lines of code per collection (think
of this as a table).
- Interact with the database using LINQ. This creates the collections (think tables),
sets the schema, etc.
- Maintain the database and evolve it by maintaining your classes from step 1. *
|
Why, in the name of all that is right, do we have to model our system twice? Once
in the database and once, in parallel, in code? With NoSQL, you have one place to
do that - in your C# classes.
* You may have to run a transformation tool if you’re making radical data changes,
but that’s true in SQL systems as well.
Choosing NoSQL for Performance
When the number of concurrent clients using your application - and thus your database
- is reasonably small (let’s say 500 users as a baseline) RDBMS can work great.
But what if that number grows? And if you are writing a web app, you definitely
want that number to grow. At 50,000 users, can you still run on a single instance
of SQL Server or MySql? How powerful does your hardware have to be to handle that?
What about at 500,000 or 5,000,000 users, still good?
I’m sure there are some of you out there thinking, “What a minute now! There are
plenty of systems with tons of users built upon relational databases.”
It’s true, there are. But how much expensive hardware and software do these require?
How easy is it to leverage *commodity* hardware and free software? A basic SQL Server
cluster might run you $100,000 just to get it up and running on decent hardware.
Rather than leveraging crazy scaling-up options, the NoSQL databases let you scale-out.
They make this possible (dare I say easy?) by dropping the relational aspects of
a database. Some NoSQL systems such as MongoDB get even better scalability by loosening
some of the durability guarantees – which they backfill somewhat with redundancy
(more on MongoDB shortly).
“Ok, ok. So it’s cheaper and simpler,” you say. “How much faster than the finely
tune system that is SQL Server 2008 can these open source NoSQL systems be?”
The answer is: MUCH MUCH FASTER. Here’s a simple comparison of running a bunch of
concurrent inserts into SQL Server 2008 and MongoDB on the same computer.
Looks like under heavy load, I’d say it’s about 100 times faster. I’m sure there
going to be tons of second guessing this graph and so on. Hold your comments please!
I’ll be posting a full performance comparison with source code soon. Let me just
say that I think the comparison was fair - I’ll back that up in a later post.
NoSQL and a New Programming Model
If we do not have joins and primary / foreign key relationships, how do we associate
related data? In NoSQL, there is a way to mimic foreign keys for certain relationships.
However the main answer is that you do not disassociate your data in the first place.
I’m sure that you’ve all heard of the
object-relational impedance mismatch.
A large part
of that mismatch comes from the fact that we normalize the data in our database
to the extreme and then use joins to reassemble that data. Not only does that cause
this so-called impedance mismatch, but those joins can be really slow and they can
be the death of any scale-out solution. The key to many of the NoSQL databases’
scalability is that they do not use joins. You simply save large swaths of your
data as a single blob (which in MongoDB’s case, is still deeply queriable).
Shortly we’ll look at an example where we build out a disconnected, offline RSS
reader that uses MongoDB and LINQ to store its data. But just think about how you
might structure your data storage if you could save entire object graphs and still
query them? Your "row" might be a Blog object which has an array of BlogEntries
which contain the entry text, link, date, etc. Then your *entire* query to pull
all the details of a single blog would hit a single “table” in the database. That
might look like this query which has one result:
var blog =
(from b in ctx.Blogs
where b.Id == requestedBlogId
select b).FirstOrDefault();
There are no joins or anything like that because you’re saving objects not columns
and those objects contain their collections already (e.g. RssEntries). There is
an important distinction to make here. These NoSQL databases generally are *not*
the same as object databases. They are what are known as document databases. There’s
actually a
big difference between the two.
Introducing MongoDB
The NoSQL database we are using in this example is
MongoDB.
This is free, open-source database which runs on Windows, Linux, and Mac OS X
systems. You can access it from many platforms including .NET, Ruby, Java, PHP,
and so on.
We’ll be using .NET and C# of course. You have several options when choosing
how
to access MongoDB from .NET but generally that means using LINQ and a light-weight
object-mapper on top of MongoDB itself. Note that common terminology might categorize
the object mapper that moves objects into and out of the database as an ORM. While
that’s OK, there is technically no "R" in this ORM because MongoDB is not relational.
Hence I’m calling simply an Object-Mapper (OM).
In MongoDB nomenclature, theses libraries are called drivers. My favorite .NET driver
is called NoRM. It’s being actively developed and was created by
Karl Seguin,
Andrew Theken,
Rob Conery,
James Avery, and
Jason Alexander.
You can find
NoRM on GitHub and discuss it in its related
Google Group.
If you want to learn more about MongoDB you should listen to these Podcast interviews:
Michael Dirolf also has a great book in the works. You can catch a preview of it
on
Safari Books Online.
Here’s the amazon page:
MongoDB: The Definitive Guide.
NoSQL in Action
Let’s write some code. The first step typically in a data-driven application is
to spec out the database. Then we’d use LINQ to SQL or Entity Framework to generate
the ORM classes. MongoDB is different. MongoDB has no schema or rather its schema
is flexible and defined via usage rather than being predefined in the database.
So our first step is to define the classes we’d be storing in the DB via NoRM.
We’re going to define 3 classes: Blog, RssEntry, and RssDetail. The Blog object
will contain a collection of RssEntry objects. In practice you might just go with
the Blog and RssEntry classes. But I wanted to model both the embedded case (Blog
+ RssEntry) and the loosely defined foreign key style relationship that mimic joins
(RssEntry + RssDetail). That way we can demonstrate both use-cases.
Here’s a taste of the Blog class:
public class Blog
{
public ObjectId _id { get; set; }
public string Name { get; set; }
public string Url { get; set; }
public string RssUrl { get; set; }
public List<RssEntry> Entries { get; set; }
// ...
}
Notice that it contains a collection (List<T> really) of RssEntry objects.
That’s the relationship supported by nesting. The Blog class just has this collection
as part of its data model.
The RssEntry class has the summary info for a blog entry:
public class RssEntry
{
public ObjectId _id { get; set; }
public Guid UniqueId { get; set; }
public DateTime PostedDate { get; set; }
public string Title { get; set; }
public string RssGuid { get; set; }
}
And the larger data is stored in the RssDetails class (for example the text of the
post):
public class RssDetails
{
public ObjectId _id { get; set; }
// this is kinda like the foreign key.
public Guid RssEntryId { get; set; }
public List<string> Categories { get; set; }
public string Link { get; set; }
public string Text { get; set; }
// ...
}
Let’s see how we insert an entire set of Blog data into the database. We begin by
generating the objects (Blog, RssEntry, etc) in memory and then serializing them
via NoRM to MongoDB much as you would in LINQ to SQL. The difference is this will
actually generate the collections (analogous to tables) if they don’t already exist
and it will define the implicit schema to match our objects:
void SaveBlogToMongoDb(
string rssUrl, XElement root, RssDataContext ctx)
{
Blog blog = new Blog();
blog.RssUrl = rssUrl;
blog.Name = GetBlogName(root);
blog.Url = GetBlogUrl(root);
blog.Entries = ParseEntries(root);
IEnumerable<RssDetails> details
= GetDetails(blog.Entries, root);
foreach (RssDetails detail in details)
{
ctx.Add(detail);
}
ctx.Add(blog);
}
Here we are using a class called RssDataContext which we wrote manually. It is very
similar to what LINQ to SQL and Entity Framework use to do the object-relational
mapping. Want to do a query? Do you know LINQ? Well then you’re all set:
var results =
from b in ctx.Blog
where b.Name.Contains( "MongoDB" )
select b;
How do you add a new entry to an existing blog and update it in the database?
void AddEntry(Blog blog, RssEntry entry)
{
blog.Entries.Add(entry);
ctx.Save(blog);
}
We leverage the fact that the blog.Entries collection is a List and just add to
it. Then save will update the record in the DB.
All this works great and is highly performant. But do be careful as not all the
LINQ operations are fully implemented yet in NoRM and some (like join) may never
be added because MongoDB doesn’t support it.
To get started, download MongoDB the tools and server here:
http://www.mongodb.org
You unzip the zip file and run the mongod.exe program. Be sure that you have created
the C:\data\db folder. It appears at first that you have to run MongoDB in a console
window. But you can register it as a Windows Service:
Here’s some helpful advice on installing MongoDB as a Windows Service (there is
a small bug you have to work around):
http://www.deltasdevelopers.com/post/Running-MongoDB-as-a-Windows-Service.aspx
There’s also a management console (and I mean "console"):
It’s a little different. You’ll get used to it. The means of interaction with the
server is through JavaScript rather than T-SQL and the storage format is a binary
form of JSON as you can see.
For a project I’m working on I’ve built a Windows Forms UI that lets me manage the
database easily by just adding an object data source and doing some drag-drop magic
in Visual Studio. Generally I look down upon that sort of development, but for an
admin tool it’s just fine.
Now It’s Your Turn!
Try it out for yourself. Download MongoDB and the NoRM driver and build some apps.
You may also want to check out the source code for my demo app:
Download Sample: RssMongoSample-Kennedy.zip
Got feedback? Write a comment or
contact me on Twitter where I'm @mkennedy or find me in
any of these other ways.
Recommended Reading:
Here are some other blogs on this subject.
Posted in Articles | ASP.NET | NoSQL | Open Source | Talks | Tools | web2.0
Thursday, March 11, 2010 1:09:49 PM (Pacific Standard Time, UTC-08:00)
I'll be in Boston, MA on March 22 to teach an open enrollment course for DevelopMentor. If you want to learn about WCF, WPF, Silverlight, LINQ, Entity Framework, and more there is still time to sign up (note the date may no longer appear on the public calendar). http://www.develop.com/course/new-net3-net35-linqMention this blog post and you'll receive a discount as well.
Posted in DevelopMentor | Talks
Wednesday, December 09, 2009 11:38:45 AM (Pacific Standard Time, UTC-08:00)
Posted in ASP.NET | DevelopMentor | Screencasts | Talks | web2.0
Thursday, November 05, 2009 8:20:00 PM (Pacific Standard Time, UTC-08:00)
At DevelopMentor we have been running a bunch of free webcasts. Last month it was TDD and Agile. This month we are running 4 webcasts celebrating the announcements around .NET 4.0, Visual Studio 2010, and PDC 2009. Join me Monday, November 23rd and register here: http://bit.ly/aspwebforms We’ll talk about integrating ASP.NET’s routing infrastructure into existing an ASP.NET WebForms application. This allows you to build SEO websites with URLs like http://dotnet.ubbuzz.com/tag/.NET+4.0 while still taking advantage of all the productivity features of WebForms such as post-backs, controls, UpdatePanel, and so on. We have room for a couple hundred more attendees so please register and be part of the fun. I promise lots of demos and somedisdainful comments about PowerPoint! Share it with your friends (social, virtual, real, and other types) using the widgets below! ASP.NET MVC: What’s that, you’d rather hear about ASP.NET MVC, not this creaky old WebForms stuff? That’s Brock Allen’s talk: http://bit.ly/intromvc WF 4: Is WF 4 and visual programming your thing? Check out Maurice de Beijer’s WF 4 talk: http://bit.ly/meetwf4 New Parallel Extensions your thing: Check out Andy Clymer’s PFX talk. (link to follow soon). Cheers, Michael
Posted in ASP.NET | DevelopMentor | Screencasts | Talks | Visual Studio
Wednesday, November 04, 2009 11:17:00 PM (Pacific Standard Time, UTC-08:00)
Recently Llewellyn Falco and I did a webcast for DevelopMentor where we demonstrated some TDD techniques and introduced Approval Tests. We let the audience choose our project and they chose Space Invaders. It was all great fun. Now the videos and MP3 streams are online and available for download.
Be sure to check out the write-up we did afterward where we talked about the tools and gave you a chance to try it for yourself: TDD Space Invaders Write-up You can also watch two other, higher level agile webcasts by Bill Nazzaro here: Agile Webcasts at DevelopMentor Cheers! Michael
Posted in DevelopMentor | Screencasts | Talks | Unit Testing
Wednesday, October 28, 2009 1:13:21 PM (Pacific Standard Time, UTC-08:00)
A joint post by Llewellyn Falco and Michael Kennedy
[Update: Get
the videos and additional downloads for this webcast.]
As a follow-up to our " Avoiding 5 Common Pitfalls in Unit Testing" article we did a webcast where we took a problem from the audience and solved it live and unrehearsed on stage. These kinds of performances are always a risk but that's part of what makes them fun. Of course, the question is could we have done it better? Here's your chance to try it for yourself (details below). The Problem: Our viewers chose to have us build the game Space Invaders. The first thing we had to do to sketch out a basic scenario we could implement. We started with a picture to remind what Space Invaders even was: This was too big of a scenario for us to tackle in the allotted 40 minutes for programming. So then we started by creating a simpler scenario which we sketched out on the "whiteboard": Click for full size image. Flushing Out the Scenario: In doing this, a couple of things were revealed about the game. First, we wanted to make the tank and aliens all be the same size so we could put them on a grid. But then we saw that our bullet wouldn't fit that story, so we introduced the idea of relative sizes. We also realized that even though we drew the block, it was too complex for the first scenario and it would have to wait. Notice that as we started writing the scenario in English, there are mistakes, irrelevancies, and problems with the order. This is OK. The thing to remember is that all of this was done for the sole purpose of creating a recipe for a scenario we could test. That scenario is the following:
[TestMethod]
public void TestSimpleKill()
{
// 1. Create a 15x10 board.
// 2. Place a 3x2 tank at 1x8.
// 3. Place a 2x2 alien at 7x3 heading west.
// 4. tank shoots
// 5. advance 4 turns
// 6. not won
// 7. advance
// 8. win
}
Now that we had the recipe, we could go about writing the code.
Here's your chance to play at home!
- Set your timer to 40 minutes.
- Create a new test project.
- Paste that method above.
- Translate the comments into code.
If you believe there's a better process, we invite you to try that as well.
We made it to step 4 during our presentation (download code below) and estimate another 15 minutes would have had the whole scenario done, tested, and well-factored.
Stories vs. Requirements (stories win):
We'd like to point out a couple of things about the story. First, it was quick to write the story. We did it in 5 minutes. Second, it translates well to code because it has behavior and objects working together. Let's compare that to the requirements that this story flushed-out.
- Need a board
- Boards should have width & height
- Boards contain game objects
- Game objects have a witdth, height
- Game objects have the ability to move each turn
- Aliens move either left or right each turn
- Bullets move either up or down each turn
- Bullets are 1x1
- Tanks are 3x2
- Aliens are 2x2
- The game is not won until all the aliens are killed
- The game is won if alll the aliens are killed
- An Alien is killed if it is hit by a bullet
- Tanks can fire
- Firing with a tank creates a bullet going up from the space directly above the tank
|
Now we want to point out that this requirements doc is much hard to understand than our story. For example, if you were to add more requirements (e.g. an alien also shoots) is that easy to determine whether we have complete requirements? It also takes much more effort to create and especially to tell if it is complete. People aren't made to handle requirement documents well but we are story-telling machines. We embrace this in our coding techniques.
We'd also like to mention some of the tools discussed at the end.
For remote collaboration we use:
Skype (audio / video)
VNC for screen keyboard sharing
RDP (windows remote desktop) -- requires Windows 2003/2008 server for pairing.
Source Control:
TortoiseSVN
TortoiseGit
Developer Tools:
Resharper
CodeRush
Testing Tools:
MsTest (in Visual Studio Professional and up)
NUnit
NCover
TortioseDiff
Approvals Tests
Approvals Tests CodeRush add-in
Rhino Mock
TypeMock
If you try this scenario yourself, please leave a comment about your experience.
Download the code and slides from the webcast here:
Code: TddWithLlewellynAndMichael.zip
Slides dmtdd.pdf
Cheers -- Michael and Llewellyn
Posted in DevelopMentor | Screencasts | Talks | Unit Testing
Monday, October 26, 2009 3:29:46 PM (Pacific Standard Time, UTC-08:00)
[Update: See the follow-up post here: TDD Invades Space Invaders]
Tuesday, October 27, 2009 at 10am Pacific time Llewellyn Falco and I will be giving a live, unscripted, and no safety-net demonstration of Test Driven Development (TDD) as part of the DevelopMentor webinar series (this particular series is a 3-part series on Agile development).
We already have a bunch of attendees registered. But we have room for as many of you who are interested in agile and TDD. Sign up here:
http://bit.ly/dm-tdd-m-and-l In addition to core TDD techniques, you will see how an amazing technique and set of tools designed by Llewellyn called Approval Tests makes writing tests as simple as verifying an image or text file. Tired of writing 50 lines of test code for every 50 lines of production code but you still want the power of TDD? You need to learn more about Approvals and we'll demo that live tomorrow!
I hope to see you all online. Feel free to help me get the word out by tweeting this or shouting it (see icons below). Cheers, Michael.
Posted in DevelopMentor | Open Source | Screencasts | Talks | Tools | Unit Testing
Wednesday, April 08, 2009 12:03:23 PM (Pacific Standard Time, UTC-08:00)
I recently wrote an article for DevelopMentor's Developments newsletter entitled Azure Storage. Read it at the
DevelopMentor website here:
http://www.develop.com/content/newsletters/aprilazure
I've republished here for my readers. Enjoy!
Developments: Azure Storage
by Michael Kennedy
[Listen to this article as a podcast: Azure-Storage-Article-Kennedy.mp3]
October 27th 2008, Los Angeles CA - It's 9 AM and Microsoft is
hosting PDC (their most forward looking developer conference).
Ray Ozzie and company are introducing Windows Azure:
A new platform which is their first foray into the nascent
world of large-scale utility computing. This scalable and reliable
platform-as-a-service functionality is commonly referred to as
"Cloud Computing" because it runs somewhere out there on the Internet.
Computing platforms that rival the reliability of the utility grids (e.g. electric and gas) which we daily take for granted have long been the stuff of dreams.
A few companies have realized this dream - Google and Amazon come to mind as a couple of the
rare exceptions who have accomplished this goal. These companies' web properties seem to handle
unbounded amounts of traffic with zero down time. The data centers, redundancies, software
engineering and operations know-how required to make this happen are exceedingly expensive.
Some reports have Google spending over $2.4 billion (that's 2,400 million dollars) on
data centers in 2007 alone.
Prior to large-scale cloud computing efforts (circa 2005), most of us could only dream of such scalability and reliability. Today we have at least three highly reputable companies offering some kind of pay as you go cloud computing platform - Microsoft, Amazon, and Google.
Microsoft's Azure is a new comer to the industry. But for .NET developers, it is not to be ignored. Azure allows you to use your existing skills to build essentially the same .NET applications you are familiar with and "deploy them to the cloud."
These scalable, reliable, and geographically-replicated applications
that run on Azure depend on data of course. Virtually all applications we write will be nothing without their underlying data. But if we simply use the tried and true methods of data storage such as the file system or a (single) database server our data is not all that scalable or reliable. Because we cannot have a scalable and reliable application without data, we need a new mechanism for storing and accessing data from our Azure applications.
Enter Azure Storage
Azure storage is the storage component of the Azure platform. It is actually three data services in one:
- Blob Storage - stores unstructured data essentially as a file, limited to 50 GB of data per blob.
- Table Storage - stores structured data that is somewhat like a
database. For full database capabilities there is a high level feature called Sql Data Services (SDS).
- Queues - provides interprocess communication functionality between various web and worker roles in your hosted services or even applications running outside of Azure. Queues can pass small xml or binary messages - less than 64 kb per message.
In this article, we will cover just the basics of the
three storage services of Windows Azure. I want to give you a sense
for what it's like to program against Azure Storage. At the base level all
access to Azure Storage uses pure REST APIs.
This means that you can access
it from any HTTP enabled platform / language. For example, to download the blob data
called "config.xml" in the container called "settings" for the Azure project "kennedy"
you would simply issue a GET to the Uri:
http://kennedy.blob.core.windows.net/settings/config.xml
To save data in a blob you do HTTP POSTs and PUTs in a similar fashion.
However, real life is full of edge cases, error handling, security, and
serialization which makes the pure HTTP model error prone. Thus, a sample library
serves as the de facto .NET API to Azure Storage and ships with the Azure SDK. It is
called StorageClient and can be found in default installs here:
C:\Program Files\Windows Azure SDK\v1.0\samples\StorageClient
We will examine working with each of the
storage services from the perspective of the StorageClient library -
but keep in mind that ultimately this library is a wrapper around a basic and open RESTful API.
Setting The Stage: The Sample Application
To explore Azure Storage I have written a simple photo sharing distributed application.
These set of applications allow users to upload photos to a photo sharing site.
These photos must be reviewed and approved by moderators of the site. Once approved, the
general public can view and interact with the photos. For a concrete example, you could
imagine writing a distributed version of the wallpaper sharing site
InterfaceLift and
deploying it on Azure in this fashion.
You can download the sample application and follow along if you want to see the full
source code and try it out yourself. Just be sure to start the Development Storage utility
that comes with the Azure SDK before running the application.
Our distributed application consists of three parts.
- The Uploader: A Windows Forms application that lets contributors upload images to the site.
- The Reviewer: A Windows Forms application that lets moderators view image submissions and either approve or reject them.
- The Website: An ASP.NET website for viewing the photos - this is our public facing application.
A typical use case might be as follows (see diagram below).
- We upload a photo submission with our uploader application. The photo is uploaded to Azure blob storage and a message is sent via an Azure Message Queue to all available reviewer applications. Additional information about the submitter is associated with the photo in Azure table storage.
- The reviewer application watches the message queue for new messages. When one arrives, the photo is added to a list of pending submissions. The reviewer can either reject (delete) the submission or approve it - move it to a permanent blob storage location where it will be publicly viewable.
- Users visit our website and can view all approved photos. This list will change in real-time because it is driven by the reviewer application. The web application simply pulls all photos from the approved photo container in Azure blob storage.
Saving Data: Creating Azure Blobs
To save data to Azure Blob Storage, you must realize blob storage follows the ACE pattern (Authority,
Container, Entity) to describe a blob. Authority is simply your Azure solution name. Containers are analogous to folders. And entities are analogous to files.
The listing below is essentially the code that runs when the uploader application uploads a pending image submission to blob storage.
Listing 1:

Sending Notifications: Azure Queuing
In addition to uploading the image to the pending images container in blob storage,
we will send a message to a message queue to notify any active or future reviewers of the new submission.
Listing 2.

Saving (More) Data: Structured Storage and Azure Tables
Finally, for the upload application, we must also save some information about the contributor.
In Azure Storage we have two reasonable places to store this information.
First, we could save this information directly in blob storage as meta-data
associated with the blob itself. This is straightforward and easy. But there is a big limitation: information in this meta-data is not queryable. Suppose I want get all images associated with a single contributor. There is no way in Azure Blob Storage to say give me all the blobs with this filter on the meta-data. You would have to pull the properties of every blob and do the comparison client-side. That's tantamount to filling a DataSet with "SELECT * FROM PendingImages" and it's a bad idea.
Instead we will use the third type of Azure Storage: Azure Table Storage.
Table Storage allows us to store data with up to 256 properties and
query this data as if it were a database. It is exactly what we need for the contributor information.
However, you must realize this is not a database. A better mental picture is a durable collection of
Dictionary object (as in Dictionary from System.Collections.Generics) with querying built
on top. I say this because there is no schema or relational constructs in Azure Table Storage. If you
need that, then you'll want Sql Data Services - a service on top of the core Azure platform.
The code to add an entry to Azure Table Storage does not fit into a single method as it's driven through the interaction of several classes we must define. Azure Table Storage can be accessed via ADO.NET Data Services (client-side) and this is the method we will use.
First we'll define a client-side schema for our entry by creating a class called Contributor which derives from the class TableStorageEntity (from the StorageClient library).
Listing 3.

Additionally we must define the tables and queries available to ADO.NET Data Services by created a class derived from TableStorageDataServiceContext and we do that below. We simply have one table called Contributors.
Listing 4.

With those two items in place, we can insert a "row" into Azure Table Storage as follows:
Listing 5.

As for querying Azure Table Storage that is very straight-forward. Because we are using ADO.NET Data Services, querying can be done via LINQ as in "from c in svc.Contributors select c.Name". Ultimately ADO.NET Data Services is also built on a RESTful API so this translates to the underlying HTTP REST calls. Alternatively, you can use that REST API directly from .NET or any other platform.
Waiting on Queues: The Reviewer Application's Code
Next, let's look at how we monitor and pull messages from Azure Queuing. Ultimately we must poll the queue using a RESTful HTTP request.
But the StorageClient resurfaces this to us as simple events.
Listing 6.

We won't cover how we move a blob from the pendingImages blob container to the approvedImages blob container which happens when a reviewer approves an image.
You can look at the sample to see how that is done.
Ultimately It's About the Website
Finally, let's look at the web application that actually displays the approved images. We don't do anything fancy such as paging or error handling that you'd see in a real application. But this will give you a good idea how to work with the blob data as a collection.
Here we'll create a BlobStorage object and access the BlobContainer approvedImageContainer as we have been in most of the listings. But then instead of saving or reading blobs, we use the ListBlobs method to simply list all the approve images in that container. In order to show the images on our webpage, we just use the BlobProperties.Uri and directly reference that in our HTML. Our ASP.NET application does not touch the data. Rather the consumers (IE, Firefox, Chrome, etc) of the HTML pull the image data directly from blob storage as they would from any web server.
Listing 7.

Now you have a good idea of the concepts and motivation behind Azure Storage. You have seen some typical usages of each of the three storage features: blob storage, table storage, and queuing. Our samples made use of the sample storage API library called StorageClient. Underlying this library we saw that Azure Storage is entirely accessed via RESTful APIs.
Want to get started? Visit http://www.azure.com and choose "Try It Now" to register for a CTP Azure account. You'll need to download the various SDK's listed on that same page. They will install the Visual Studio projects required for working with Azure as well as the Development Storage and Development Fabric so you can develop and debug your applications before deploying them to the cloud.
If you want some intensive, expert-lead training on Azure and associated .NET 4.0 topics be sure to contact DevelopMentor. Or call 800.699.1932 to find out what classes we have available today.
Posted in Articles | ASP.NET | Azure | DevelopMentor | Talks
Tuesday, March 24, 2009 11:02:03 AM (Pacific Standard Time, UTC-08:00)
I recently got the chance to record a screencast discussing REST-oriented web services in WCF. If you're interested in WCF you should definitely check it out because WCF and REST make an awesome combination.
WCF-REST-Kennedy-Peepleocity.wmv 35 MB (WMV HD)
WCF-REST-Kennedy-Peepleocity.mp4 96 MB (MP4 / iPad)
I cover building WCF services using REST princples, the WebGet and WebInvoke attributes, working with the SyndicationFeed & Rss20FeedFormatter classes, and configuration-free WCF hosting in IIS.
You can also download the source code of the project built in the screencast.
Finally, if you're willing to do without video you can download just the audio as an MP3.
Posted in DevelopMentor | Screencasts | Talks | web2.0
Tuesday, March 03, 2009 3:14:04 PM (Pacific Standard Time, UTC-08:00)
My esteemed colleague, friend, and fellow instructor at DevelopMentor Jason Whittington gave a great presentation on advanced .NET debugging recently at the Oklahoma City Developer's Group. They luckily recorded it on video and published it on their website so that it may " live on in the Google".
Debugging The Future: The Video by Jason Whittington
If you want to debug .NET applications right down to their memory footprint, this talk is for you. If you like this type of presentation, be sure to check out the classes we offer at DevelopMentor.
Posted in DevelopMentor | Screencasts | Talks
Monday, August 25, 2008 7:53:30 PM (Pacific Standard Time, UTC-08:00)
ClickOnce is a great deployment model for many Windows applications built with the .NET Framework. Too bad it isn't supported for C++, VB 6, or other technologies. Or is it...
Surprisingly, you can deploy your unmanaged apps with ClickOnce. You just need a tiny .NET app to get it started.
Here's how it works:
- Take your existing C++ project.
- Add to the solution a .NET console application.
- Change the project settings on the console application to "Windows Application".
- Write the following code for your Main method of your .NET launcher application:
static void Main(string[] args)
{
try
{
Process.Start( "TheRealApp.exe" );
}
catch ( Exception x )
{
string msg = "Error launch application:\n\n" + x;
string cap = "Error Launching Application";
MessageBox.Show( msg, cap, MessageBoxButtons.OK, MessageBoxIcon.Error );
}
}
- Set the build path of your C++ app to be in the main folder for your .NET application.
- Add the C++ app and its libraries (if any) as existing items in the .NET app.
- Change the build action to "Always Copy" as shown here:
- Then you publish your .NET application and when it runs, TADA!, the C++ app is deployed, versioned, and kept up to date as well.
 If you want to try it yourself, you can run this sample application here: Run Michael's Useless C++ App via ClickOnce...You can also download the source.
Posted in DevelopMentor | Talks | Visual Studio
Wednesday, July 02, 2008 10:08:00 AM (Pacific Standard Time, UTC-08:00)
When I talk about LINQ people often ask whether it’s possible to have dynamic queries with LINQ. An example of this is presenting the user with a UI that allows them to optionally filter by some criteria or other and sort by some criteria. This was straightforward if you built-up a SQL string in code. The compiled, static nature of LINQ makes dynamic queries appear difficult at first. They are not and this video and sample application shows you how to accomplish just this.  Screen shot of sample application
You
can download the source code here: Kennedy-Dynamic-Linq.zip (88.34 KB)And you can watch the screencast video here: Kennedy-Dynamic-Linq.wmv (35.7 MB)Of course, if you like this post, please kick it:
Posted in DevelopMentor | Talks | Visual Studio
Friday, April 04, 2008 6:16:07 AM (Pacific Standard Time, UTC-08:00)
You may have heard that the sessions for Mix 08 were posted online and you can view or download them until your heart is content. Mix 08 is becoming an increasingly important event in the .NET space. I spent a fair amount of time watching the sessions in the evenings. I was teaching a class and couldn't attend myself. So who wouldn't want to curl up with a cold beer, a warm laptop, and a bunch of .NET presentations? That's what I want to know. But, maybe you don't have time to go through all 89 sessions? Don't despair, I'll distill them down for you. BTW, I'd love to link directly to them, but SilverLight and Flash are broken
web metaphors (you can't link their content) so you'll have to find
them at http://sessions.visitmix.com/. Without further ado, I present to you my top 5 most important sessions from Mix 08: - T22 - Developing ASP.NET Applications Using the Model View Controller Pattern
Scott Hanselman
- T01 - Creating a RESTful API with Windows Communication Foundation
Aaron Sloman and Haider Sabri
- UX03 - The Back of the Napkin: Solving Design Problems (and Selling Your Solutions) with Pictures
Dan Roam
- T11 - What's New in Windows Presentation Foundation 3.5
Rob Relyea
- T26 - Building Applicaitons and Services with.NET Framework 3.5
Justin Smith
Of course the keynotes were great too - but they don't really belong in this category. So get out there and learn something! :) And kick it if you like it:
Posted in Talks
Wednesday, February 06, 2008 9:33:32 PM (Pacific Standard Time, UTC-08:00)
It was all going so smoothly.
Jason Whittington, Mark
Smith and I were teaching the big DevelopMentor
event here in Los Angeles (Guerrilla.NET)
when my presentation on the ThreadPool took a nose dive. It started with a great
joke involving Wilson (the volleyball from Cast Away).
Wilson and I built an application to compute a multiplication table where each
computation was (artificially) slow. To speed it up we threw it at the thread pool using
delegate.BeginInvoke. We figured that the ThreadPool would allocate
25 or so threads and the table would display quickly. Here's the expected output -
pretty much the same thing we've seen since about .NET 1.0:
Each color represents the thread that did that computation.
For the last 7 years, the behavior has been that as the ThreadPool was overloaded,
it would steadily start up new threads at the rate of one every 500 milliseconds until it hits
its upper limit (typically). Using Performance Monitor (perfmon) we can watch the
thread pool adding threads. It usually looks something like this:
Much to our surprise we saw completely different behavior. The thread pool added the
first 15 or so threads quickly (as expected) but then stalled. New threads were
not created every 500ms, instead they were added at increasingly long
intervals. My demo took almost twice as long to run as it had the last time
I did this demo a few months ago.
Jason, Mark, and I took this code, ported it back to .NET 1.1 and ran it side-by-side
with .NET 3.5 and here's what we saw (blue = 3.5, red = 1.1):
As of .NET 3.5 the upper limit of the ThreadPool was increased:
Knew that.
But, it appears that v3.5 of the CLR changes the policy for adding threads to the
thread pool. Rather than adding threads regularly when under load the thread pool uses
a logarithmic backoff. My colleague Jason Whittington remarked that this behavior
looked similar to the behavior of the thread pool in "Rotor"
(the shared-source version of the CLR). We speculated that this backoff algorithm
makes sense given the new 250-thread per CPU maximum - it would take a long time
to reach that if the runtime waits longer and longer to start a new thread. The 250-thread
limits makes it less likely that your application will deadlock the thread pool, and the
exponential backoff algorithm keeps the thread pool from creating too many threads too quickly.
Why should you care? Usually you won't, but it could have dramatic impact if you
count on that behavior. For example, in our contrived case, .NET 1.1 ran about twice
as fast as .NET 3.5 ( ! ):
Here's the program and source code (trimmed down to run in both .NET 1.1 and 3.5).
Math.zip (6.38 KB)
Needless to say, Wilson and I felt kinda stood up.
Please see the follow up post on Breaking changes and now there is screencast, movie version as well.
Posted in DevelopMentor | Talks | Visual Studio
Thursday, October 19, 2006 5:42:00 AM (Pacific Standard Time, UTC-08:00)
I'll be speaking tonight at The Central New Jersey .NET User Group (http://www.njdotnet.net) on one of my favorite topics: Five Fundamental Object Oriented Design Principles for Agile Development.
Thanks to Jason Beres (http://www.geekswithblogs.net/jberes) for inviting me to speak at his user group and welcoming me to New Jersey.
Once you've attended the talk, you might be interested in downloading the slides and sample code:
http://www.MichaelCKennedy.net/Talks/Downloads/NJdotNet/AgileDesign.zip
You also might want to look into some of the tools I was using in Visual Studio. See an older post on my tricked out Visual Studio.
Posted in Talks
Wednesday, May 10, 2006 4:31:51 PM (Pacific Standard Time, UTC-08:00)
Thanks to Reza Madani and Mike Vincent for having me speak at the combined meeting of OC VBUG and OC C# last night. It was a packed house which was wonderful, but I felt bad for folks who had to stand in the back for 2 hours. The audience participation was great. Thanks to everyone who asked questions or had comments. You can download the slides and samples from my website here: http://www.michaelckennedy.net/Talks/Downloads/OCVBandCSharp/AgileDesign.zipMany attendees also noticed the tricked-out Visual Studio I was using during the presentation. That was from CodeRush and Refactor! both highly recommended. You can check those out here: http://www.devexpress.com/Products/NET/CodeRush/http://www.devexpress.com/Products/NET/Refactor/and DevExpress has some very interesting screen-casts demos here: http://www.devexpress.com/Products/NET/CodeRush/Training.xmlAlso I was using the new Vista programming font Consolas, which you can download here: Consolas Font Pack for Microsoft Visual Studio 2005
Posted in Talks
Tuesday, May 02, 2006 4:28:23 PM (Pacific Standard Time, UTC-08:00)
Thanks to David McCater for hosting my Agile Object Oriented Design talk at San Diego .NET Developers Group on May 2nd. I met some great guys and thoroughly enjoyed talking about agile and OOD topics with everyone. You can download the slides at:
http://www.michaelckennedy.net/talks/downloads/sddotnetdev/agiledesign.zip
Posted in Talks
Monday, April 10, 2006 7:06:40 AM (Pacific Standard Time, UTC-08:00)
Dave, over at extremeplanner.com, wrote a nice summary of the talk I gave at XPSD. I completely agree with him that "a review of these fundamentals [is] refreshing." As you can see, I'm doing my part to spread the word:
Many pundits and authors in the software industry are busy promoting the next big product/feature/etc that they seem to forget that these things only provide second order benefits as compared to the more "fundamental" ideas such as good object oriented design. For example, looking through some of the user groups' upcoming meetings you'll find topics like:
- Looking at some Cool New Features of ASP.NET 2.0: Managing Membership and Role Management
- Leveraging the New Windows Mobile APIs
- Visual Studio Team System
- Closer to the Data: Using Managed Code in the Database with SQL Server 2005
Lots of neat, whiz-bang features, but I would contend that 90% of the audience would be much better off if they had heard a good talk about solid OOD or TDD or this list goes on. It's been my experience that much of these old (5-15 years), tried and true fundamentals are unfamiliar to most developers.
So here is my call to action:
If you are a speaker or author, reach for something big for your next topic. Yes it's easier to write/talk about the details of the next version of feature X. But you'll do your audience a service if you reach higher, think bigger, and go for something "fundamental."
If you are attending user groups and code camps, ask for and expect something more than a talk on the of latest version of product X.
Posted in Talks
Wednesday, April 05, 2006 10:39:00 AM (Pacific Standard Time, UTC-08:00)
I am tenatively scheduled to speak in San Diego, CA at the “San Diego .NET Developers Group” (http://sddotnetdg.org/). We haven’t arranged a date yet, but I think it will be around late spring / early summer.
This time I will again be presenting a talk on object oriented design and agile development, probably a slight variation on the general topics from my other talks.
I’ll provide more details as they become available.
Posted in Talks
Sunday, April 02, 2006 10:02:33 PM (Pacific Standard Time, UTC-08:00)
I’m very pleased to announce that I’ll be speaking in Orange County, CA on May 9th from 6pm onward.
I’ll will be presenting a talk on object oriented design and agile development at a joint meeting of the “Orange Country C# Developers Group” (http://sddotnetdg.org/) and the “Orange County VB.NET User Group” (http://www.ocvbug.org/).
I encourage anyone in the Anaheim / L.A. area interested in the interplay between object oriented design and agile software development to attend.
Posted in Talks
Saturday, April 01, 2006 9:45:57 PM (Pacific Standard Time, UTC-08:00)
[NOTE: This recent post refers to a much older (March 3, 2006) event.]
I want thank Carlton Nettleton and June Clarke for inviting me to present my “Five Fundamental Object Oriented Design Principles for Agile Development” talk at XPSD this past March. The audience participation was great and I really learned a lot myself!
You can download the slides and code samples here:
http://michaelckennedy.net/Talks/Downloads/XPSD/AgileDesign.zip
Posted in Talks
Saturday, April 01, 2006 9:40:58 PM (Pacific Standard Time, UTC-08:00)
[NOTE: This recent post refers to a much older (Jan 21, 2006) event.]
I want to thank everyone who attended my talk at the first Southern California Code Camp. I really enjoyed speaking there and I got the sense that many of you got something from the talk. In case you weren’t there and are interested, here’s the abstract:
Five Fundamental Object Oriented Design Principles for Agile Development
This session will present five object oriented design principles that facilitate agile software development. These general design principles promote the creation of testable, maintainable, and reusable software. The principles include the Open Closed Principle and the Liskov Substitution Principle. The interaction between Agile Development and these principles will be demonstrated using several code samples.
You can download the slides and code samples here:
http://michaelckennedy.net/Talks/Downloads/SoCalCodeCamp/AgileDesign.zip
Posted in Talks
|