Moving large numbers of records with SqlBulkCopy

Sometimes you have to move large amounts of data in, out, or around SQL. There are plenty of ways to do that but few are as fast as using SqlBulkCopy. It is not fast. It is blazingly fast. As with anything else it has its place and limitations but it is well worth understanding at the least.
It is pretty easy to work with as the below example shows…
static void CopyData(DataTable sourceTable, SqlConnection destConnection)
{
// new method: SQLBulkCopy:
using (SqlBulkCopy s = new SqlBulkCopy(destConnection))
{
s.DestinationTableName = “SalesCopy”;
s.NotifyAfter = 10000;
s.SqlRowsCopied += new SqlRowsCopiedEventHandler(s_SqlRowsCopied);
s.WriteToServer(sourceTable);
s.Close();
}
}

When used properly it can make seemingly large data volume transactions trivial.

Euler’s identity: a mental segue

Working with code as a creator, configuror (is this really a word?), or troubleshooter has rewards and costs. One of the rewards is seeing / discovering crisp and novel logical patterns used to translate business processes from physical to digital.
Recently I was discussing with somebody how best to present large amounts of somewhat abstract data related to individuals with a goal of ‘perceiving’ subject matter experts in as fluid a fashion as possible. (Clearly I struggled with this sentence.) Musing for a few days on the topic I started to visualize a way to map each persons data points onto a sphere and what the values of those data points could, and should, do to the sphere and its surroundings. Then I remembered something I had not thought or heard anybody mention in a long long time. Euler’s identity.
Euler’s identity is not something most people think or even know about. Considered by many to be exceptionally remarkable for its mathematical beauty. It is often mentioned as the literal pinnacle of mathematical elegance and beauty. And it looks like this:
e^{i pi} + 1 = 0,!
It comprises of three basic arithmetic operations that occur exactly once each:
  • addition
  • multiplication
  • exponentiation.
The identity also links five fundamental mathematical constants:
  • The number 0.
  • The number 1.
  • The number ”π”, which is ubiquitous in trigonometry, geometry of Euclidean space, and mathematical analysis (π ≈ 3.14159).
  • The number ”e”, the base of natural logarithms, which also occurs widely in mathematical analysis (e ≈ 2.71828).
  • The number ”i”, imaginary unit of the complex numbers, which contain the roots of all nonconstant polynomials and lead to deeper insight into many operators, such as integration.

So what does that mean? Euler’s formula demonstrates, albeit for still not terribly clearly known reasons, that there is an intrinsic connection between complex exponential functions and trigonometric transformations.

So… what does that really mean? Sometimes other people put it best. “The true beauty of Euler’s Identity comes from the fact that, while the true nature of several of these constants continues to remain a mystery to mathematicians (though it is clear that they possess many real-world manifestations), within the confines of this equation they all work together in such a way that they interlock like pieces of a mathematical jigsaw puzzle, the end result of which has the mathematical traveler ending up right back where he began – at the journey’s origin.
Okay…
If it does not ‘make sense’ don’t worry. The mathematician Carl Friedrich Gauss supposedly said that if this formula was not immediately apparent to a student upon being shown it, that student would never become a first-class mathematician. Few people are. I know I’m not. But that still does not help if you cannot “see” it, so here’s a picture. A moving one that hopefully will act to shed some light on what Euler’s identity tells us. (I got the picture and the text below straight from Wikipedia.)

The exponential function ez can be defined as the limit of(1 + z/N)N, as N approaches infinity, and thus e is the limit of (1 + iπ/N)N. In this animation N takes various increasing values from 1 to 100. The computation of (1 + iπ/N)N is displayed as the combined effect of N repeated multiplications in the complex plane, with the final point being the actual value of (1 + iπ/N)N. It can be seen that as N gets larger (1 + iπ/N)N approaches a limit of −1.


If you’re still not sure what it is, Midnight tutor has a good video here: http://www.midnighttutor.com/EulerFormula.html

In case you were wondering. No, it did not help solve the original business need. But it did help to continue to keep alive the awareness that everyday people I work with produce clever, and sometimes stunning, solutions with logical blocks. It’s like working with artists who craft in ether.

Visual Studio 2010 problem with Vault

Working with Visual Studio 2010 and SQL 2008 I got this message out of the blue:
‘System.__ComObject’ to interface type ‘Microsoft.VisualStudio.OLE.Interop.IServiceProvider’. This operation failed because the QueryInterface call on the COM component for the interface with IID ‘{6D5140C1-7436-11CE-8034-00AA006009FA}’ failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).

Whoah… After a reboot, some thought, a few more attempts to build, and a bit of luck, I deduced with reasonable certainty that Vault, which I use for version control, was the culprit.
So I closed Studio 2010 and removed Vault. Then I opened the same solution and did a build with no problems. Next step is to figure out what to do about getting Vault back into play. Or not.

Search Server Express 2010: Trial period for this product is about to expire

Recently I did a Search Server 2010 Express deployment. Everything went without a hitch. Preflight, Installation, configuration, and first few crawls all worked the first time. (As they should!)
After 48 hours I cycled back to check the status of the system and found an issue. Specifically in Central Administration > Review Problems and Solutions > All Reports I found under the Configuration Category “Trial period for this product is about to expire.
Diving in for further detail I saw that the actual source of the error was that “The licensing information not available.” After a little bit of further researching I confirmed that to fix it all you have to do is:
  1. Go to “Central Admin” => Monitoring => “Review Rule Definitions”
  2. Click on “Trial period for this product is about to expire” rule under configuration
  3. Edit it
  4. Uncheck the “Enabled” checkbox.
  5. Save
Useful links

IIS Media Services 4.0 Beta is now available

There’s a new download available for IIS Media Services. Though still in Beta, real Beta not “Google Beta”, some new key features are:
  • One Encoding Format, One Server, Many Devices – including iPhone
  • Easy Content Conversio
  • Content Creation
  • Rich Player Options
  • New Live Smooth Streaming Features
If you are not already familiar with it, IIS Media Services is an integrated HTTP-based media delivery platform, that delivers true HD (720p+) live and on-demand streaming, DVR functionality, and real-time analytics support to computers, TVs, and mobile devices. By offering a complete multi-format media delivery platform and a proven Web server, highly immersive websites can now be managed from a single Web platform: IIS.

Download links

SharePoint 2010 Reference: Software Development Kit

The Microsoft SharePoint 2010 Software Development Kit (SDK) includes documentation and code samples for Microsoft SharePoint Foundation 2010 and for Microsoft SharePoint Server 2010, which builds upon the SharePoint Foundation 2010 infrastructure. The documentation includes detailed descriptions of the technologies that SharePoint Server 2010 and SharePoint Foundation 2010 provide for developers, reference documentation for the server and client object models, and step-by-step procedures for using these technologies and object models and programming with them.

This SDK also includes best practices and setup guidance to help you get started with your own custom applications that build and extend upon the SharePoint Foundation 2010 and SharePoint Server 2010 platforms.

Silverlight: Getting Started

Silverlight is not the easiest thing to get good at. In fact even if you are a seasoned .NET developer I would still call it a non trivial venture. Over at http://jesseliberty.com/2010/06/03/getting-started-with-silverlight-a-path-through-the-learning-material/ there’s a good short list on what to do if you want / or have to get into Silverlight development. Essentially he, Jesse Liberty, says:

“1. Go to the GetStarted page and download the components you need

2. Work your way through #1-5 in the “Start Learning Silverlight” section on that same page

3. Take a look at my Key Topic on Starting Silverlight (see the sidebar on the right)

4. Take a look at the Silverlight How-Do-I videos, picking subjects that seem interesting to you.

5. Join the forums, especially the Getting Started forum. They are the very best place to get fast answers. See my Note On Posting Questions On Forums.

UI Dragging : Dragdealer JS

Dragdealer is a drag-based JavaScript component that embraces endless front-end solutions. Elegantly crafted for JavaScript-aware coders.

The script is pretty flexible with options like:

  • working horizontally or vertically
  • defining the initial positions of the slider (x/y)
  • setting the number of steps for the slider
  • snapping to the steps
  • and much more.

Dragdelaer JS has callbacks on both release and move events which makes any type of interactions possible.

Speed up your PHP like Facebook does with HipHop for PHP

It’s a fact that script based languages and frameworks slow you down. PHP and its huge amount of frameworks is no exception to that rule. Until you realize that it is used on the busiest site out there – Facebook.
Scripted languages typically are embraced as they have advantages in being simple to learn, code, and debug while counterbalanced with a definite sacrificing of speed. On the other side of the spectrum, compiled languages usually have a steeper learning curve and are much more strict; counter balanced with the clear fact that they are considerably faster and better optimized for handling memory. Facebook seems to have found a balance between both worlds with HipHop for PHP.
Facebook developers write lots of their core services in other languages, and then PHP is used as the glue that queries those services and builds the actual pages itself. Still, they have found that that really isn’t fast enough for what they want, so they went and wrote their own compiler, or rather an optimizer, called HipHop, which compiles PHP down to C++ which can then be compiled by G++.
You can download HipHop from here: http://github.com/facebook/hiphop-php

If you’re interested in using/trying some PHP frameworks you can start by looking at:

Data Visualization: Earthquakes

Visualization of data is a topic I have a more than passing interest in. It’s a fundamental part of my daily routine as I help people understand what they have and show them new methods and tools with which they can discern patterns that may be hidden.
Sometimes I find things that are interesting or just well done. Such as this earthquake data from the USGS. Looking at the image below you can see data points (location) with value (seismic strength) and recency (date/time) clearly and concisely.