A Thought on Monte Carlo Simulation Using Parallel Asynchronous Web Services with .NET and SharePoint

Monte Carlo Simulation is a technique used to estimate the likely range of outcomes outputted by a complex process by simulating the actual process with randomly selected data generating conditions that are true to the process model a large number of times. (In fact, the more you do it the better your data.) The Monte Carlo method is best applied whenever a deterministic solution would either be too computationally intensive or if such a solution does not exist whatsoever.

Monte Carlo Simulation is used in/with

  • Physical sciences
  • Design and visuals
  • Finance and business
  • Telecommunications
  • Games

Monte Carlo Simulation is not a “what if” process. What if’s require single point estimates and use deterministic modeling. Basically you are using best case, worst case, and so on. By using Monte Carlo you consume large random samplings, sourced from probability distribution functions, to produce a large range of outputs which in turn can allow you with greater confidence to produce a narrower range of outputs. In other words you are not using equal weights for each scenario.

Why is this pertinent? Well, stay with me on this one, Markov chain methods are extremely useful for generating sequences of random numbers to accurately reflect rather complicated desired probability distributions, via a process called Markov chain Monte Carlo methods. A tool that is used to generate simulations from a probability distribution…

The Google PageRank of a webpage is defined by a Markov chain.

And the penny drops…

Now, back to the point.

Depending on the degree of accuracy ultimately required, millions or billions of points may need to be tried. Distributing billions of point calculations across multiple servers running Monte Carlo Simulations via web services would parallelize the process and generates results VERY quickly. Good in concept but how to do it?

As defined by the W3C a web service is “a software system designed to support interoperable machine-to-machine interaction over a network.” Running web services on IIS has advantages not limited to:

  • You can grow your “cluster” by just deploying the web service to new nodes.
  • Each web service call with IIS is a thread which should have obvious and positive performance implications.
  • Web services provide a relatively simple and straightforward method of distributing parallel problems across multiple compute platforms.
  • Web services are written like traditional functions, they are easily parallelized without hand-coding a multi-threaded application, custom writing a message passing interface or using other high performance computing management software.

Needless to say, unless your requirements can be served by parallel computations, which would have no dependency on others in the pipe, this is going to become very difficult or rather “challenging” 🙂 very, very quickly.

So how could SharePoint fit in? SharePoint is perfect for acting as a landing point for your data. In and out. Companies benefit by building intelligence into their document libraries and lists with workflows. With workflow, SharePoint can act as a central hub for the data, sending it out to a queue which distributes to nodes on the network. Upon return, the data could be used to populate lists, document libraries, notify people/groups, and more. Search, BDC, Security, and all the other features in SharePoint make this concept a compelling one.