Data Analysis & Simulation

Recently we have released a new version of our SDK. In this update, we have added a new property that lets you obtain the current licensing status of the SDK – for instance, you can determine whether the SDK is currently running in trial mode (using the Evaluation License), and if so, how many days are left until the evaluation period expires.

Consider the following scenario: you are building an application with a modular structure that, apart from its core feature set, provides some additional functionality through a number of modules, or add-ins, which can be installed and enabled on an optional basis. Now, suppose one of these modules uses the simulation or distribution fitting features of the SDK, and you want to give your users an ability to evaluate it prior to making a purchase decision. The new version of the SDK lets you easily integrate this logic into your applications, allowing you to create more flexible solutions that better meet your customers’ needs.

What is Cloud Computing?

For some time now, there has been a lot of buzz around cloud computing – the relatively new computing paradigm in which the resources, software, and information are shared on the computer clusters and delivered to the users on demand through the Internet. The idea behind cluster computing is not new: if your applications require a lot of computing resources or impose very strict reliability requirements which cannot be met by a single personal computer or a server, you can link a group of computers into a cluster that will provide a much better performance.

Why Not Build a Cluster Yourself?

Building and maintaining a computer cluster in your organization may have some downsides, such as large upfront investments into technology infrastructure.and high running costs. Of course, there are companies that will do the job of building and managing a computer cluster for you, but anyway, the bottom line is: depending on how loaded your cluster is going to be, it may or may not be economically feasible for your company to run it on-site. For instance, if you need to quickly perform a very CPU-intensive calculation (e.g. render a complex 3D scene), but only once a day, chances are the cluster will not pay off.

And here’s where cloud computing comes into play: you can have access to great computing resources, pay only as you use them, and not worry about the underlying technology infrastructure. These factors combined can provide a great economic benefit, and some major Internet players, including Amazon and Google, are already offering cloud computing platforms for those who want to make their businesses more efficient.

Is Cloud Computing a Good Fit for Risk Analysis?

As one might guess, not just any kind of application can be efficiently run on the cloud. Because at the core of a cloud is a number of computers linked into a cluster, it is very good at processing a large number of independent tasks, such as requests to a web server. That might be the reason why the cloud computing platforms offered by Amazon and Google are mostly used to run websites.

If you consider risk analysis, it looks like an ideal application to be run on the cloud: an input model of several megabytes that can be easily sent to the cloud, a need for huge computational resources to quickly perform Monte Carlo simulation and distribution fitting, sometimes a need for a lot of storage to hold intermediate results, and relatively small-sized analysis results that can be sent back to the users as text and graphics. Add to that the ever-increasing complexity of risk models used across various industries, causing analysts to wait for hours while their simulations are running, and you have a potentially good opportunity to make risk analysis more economically efficient.

To perform further research in this field, we have partnered with Supportex, a technology services company based in Czech Republic, Europe. Supportex has some good experience providing a cloud computing platform for solving problems much more complex than just processing requests to a web server, that is why we have decided to rely on their hardware infrastructure and domain knowledge to run some test applications and see if cloud computing can be of real help in the field of risk analysis.

EasyFitXL – the distribution fitting add-in for Excel – was first introduced with the release of EasyFit 4.0 back in 2007. When designing EasyFitXL, we did a lot of research as to which Excel versions to support. At that time, the latest version of Excel was Excel 2007, which included some new useful features, such as the support for larger worksheets and multi-threaded worksheet recalculation capability. However, many customers were not rushing to upgrade to Excel 2007 because of it’s controversial Ribbon Interface, so we had to make EasyFitXL compatible with the previous version – Excel 2003.

According to some publicly available data, Excel 2002 and Excel 2000 still had a considerable user base, so we have made a decision to support these two older versions as well. As a result, EasyFitXL initially included support for Excel versions from 2000 through 2007, covering perhaps over 99% of all Excel installations in the world.

Last month Microsoft has released Excel 2010 which does not make a big difference in terms of data analysis, however, with its release we started receiving compatibility complaints from our customers, so we performed an in-depth testing and released an updated version of EasyFit (available for download).

Recently a customer has contacted us and noted that the Inverse Cumulative Distribution Function (the Quantile Function) of the Inverse Gaussian distribution implemented in EasyFit works well for lambda=1902.1, mu=41857.0 and P=0.9, but fails for the same lambda & mu and P=0.99. Last week we have released an updated version of EasyFit that fixes the problem, and in this post we would like to elaborate more on the issue.

Evaluating the Inverse CDF of the Inverse Gaussian Model
Since the CDF of the Inverse Gaussian distribution is quite complicated (expressed in terms of the two Laplace Integrals), the Inverse CDF of this model is not available in closed form, and cannot be easily evaluated for a given set of distribution parameters. Initially, we have implemented an iterative approximation algorithm that evaluates the ICDF(P) using the CDF as well as the PDF to speed up the calculation. The algorithm itself works very well over a great range of input parameters, however, we have placed a limitation on how many iterations it is allowed to perform.

Because EasyFit is considered an interactive data analysis tool, we are always looking for a balance between the feature set and the performance, which is especially important when using EasyFit with Excel worksheets calculated in real time. The limitation on the number of iterations is necessary to make sure the algorithm doesn’t fall into an “infinite loop”, meaning the situation when it’s unable to reach the specified accuracy regardless of how long it continues to work. The problem usually happens when we are hitting the precision limitations of the computer’s CPU: in theory, the algorithm must converge in a limited number of steps, but in reality, it will just continue iterating over and over again without any accuracy improvements.

As a solution, we have made some improvements to the algorithm, making it more robust and efficient, so it now works with the same accuracy, but for a larger range of input parameters. For example, considering the parameters that initially caused the problem (lambda=1902.1 and mu=41857.0), the ICDF(P) can be evaluated for values of P up to 0.999925, which is more than enough for most statistical analysis applications.

Should You Upgrade?
Since this minor issue does not affect the accuracy of distribution fitting, you only need to upgrade if you are experiencing problems evaluating the Inverse CDF of the Inverse Gaussian distribution for P>0.9, otherwise EasyFit 5.2 will still work well for you.

The new Software Development Kit enabling you to easily add Monte Carlo simulation and distribution analysis features to your applications is now available for public beta testing – please download the free beta version and take a look at the code examples (available in several languages, including C#, VB.NET, C++, and Visual Basic for Applications).

We would be glad to receive any feedback, questions or suggestions from you, so please feel free to drop us a line and let us know what you think about this product and how we can make it better.

Earlier in December 2008, we announced our plans to create the Software Development Kit that would allow software developers to integrate the simulation and probabilistic analysis functionality of EasyFit into their custom software solutions. Since then, we have been working hard to bring the SDK to market. Despite some delays, we are glad to announce that we have successfully completed the development, and this month we plan to release the beta version of the product.

Become a Beta Tester Today!
Taking part in our beta testing program would not just grant you access to the cutting edge software technology – the most active beta testers will also be able to purchase the SDK at an introductory price before the production version of the SDK is released. To sign up, please contact us with your specific requirements (let us know how you plan to use the SDK and what development platform (C#, VB.NET etc.) you are using.)

We have just released EasyFit Version 5.1 – the update that fixes a bug causing an incorrect calculation of the chi-squared GOF statistic for small sample sizes. To upgrade, uninstall EasyFit 5.0 from your computer, then download and install the latest version.

Since fitting probability distributions to large data sets can be a time-consuming task, we are currently researching the possibility of using multi-core processors to make EasyFit work faster. During the past several years, major processor manufacturers have been promoting the multi-core technology on the desktop processors market. Multiple cores in a single chip allow for better performance/price ratio on a range of tasks, however, existing software needs to be updated accordingly to take full advantage of this type of hardware.

We have modified the original distribution fitting algorithm to utilize all cores available on a system, and used it to fit distributions to a simulated set of 200,000 data points. In a series of tests on an Intel dual-core processor, the new algorithm executed almost twice as fast, yielding up to 90% performance increase, compared to the version currently used in EasyFit. These are very good results, and we will definitely be including this feature into the next release of EasyFit.

On a related note, last week we were contacted by a customer regarding our upcoming Simulation & Probabilistic Analysis SDK. They need to analyze large volumes of data, and from their description of the problem we estimated that the typical analysis would take up to 20 hours on a modern PC. With the new distribution fitting algorithm, it can take less than 12 hours on a dual-core CPU, or even less on quad-core processors popular in the server space. In a decision making environment where several hours can mean the difference between profit and loss, this is a very important improvement.

Lately we have received a couple messages from customers asking if it’s possible to use the Monte Carlo simulation and distribution fitting features of EasyFit in their own software applications. The short answer is yes, but these features are limited to calculating some distribution functions in Excel VBA. There’s currently no way to run simulations, fit distributions to data, perform goodness of fit tests, or use distribution functions from C#, C++, VB.NET, and other programming languages.

To fill the gap, we are considering to create a Simulation & Probabilistic Analysis Software Development Kit (SPA SDK) for software developers who need to deal with risk & uncertainty in their applications, but don’t have time or expertise to design and implement the required features on their own. We already have in place the tried and true technology that’s a basis for our distribution fitting products EasyFit and EasyFitXL, so creating an SDK would be possible in a short period of time.

Since we have had only a few requests for an SDK, we would like to know whether you would be interested in such kind of product. Below is our vision for the SDK – you are welcome to express any thoughts or specific requirements you might have. Please feel free to contact us and we will take your input seriously.

Update: The free beta version of the SDK is now available for download – please click here for details.

What is a Simulation & Probabilistic Analysis Software Development Kit (SPA SDK) ? (more…)

EasyFit ships with a comprehensive help file providing detailed information on all aspects of fitting distributions to data and interpreting the analysis results. For instance, it includes the description of supported distributions, goodness of fit tests, and output graphs.

If you are considering to try EasyFit but not sure if it has a particular feature you need, you can refer to the EasyFit help online which we have made available on our website for your convenience. Of course, you are still welcome to contact us for any questions regarding EasyFit or fitting distributions in general.