In our eCommerce business we have around 55 servers, its a good mix of Linux Centos/Red Hat and Windows servers spread out between two data centres. Like a lot of companies we started with a Linux based environment and over the years Windows servers moved in, and with windows based web systems, IIS becomes the default web server of choice (default choice I should note). Recently one of our key production IIS web servers started to lockup when the Linux server was proxy passing requests to it at peak times during the day, and it doesn’t happen everyday, we could run for several weeks and then suddenly the IIS6 worker process assigned to a particular App Pool stops responding, its memory consumption goes through the roof and I assume its resource usage in other key areas sky rockets as well.

Finding the root cause

Often restarting key App Pools in IIS has no effect and usually a complete restart of IIS is required, sometime twice, so clearly something is not right. This led me to decide to track the IIS environment in Nagios our monitoring tool of choice so we could get a handle on how the issue arrises and possibly find the root cause of the problem. We currently have around 500 Nagios processes being monitored with around 4 Nagios servers and we are now moving to Icinga as the user interface is much nicer. With Nagios we already graph a number of key items and I have even written some custom graphing that allows us to see the flow on effects of one system to another, so adding key indicators from IIS appeared to be a good idea to see the load of individual components in IIS. After some investigation I decided that I would write an NRPE plugin in C# that could be called to return key metric from the IIS processes. This has led me to fully investigate how IIS works and how much info is missing when you search the web for working IIS coding examples and related App Pool information.

Defining the NRPE Plugin

An NRPE plug is not only capable of returning data to Nagios, but its a great way for a custom written program to interogate a machine and build graphs and operating models outside of Nagios. NRPE can be called via cron using the check_nrpe program. Using the returned data you can pump it into anything, in this case I was going to build a graph and design a business processs flow to show the request response cycle of our production systems.

Initially I wanted thread and memory usage and just graph them, but then I realised that there was a whole lot more to IIS, so before I could define the plugin’s specs I needed to understand how components in IIS work and relate to each other so that I could make the NRPE plugin more versatile, at a minimum it needed to be able to perform the following tasks:

  • Return the IIS general status
  • Return an enumeration of hosted web sites
  • Return an enumeration of Application Pools
  • Return the relationship between a web site and the App Pool
  • Return metrics on a particular Application Pool

Code Samples

Below are some code samples for various snippets of what I found on the web or worked out through experimentation. in the C# IDE you will need to add references to some assemblies. I added the following assemblies during the course of development of the code:

  1. System.DirectoryServices
  2. System.Management.Instrumentation
  3. System.ServiceProcess
  4. System.Web

IIS Version

Prints out the version of IIS, useful to determine what IIS version is running and hence what can be called and not called.

public Version GetIisVersion()
        {
            using (RegistryKey componentsKey =
            Registry.LocalMachine.OpenSubKey(@"Software\Microsoft\InetStp", false))
            {
                if (componentsKey != null)
                {
                    int majorVersion = (int)componentsKey.GetValue("MajorVersion", -1);
                    int minorVersion = (int)componentsKey.GetValue("MinorVersion", -1);
                    if (majorVersion != -1 && minorVersion != -1)
                    {
                        return new Version(majorVersion, minorVersion);
                    }
                }
                return new Version(0, 0);
            }
        }

IIS Status

 public bool CheckIISRunning()
        {
            ServiceController controller = new ServiceController("W3SVC");
            return controller.Status == ServiceControllerStatus.Running;
        }

Enumerate the hosted web sites.
This code sample enumerates the web sites and also lists the App Pool the web site runs in, only after much experimentation did I find this in one of the App Pool properties (AppPoolId).

using System.DirectoryServices;

     public void EnumerateWebsites()
        {
            try
            {
                DirectoryEntry w3svc = new DirectoryEntry("IIS://localhost/w3svc");
                foreach (DirectoryEntry de in w3svc.Children)
                {
                    if (de.SchemaClassName == "IIsWebServer")
                    {
                        Console.Write("Web Site= ["+de.Properties["ServerComment"][0].ToString()+"]  ");
                        DirectoryEntry dir = new DirectoryEntry("IIS://localhost/w3svc/" + de.Name.ToString() + "/root");
                        Console.WriteLine("App Pool="+dir.Properties["AppPoolId"][0].ToString());
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error in EnumerateWebsites: " + ex.Message);
            }
        }

Enumerate Application Pools
Lists all application pools.

public void EnumerateAppPools()
        {
            DirectoryEntry W3SVC = new DirectoryEntry("IIS://localhost/w3svc", "", "");
            foreach (DirectoryEntry Site in W3SVC.Children)
            {
                if (Site.SchemaClassName == "IIsApplicationPools")
                {
                    foreach (DirectoryEntry child in Site.Children)
                    {
                        Console.WriteLine(child.Name+" | "+child.Path);
//                        Console.WriteLine("Parent=" + child.Parent.Name); // parent is "AppPools"
                    }
                }
            }
        }
{

Dump Web Site Data
Code to dump out everything about a web site entry.

public void DumpWebsiteDetails()
        {
            Console.WriteLine("Dump()");
            try
            {
                DirectoryEntry w3svc = new DirectoryEntry("IIS://localhost/w3svc");
                foreach (DirectoryEntry de in w3svc.Children)
                {
                    if (de.SchemaClassName == "IIsWebServer")
                    {
                        Console.WriteLine(de.Properties["ServerComment"][0].ToString());
                        Console.WriteLine("Dump :" + de.Name.ToString());

                        DirectoryEntry dir = new DirectoryEntry("IIS://localhost/w3svc/" + de.Name.ToString() + "/root");
                        Console.WriteLine("There are " + dir.Properties.Count + " properties available");
                        
                        foreach (string elmentName in dir.Properties.PropertyNames)
                        {
                            PropertyValueCollection pvc = dir.Properties[elmentName];
                            if (pvc.Count > 1)
                            {
                                Console.WriteLine("Dumping props for " + elmentName);
                                Console.WriteLine("=============================================");
                            }
                            for (int i = 0; i < pvc.Count; i++)
                            {
                                Console.WriteLine(i.ToString()+".   Element name="+elmentName+"="+pvc[i].ToString());
                            }
                        } 

                    }
                    else
                    {
                        Console.WriteLine("Schema="+de.SchemaClassName);
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error in Dump: " + ex.Message);
            }
        }

Dump AppPool Details
I used this to get everything I could about an App Pool.

        public void AppPoolDetails()
        {
            DirectoryEntry W3SVC = new DirectoryEntry("IIS://localhost/w3svc","","");
            foreach (DirectoryEntry Site in W3SVC.Children)
            {
                Console.Write(Site.SchemaClassName+"->");
                if (Site.SchemaClassName == "IIsApplicationPools")
                {
                    foreach (DirectoryEntry child in Site.Children)
                    {
                        Console.WriteLine("    AppPool: " + child.Name + " Path=" + child.Path);
                        Console.WriteLine("Parent="+child.Parent);

                        PropertyCollection pc = child.Properties;
                        IDictionaryEnumerator ide = pc.GetEnumerator();
                        ide.Reset();
                        while (ide.MoveNext())
                        {
                            PropertyValueCollection pv = ide.Entry.Value as PropertyValueCollection;
                            Console.WriteLine(" --- " + ide.Entry.Key.ToString() + "=" + pv.Value.ToString());
                        }
                    }
                }
            }
        }

Recycle AppPool
Untested code to Recycle an App Pool.

public bool RecycleAppPool(string serverName, string appPoolName)
        {
            DirectoryEntry appPools = new DirectoryEntry("IIS://" + serverName + "/w3svc/apppools");
            bool status = false;
            foreach (DirectoryEntry AppPool in appPools.Children)
            {
                if (appPoolName.Equals(AppPool.Name, StringComparison.OrdinalIgnoreCase))
                {
                    AppPool.Invoke("Recycle", null);
                    status = true;
                    break;
                }
            }
            appPools = null;
            return status;
        }

At time of writing I am still working on the thread App Pool relationship.

Advertisements