Real Time monitoring Windows Event Logs using Reactive Extensions

In this post, I want to demonstrate how I have used Reactive Extensions to monitor Windows Event Log entries as they get added to Event Logs by a specific provider.  The main piece that enables us to do this is Tx.All a single nuget package that exposes event log events as observables using Reactive Extensions. I will create two console projects, one for monitoring event logs and another to generate Event Logs.   

Part 1 – Monitoring Console App

Create a blank .NET console project (do not select .NET core).  Then Install Tx.All Nuget Package and your packages.config file should look as shown below.

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="System.Reactive" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Core" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Interfaces" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Linq" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.PlatformServices" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Windows.Threading" version="3.1.1" targetFramework="net47" />
  <package id="Tx.All" version="2.1.1" targetFramework="net47" />
  <package id="Tx.Core" version="2.1.1" targetFramework="net47" />
  <package id="Tx.SqlServer" version="2.1.1" targetFramework="net47" />
  <package id="Tx.Windows" version="2.1.1" targetFramework="net47" />
  <package id="Tx.Windows.TypeGeneration" version="2.1.1" targetFramework="net47" />
</packages>

Then use the following code to query specific provider inside Event Log. We are interested into Application event logs and our provider is ConsoleEventLogGenerator.

using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.Linq;
using System.Reactive.Linq;
using System.Text;
using System.Threading.Tasks;
using Tx.Windows;

namespace ConsoleMonitor
{
    class Program
    {
        static void Main(string[] args)
        {
            IObservable<EventRecord> evtx = EvtxObservable.FromLog("Application");
            IDisposable d = evtx.Where(y => y.ProviderName == "ConsoleEventLogGenerator").Subscribe(e => {
                Console.WriteLine(e.FormatDescription());
            });

            Console.WriteLine("Waiting for Event Log entry for provider - \"ConsoleEventLogGenerator\"");
            Console.Read();
        }
    }
}

Now we will create another console project to create event in the Event Log. 

Part 2 – Create another Console Project.

In the following code we are creating lots of events log entries and our source is ConsoleEventLogGenerator. This string has to match the string shown in the earlier section.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleEventLogGenerator
{
    class Program
    {
        static void Main(string[] args)
        {
            EventLog log = new EventLog("Application");
            log.Source = "ConsoleEventLogGenerator";

            for (int i = 0; i < 100; i++)
            {
                Thread.Sleep(1000);
                log.WriteEntry("test " + i);
            }
        }
    }
}

In the next section you will fire up both the application side by side and you will see events showing up in real time.

I have created a simple console app like this to monitor event log when I am debugging a certain application that writes everything to event log.  Seeing output in real time helps a lot in debugging.

Why I don’t see any data disk attached inside my Azure Virtual Machine?

In Azure, you can attach multiple data disks to your virtual machine based upon the type of virtual machine size you select. These can be standard disks or premium disks.  If you attach a data disk then they won’t magically show up in the file explorer. You will have to initialize a new data disk and create a volume from the server manager if you are on windows server operating systems. From Azure I have added 500GB data disk. In the below screenshot the highlighted disk is the one we want to initialize.

If you open disk management, then it will ask you to initialize the disk.

Click Ok.

You can now just right click disk that you created and create a New Simple Volume.

Now you know the game right, just keep clicking next until you want to change a setting and you are done.

After you follow the prompts you can open file explorer and see that a new drive is available.

You can do the same thing without opening Disk Management. This can be done from Server Manager window itself the wizard is bit nicer looking.  Have fun with your data disks. 

Iometer is how you measure IOPS

Every time I saw IOPS in Azure I always thought that what the heck is this. I understood the concept but never gave a hard look at it. The developer in me just glanced over it and just selected a VM size that felt good enough.

 

The issue becomes when you have to evaluate virtual machine size that is ideal for your workload so you don’t blow up your budget by selecting larger VMs or have performance issues with applications by selecting smaller VMs.

So one of the parameters you will have to consider is IOPS, i.e. I/O per sec for disk and network.  While you can monitor for performance counters it is not the most intuitive way. Iometer is a free tool which you can use to measure these metrics and do some benchmarking.  If you have existing VMs you can use this tool and figure out which VM size in Azure you might need.

In the below screen shot I am trying to measure performance of my regular drive.

 

And in the next test I am measure performance of my SSD drive.  You can see the difference between two drives. SSDs are faster.

Please share you tips and tricks or article regarding measuring and benchmarking for IOPs performance.

Azure Networking Tip for Developer - Subnet Calculator

Creating a virtual network inside of Azure, you will be required to come up with your subnet and address ranges you want to have in that subnet. I cannot easily come up with all the address range I want to have in a subnet and so I came across subnet calculator.  For networking people this stuff is very basic. They can just spit out IP address like popcorn popping out. But I can’t and hence I use subnet calculator. Play around and check for yourself if you are new to this.

Setup Integrated Terminal in VS Code to use Anaconda Python command prompt

I recently discovered about Anaconda for doing Python development.  Anaconda simplifies setting up data science development environment. It installs all the packages for Python, installs Python, installs R etc. It just works. I have tried developing on Python before and it was rough.  Anaconda is great.

I like VS Code as an editor and I want to do Python development using it.  VS Code has Integrated Terminal into it for launching Python scripts. You can have PowerShell or cmd.exe configured. I want to replace it with Anaconda command prompt which is nothing but cmd.exe with some arguments.

Inside VS Code, open user settings page and navigate to Integrated Terminal section

Click on Replace in Settings and User Settings window will open up. There you will add cmd.exe and arguments settings for Anaconda command prompt.

"terminal.integrated.shell.windows": "C:\\Windows\\System32\\cmd.exe"

Regarding arguments. Well how do we find the arguments for Anaconda command prompt. Right click on the command prompt for Anaconda icon and then click properties.

As shown in the screen shot above everything after cmd.exe is arguments. Take those and put it inside [] as shown below.

"terminal.integrated.shellArgs.windows": [

"/K","C:\\Users\\myusername\\AppData\\Local\\Continuum\\Anaconda3\\Scripts\\activate.bat C:\\Users\\myusername\\AppData\\Local\\Continuum\\Anaconda3"

],

VS Code also allows you to do debugging with Python, provided you install Python and Python for VS Code extensions. After you install you can start debugging your Python script. But hold on, if your python path doesn’t exist inside environment variable then VS Code will error out asking you to modify launch.json file and modify the pythonPath. Inside your user settings file. Add the following line regarding your Anaconda3 location where python.exe is located as shown below.

"python.pythonPath": "C:\\Users\\myusername\\AppData\\Local\\Continuum\\Anaconda3"

After doing this you will be able to do debugging with VS Code with Python using Anaconda. If you are getting started with Python then I highly recommend using Anaconda for your environment setup. Enjoy!