Wednesday, December 6, 2017

Real Time monitoring Windows Event Logs using Reactive Extensions

In this post, I want to demonstrate how I have used Reactive Extensions to monitor Windows Event Log entries as they get added to Event Logs by a specific provider.  The main piece that enables us to do this is Tx.All a single nuget package that exposes event log events as observables using Reactive Extensions. I will create two console projects, one for monitoring event logs and another to generate Event Logs.   

Part 1 – Monitoring Console App

Create a blank .NET console project (do not select .NET core).  Then Install Tx.All Nuget Package and your packages.config file should look as shown below.


<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="System.Reactive" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Core" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Interfaces" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Linq" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.PlatformServices" version="3.1.1" targetFramework="net47" />
  <package id="System.Reactive.Windows.Threading" version="3.1.1" targetFramework="net47" />
  <package id="Tx.All" version="2.1.1" targetFramework="net47" />
  <package id="Tx.Core" version="2.1.1" targetFramework="net47" />
  <package id="Tx.SqlServer" version="2.1.1" targetFramework="net47" />
  <package id="Tx.Windows" version="2.1.1" targetFramework="net47" />
  <package id="Tx.Windows.TypeGeneration" version="2.1.1" targetFramework="net47" />
</packages>

Then use the following code to query specific provider inside Event Log. We are interested into Application event logs and our provider is ConsoleEventLogGenerator.

using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.Linq;
using System.Reactive.Linq;
using System.Text;
using System.Threading.Tasks;
using Tx.Windows;

namespace ConsoleMonitor
{
    class Program
    {
        static void Main(string[] args)
        {
            IObservable<EventRecord> evtx = EvtxObservable.FromLog("Application");
            IDisposable d = evtx.Where(y => y.ProviderName == "ConsoleEventLogGenerator").Subscribe(e => {
                Console.WriteLine(e.FormatDescription());
            });

            Console.WriteLine("Waiting for Event Log entry for provider - \"ConsoleEventLogGenerator\"");
            Console.Read();
        }
    }
}

Now we will create another console project to create event in the Event Log. 

Part 2 – Create another Console Project.

In the following code we are creating lots of events log entries and our source is ConsoleEventLogGenerator. This string has to match the string shown in the earlier section.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleEventLogGenerator
{
    class Program
    {
        static void Main(string[] args)
        {
            EventLog log = new EventLog("Application");
            log.Source = "ConsoleEventLogGenerator";

            for (int i = 0; i < 100; i++)
            {
                Thread.Sleep(1000);
                log.WriteEntry("test " + i);
            }
        }
    }
}

In the next section you will fire up both the application side by side and you will see events showing up in real time.

image

I have created a simple console app like this to monitor event log when I am debugging a certain application that writes everything to event log.  Seeing output in real time helps a lot in debugging.

Sunday, December 3, 2017

Why I don’t see any data disk attached inside my Azure Virtual Machine?

In Azure, you can attach multiple data disks to your virtual machine based upon the type of virtual machine size you select. These can be standard disks or premium disks.  If you attach a data disk then they won’t magically show up in the file explorer. You will have to initialize a new data disk and create a volume from the server manager if you are on windows server operating systems. From Azure I have added 500GB data disk. In the below screenshot the highlighted disk is the one we want to initialize.

image

If you open disk management, then it will ask you to initialize the disk.

image

Click Ok.

You can now just right click disk that you created and create a New Simple Volume.

image

Now you know the game right, just keep clicking next until you want to change a setting and you are done.

image

image

image

image

image

After you follow the prompts you can open file explorer and see that a new drive is available.

image

You can do the same thing without opening Disk Management. This can be done from Server Manager window itself the wizard is bit nicer looking.  Have fun with your data disks. 

Wednesday, November 29, 2017

Iometer is how you measure IOPS

Every time I saw IOPS in Azure I always thought that what the heck is this. I understood the concept but never gave a hard look at it. The developer in me just glanced over it and just selected a VM size that felt good enough.

image 

The issue becomes when you have to evaluate virtual machine size that is ideal for your workload so you don’t blow up your budget by selecting larger VMs or have performance issues with applications by selecting smaller VMs.

So one of the parameters you will have to consider is IOPS, i.e. I/O per sec for disk and network.  While you can monitor for performance counters it is not the most intuitive way. Iometer is a free tool which you can use to measure these metrics and do some benchmarking.  If you have existing VMs you can use this tool and figure out which VM size in Azure you might need.

In the below screen shot I am trying to measure performance of my regular drive.

clip_image001 

And in the next test I am measure performance of my SSD drive.  You can see the difference between two drives. SSDs are faster.

clip_image001[4]

Please share you tips and tricks or article regarding measuring and benchmarking for IOPs performance.

Tuesday, November 28, 2017

Azure Networking Tip for Developer - Subnet Calculator

Creating a virtual network inside of Azure, you will be required to come up with your subnet and address ranges you want to have in that subnet. I cannot easily come up with all the address range I want to have in a subnet and so I came across subnet calculator.  For networking people this stuff is very basic. They can just spit out IP address like popcorn popping out. But I can’t and hence I use subnet calculator. Play around and check for yourself if you are new to this.

image

Thursday, August 24, 2017

Setup Integrated Terminal in VS Code to use Anaconda Python command prompt

I recently discovered about Anaconda for doing Python development.  Anaconda simplifies setting up data science development environment. It installs all the packages for Python, installs Python, installs R etc. It just works. I have tried developing on Python before and it was rough.  Anaconda is great.

I like VS Code as an editor and I want to do Python development using it.  VS Code has Integrated Terminal into it for launching Python scripts. You can have PowerShell or cmd.exe configured. I want to replace it with Anaconda command prompt which is nothing but cmd.exe with some arguments.

Inside VS Code, open user settings page and navigate to Integrated Terminal section

image

image

Click on Replace in Settings and User Settings window will open up. There you will add cmd.exe and arguments settings for Anaconda command prompt.

"terminal.integrated.shell.windows": "C:\\Windows\\System32\\cmd.exe"

Regarding arguments. Well how do we find the arguments for Anaconda command prompt. Right click on the command prompt for Anaconda icon and then click properties.

image

image

As shown in the screen shot above everything after cmd.exe is arguments. Take those and put it inside [] as shown below.

"terminal.integrated.shellArgs.windows": [

"/K","C:\\Users\\myusername\\AppData\\Local\\Continuum\\Anaconda3\\Scripts\\activate.bat C:\\Users\\myusername\\AppData\\Local\\Continuum\\Anaconda3"

],

VS Code also allows you to do debugging with Python, provided you install Python and Python for VS Code extensions. After you install you can start debugging your Python script. But hold on, if your python path doesn’t exist inside environment variable then VS Code will error out asking you to modify launch.json file and modify the pythonPath. Inside your user settings file. Add the following line regarding your Anaconda3 location where python.exe is located as shown below.

"python.pythonPath": "C:\\Users\\myusername\\AppData\\Local\\Continuum\\Anaconda3"

After doing this you will be able to do debugging with VS Code with Python using Anaconda. If you are getting started with Python then I highly recommend using Anaconda for your environment setup. Enjoy!

image

Thursday, December 15, 2016

Easily add authorization rules to your Azure App

In the last post we saw how you can configure Authentication for your Azure App using Google.  You can configure Authentication using other providers too like Facebook, Twitter or Microsoft Azure Active Directory.  And the good thing is that you can do this without modifying your deployed application.

In this post I want to show you a new Azure App service feature called Authorization which you can easily configure by just adding authorization.json file to your application root directory.  Let’s take a look at this authorization.json file.  For my simple example I have created as follows:

{
  "routes": [
    {
      "path_prefix": "/",
      "policies": { "unauthenticated_action": "AllowAnonymous" }
    },
    {
      "path_prefix": "/Admin",
      "policies": { "unauthenticated_action": "RedirectToLoginPage" }
    }
  ]
}

First you have a routes property which a collection of url authorization rules.  There are different properties for each rule.  In the above example, the first rule has path_prefix property set as ‘/’ and policy set as “unauthenticated_action” to “AllowAnoymous”. This means when I visit the home page allow anonymous access.  But here is a caveat, this is only going to work if you have “Action to take when request is not authenticated” option set in the azure portal to “Allow Anonymous requests (no action)”.  It was a bit confusing to me because I had this option set to “Login with Google” and it would not allow anonymous access with .json file.  Check out my screenshot of my settings.

image

The next authorization rule in the above json says that when I navigate to /Admin page then redirect me to Login page and in my example it will be navigated to Google’s login page.

The basic schema of this .json file is as follows as shown in this msdn article.

{
  "routes": [
    {
      "http_methods": [ "GET", "POST", "PUT", ... ],
      "path_prefix": "/some/url/prefix",
      "policies": {
        "unauthenticated_action": "AllowAnonymous|RedirectToLoginPage|RejectWith401|RejectWith404"
      }
    },
    ...
  ]
}

You might be interested in knowing what does “http_methods” property does. It says that if a request is “GET” or “PUT” or “POST” then only this rule will be taken into effect for this url.  If you want to restrict access to a particular page only if it is accessed via a “GET” request then you can just specifiy “GET”.

In path prefix you can have wildcards too which is very important if the url can have variable segments. For example, /product/1/edit then you can have path_prefix as /product/*/edit.  For simple web applications this feature works great and I like how turn key solution this is.

Once again for this feature to work the authorization.json file has to be at the root of the application folder.

Tuesday, November 29, 2016

Troubleshooting TF14098 permission denied while deleting a branch

Recently I encountered the error TF14098: Access Denied: User DOMAIN\user needs Checkin permission(s) for $/teamproject/folder/* while delete a TFS branch.  In this post, I want to share experience troubleshooting this issue.  I am using TFS 2015 that is hosted on permises with latest update. I am also a TFS admin with project collection administrator rights. 

When I first saw this error, I was felt that I knew why this happened.  We had two branches DEV and TEST. TEST was branched from DEV.  DEV had folder called Binaries and in it there was a particular .dll which had explicit DENY checkin permissions set for [Project]\Contributors, [Project]Project Administrators, [Collection]\Project Collection Administrators. I thought that since TEST was branched from DEV all those permissions would have been in effect in TEST.  I right clicked on the TEST/Binaries/our.dll from within TFS source control explorer, then click on Advanced and then on Security.

image

After that I made sure that those three permission were set to Allow and nothing was deny.  I was confident that this would work but it did not worked. Again the same error. Access Denied.

Next I opened visual studio developer command prompt in administrator mode and then navigated to the actual workspace folder locally and then ran the command tf permission or tf vc permission as follows:

>tf vc permission $/project/test /recursive | clip

This will copy all the permissions on files in your workspace recursively and copy it to your clipboard which you then can paste inside notepad and do search on deny. I found the Groups that had Deny permission set for that particular file.  I double checked that file’s permission inside TFS source control explorer thinking something might have gone wrong.  I tried deleting the branch again. Same error. Then I deleted just that file to get rid of the permission error.  This time I was super confident that this would fix the issue since there is no file locally or on the server.  I tried again. Same error.  Then I read that in this msdn forum that permissions can even exists on a path even if no file exists on the server.  Which makes sense if you think about it that on the server path could still exists even if there are no file in a branch since a branch only has pointers to the contents in it.  I used the following command to set allow on that file as follows:

>tf vc permission $/project/test/binaries/our.dll /allow:* /group:”[Project]\Contributors”,”[Project]\Project Administrators”

I tried deleting the branch again inside visual studio and then tried checking in again and finally it worked this time.  I hope this helps.