Azure DevOps – what’s new?

Today Microsoft announced the relaunch of Visual Studio Team Services (VSTS). VSTS is now renamed to Azure DevOps.

The reason for the name change is that they want to decouple the Suite from the Visual Studio brand and the perception that it is .NET-only whereas Azure is associated with cross-platform – Cloud for all. But of course, Azure DevOps will work for whatever type of cloud you are using.

What’s new?

  • The service URLs moved from

    https://YOURORG.visualstudio.com to https://dev.azure.com/YOURORG

    There are long-term redirects for existing customers in place.

  • The offered services are renamed:
    • Work 👉 Azure Boards
    • Nav-Code Code 👉 Azure Repos
    • Nav-Launch Build and release 👉 Azure Pipelines
    • Nav-Feed Packages 👉 Azure Artifacts
    • Nav-TestTest 👉 Azure Test Plans

  • Azure DevOps GitHub Integration:
    Azure Pipelines is now available in the GitHub marketplace.
  • Azure Pipelines for Open Source projects
    Azure Pipelines includes 10 free concurrent pipelines with unlimited build minutes and users (for Mac, Windows and Linux).

 

Azure App Services: Determine supported dotnet core version

If you try to use the latest .NET Core version 2.1.3 within your Azure Web or API App, you will receive the error code 502.5.

After you enabled logging you will find an error similar to this:

It was not possible to find any compatible framework version
The specified framework 'Microsoft.AspNetCore.App', version '2.1.3' was not found.
  - Check application dependencies and target a framework version installed at:
      D:\Program Files (x86)\dotnet\
  - Installing .NET Core prerequisites might help resolve this problem:
      http://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409
  - The .NET Core framework and SDK can be installed from:
      https://aka.ms/dotnet-download
  - The following versions are installed:
      2.1.0-rc1-final at [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
      2.1.2 at [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

So one way to determine the installed dotnet core versions is to look at the error log. But you can also execute the following command within the Debug Console on the Kudu Engine (Advanced Tools in the Azure Portal):

dotnet --list-sdks

This will displays the installed .NET Core runtimes:

dotnet-list-sdks

Automatically pick the latest dotnet core version

You can simply avoid running into this error by omitting the Version attribute on the Microsoft.ApsNetCore.App PackageReference within your Project file (*.csproj):

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="wwwroot\" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App"/>
  </ItemGroup>
</Project>

Now, as soon as the Azure App Services gets the new 2.1.x dotnet core sdk, your app will automatically start using it.

File upload through ASP.NET Core middleware

In my previous article we discussed the different options to implement file upload for cloud applications.

In this article I want to provide you an example of how to implement a file upload through a middleware.:

middleware_upload

Example: File Upload to Azure Blog Storage using Angular and ASP.NET Core

We will scaffold our application using the Angular template which is part of the .NET-Core CLI and create a component using the Angular CLI.

Prerequisites

* You can either manually create the Storage Account within the Azure Portal or by using the following ARM Template:

Scaffold the project

To scaffold the project we use the dotnet new command:

dotnet new angular --name file-upload

Implement the middleware

To store the uploaded files to Azure Blob Storage we need to specify the connection string within the appsettings.json:

"ConnectionStrings": {
    "StorageAccount": "UseDevelopmentStorage=true"
    },

If you don’t have the Azure Storage Emulator installed you have to replace the value with an actual Azure Storage Account connection string.

Install WindowsAzure.Storage NuGet Package

You can install the NuGet package using the following command:

dotnet add package WindowsAzure.Storage

File upload implementation

We will implement the file upload in a new Controller called AssetController. The controller only exposes a single method called UploadAssetAsync which takes an IFormFile with the name asset (note: IFormFile is suitable for uploading small files, if you have to deal with large files you have to consider implementing streaming or uploading the files directly from the client to a data store. Further information here.).

The UploadAssetAsync method uploads the passed file to the previous specified Azure Blob Storage and returns the URI of the new blob:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.WindowsAzure.Storage;

namespace file_upload.Controllers
{
    [Route("api/[controller]")]
    public class AssetController : Controller
    {
        private readonly IConfiguration _configuration;
        public AssetController(IConfiguration config)
        {
            _configuration = config;
        }

        [HttpPost]
        public async Task UploadAssetAsync([FromForm]IFormFile asset)
        {
            CloudStorageAccount storageAccount = null;
            if (CloudStorageAccount.TryParse(_configuration.GetConnectionString("StorageAccount"), out storageAccount))
            {
                var client = storageAccount.CreateCloudBlobClient();
                var container = client.GetContainerReference("fileupload");
                await container.CreateIfNotExistsAsync();

                var blob = await container.GetBlobReferenceFromServerAsync(asset.FileName);
                await blob.UploadFromStreamAsync(asset.OpenReadStream());

                return Ok(blob.Uri);
            }

            return StatusCode(StatusCodes.Status500InternalServerError);
        }
    }
}

Disable HttpsRedirection in Development

Before we can run the application we should disable the Https redirection for the development environment (note: you can also install a localhost certificate). Otherwise we will get a warning that the site is not secure. This can be done by replacing the following line in the Configure method within the Startup.cs

app.UseHttpsRedirection();

with:

if (!env.IsDevelopment())
{
   app.UseHttpsRedirection();
}

Implement the frontend

We will implement the file upload in a new component. To create the new component we use the Angular CLI:

ng g component fileUpload

Note: Ensure you invoke the CLI commands within the ClientApp directory.

Configure Routing

Now we add routing to our new component within the app.module.ts:

{ path: 'file-upload', component: FileUploadComponent },

Then we add a link to the component inside the nav-menu.component.html:

<li>
    <a>
        <span class='glyphicon glyphicon-cloud-upload'></span> File Upload
    </a>
</li>

Install PrimeNG

We use the PrimeNG NPM package to implement the file upload:

npm install primeng --save 

Add the file upload

The last thing we have to do is to add the file upload to the file-upload.component.html. The name attribute value must match with the IFormFile parameter name in the middleware (in our case “asset“). The url is /api/Asset which is the address of the UploadAssetAsync middleware web method:

 
<p-fileUpload #fubauto mode="basic" name="asset" url="/api/Asset" maxFileSize="1000000" auto="true"
        chooseLabel="Browse"></p-fileUpload>

That is it. We don’t need to implement any further upload mechanism. To start the application we can use the dotnet CLI:

dotnet run

The source code can be found in my GitHub repository.

File upload in Cloud Applications: The Options

Almost every web application requires some form of file upload.  You may want to allow a user to upload a profile picture or to import any kind of data.

Multiple ways to implement the file upload

Depending on the size of the files and the regularity of the upload you have two options to implement the upload:

Directly upload the file to a data store

The fastest and resource friendliest way is to directly upload the file from the client to a data store. This typically requires the client to have the security credentials for the data store:

direct_uploadBut giving security credentials to potential untrusted clients isn’t a realistic approach for most web applications. Instead you want to use a token that provides clients restricted access to a specific resource for a limited validity period. This pattern is known as Valet Key pattern.

Upload the file through a middleware

The second option is to upload the file to your middleware (API) which will handle the movement of the data to the data store.

middleware_upload

This approach prevents us from exposing any information about the underlying data store to the client. We could even change our Data Store (e. g. from Azure Blob Storage to Azure File Storage) without updating the client.

The downside is that it absorbs valuable resources from our middleware like compute, memory and bandwidth.

Find outdated Azure ARM QuickStart Templates on GitHub

In my previous post Determine latest API version for a resource provider I showed you how to retrieve the latest API version for a specific resource provider using the Get-AzureRmResourceProviderLatestApiVersion cmdlet. In this post I will use the cmdlet to find any outdated resource provider within an ARM template. Also, we will analyze the Azure quickstart templates on GitHub!

To analyze an ARM template we have to convert the JavaScript Object Notation (JSON) to a custom PowerShell object that has a property for each field in the JSON.
We will use the following lean template for testing purpose (download):

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "name": "myuniquestorageaccountname",
            "apiVersion": "2016-01-01",
            "location": "West Europe",
            "sku": {
                "name": "Standard_LRS"
            },
            "kind": "Storage"
        }
    ]
}

The first thing we have to do is to load the file using the Get-Content cmdlet and pipe the result to the ConvertFrom-Json cmdlet. To retrieve the resources we further pipe the result to the Select-Object cmdlet:


Get-Content leanArmTemplate.json |
        ConvertFrom-Json |
        Select-Object -ExpandProperty resources

This will give us all necessary information like the resource provider and the used API version:

type       : Microsoft.Storage/storageAccounts
name       : myuniquestorageaccountname
apiVersion : 2016-01-01
location   : West Europe
sku        : @{name=Standard_LRS}
kind       : Storage

Now finally to analyze the template, we have to iterate over each resource and retrieve the latest API version using the previous mentioned Get-AzureRmResourceProviderLatestApiVersion cmdlet. To get a handy output we create a new PsCustomObject containing the resource type, the used and latest API Version and a flag that specifies whether the used API version is the latest:

Get-Content leanArmTemplate.json |
        ConvertFrom-Json |
        Select-Object -ExpandProperty resources |
         ForEach-Object {
            $latestApiVersion = Get-AzureRmResourceProviderLatestApiVersion -Type $_.type
            [PsCustomObject]@{
                Type = $_.type
                UsedApiVersion = $_.apiVersion
                LatestVersion = $latestApiVersion
                Latest = $_.apiVersion -eq $latestApiVersion
            }
        }

This is the output:

Type                              UsedApiVersion LatestVersion Latest
----                              -------------- ------------- ------
Microsoft.Storage/storageAccounts 2016-01-01     2018-02-01     False

In this example, the used API for the resource provider Microsoft.Storage/storageAccounts is not up to date.

For the sake of convenience I wrapped the code into a Get-OutdatedResourceProvider cmdlet and added some special treatment to handle nested resources:

function Get-OutdatedResourceProvider
{
    [CmdletBinding()]
    [Alias()]
    [OutputType([string])]
    Param
    (
        [Parameter(Mandatory = $true, Position = 0)]
        [string]$Path,

        [switch]$IncludePreview
    )
    Get-Content $Path |
        ConvertFrom-Json |
        Select-Object -ExpandProperty resources |
        ForEach-Object {
        $latestApiVersion = Get-AzureRmResourceProviderLatestApiVersion -Type $_.type
        [PsCustomObject]@{
            Type = $_.type
            UsedApiVersion = $_.apiVersion
            LatestVersion = $latestApiVersion
            Latest = $_.apiVersion -eq $latestApiVersion
        }

        # a resource may include sub resources
        foreach ($subResource in $_.resources)
        {
            $latestApiVersion = Get-AzureRmResourceProviderLatestApiVersion -Type ('{0}/{1}' -f $_.type, $subResource.type)
            [PsCustomObject]@{
                Type = '{0}/{1}' -f $_.type, $subResource.type
                UsedApiVersion = $_.apiVersion
                LatestVersion = $_.apiVersion -eq $latestApiVersion
            }
        }
    }
}

🕵️‍♂️🕵️‍♂️🕵️‍♂️ Now it’s time to analyze some templates  🕵️‍♂️🕵️‍♂️🕵️‍♂️.

Lets clone the whole azure-quickstart-templates Git repository:

git clone https://github.com/Azure/azure-quickstart-templates.git

Then we can filter all ARM templates using the Get-ChildItem cmdlet:

$armTemplates = Get-ChildItem 'azure-quickstart-templates' -Filter 'azuredeploy.json' -Recurse

To analyze them we iterate over each template and pass it to the above created Get-OutdatedResourceProvider cmdlet. Because there might be some invalid templates, we try to parse the JSON first and filter all valid templates:

$invalidTemplates = @()
$analyzedTemplates = $armTemplates |
    ForEach-Object {
    $template = $_
    try
    {
        $null = $template | Get-Content | ConvertFrom-Json
        Get-OutdatedResourceProvider -Path $template.FullName
    }
    catch
    {
        $invalidTemplates += $template
    }
}

Finally we gather and output some metrics:

$validAnalyzes = $analyzedTemplates | Where-Object LatestVersion
$uptodateproviderCount = $validAnalyzes | Where-Object LatestVersion -eq TRUE | Measure-Object | Select-Object -ExpandProperty Count
$outdatedproviderCount = $validAnalyzes | Where-Object LatestVersion -ne TRUE | Measure-Object | Select-Object -ExpandProperty Count

Write-Host "Found $($armTemplates.Count) ARM templates in the Azure quickstart repository ($($invalidTemplates.Count) of them are invalid).
They are using $($analyzedTemplates.Count) resource providers. I was able to determine the latest version for $($validAnalyzes.count) resource provider. $uptodateproviderCount of them are using the latest API version whereas $outdatedproviderCount are using an outdated API." -ForegroundColor Cyan

Here is the output:

Found 675 ARM templates in the Azure quickstart repository (3 of them are invalid).
They are using 3873 resource providers. I was able to determine the latest version for 3288 resource provider. 18 of them are using the latest API version whereas 3270 are using an outdated API.

Note: There is no need to always use the latest API version. You should consider upgrading only if you encounter issues with the resource provider or if you want to use new features that are not part of the currently used API.

You can download the complete script and run it for yourself here.

Determine latest API version for a resource provider

Azure Resource Manager templates are great to deploy one or more resources to a resource group.

A mandatory part of an ARM template is the resources section which defines the resource types that are deployed or updated. Here is an example:

"type": "Microsoft.Storage/storageAccounts",
"name": "[variables('storageAccountName')]",
"apiVersion": "2016-01-01",
"location": "[parameters('location')]",
"sku": {
	"name": "Standard_LRS"
},
"kind": "Storage",
"properties": {}

The name of a resource type has the following format: {resource-provider}/{resource-type}. In this example Microsoft.Storage/storageAccounts. The apiVersion specifies the version of the API that is used to deploy / manage the resource.

A resource provider offers a set of operations to deploy and manage resources using a REST API. If a resource provider adds new features, it releases a new version of the API. Therefore, if you have to utilize new features you may have to use the latest API Version. To determine the latest version of a resource type you can use the following Get-AzureRmResourceProviderLatestApiVersion cmdlet:

function Get-AzureRmResourceProviderLatestApiVersion
{
    [CmdletBinding()]
    [Alias()]
    [OutputType([string])]
    Param
    (
        [Parameter(ParameterSetName = 'Full', Mandatory = $true, Position = 0)]
        [string]$Type,

        [Parameter(ParameterSetName = 'ProviderAndType', Mandatory = $true, Position = 0)]
        [string]$ResourceProvider,

        [Parameter(ParameterSetName = 'ProviderAndType', Mandatory = $true, Position = 1)]
        [string]$ResourceType,

        [switch]$IncludePreview
    )

    # retrieving the resource providers is time consuming therefore we store
    # them in a script variable to accelerate subsequent requests.
    if (-not $script:resourceProvider)
    {
        $script:resourceProvider = Get-AzureRmResourceProvider   
    }

    if ($PSCmdlet.ParameterSetName -eq 'Full')
    {
        $ResourceProvider = ($Type -replace "\/.*")
        $ResourceType = ($Type -replace ".*?\/(.+)", '$1')
    }
    
    $provider = $script:resourceProvider | 
        Where-Object {
        $_.ProviderNamespace -eq $ResourceProvider -and 
        $_.ResourceTypes.ResourceTypeName -eq $ResourceType
    }

    if ($IncludePreview)
    {
        $provider.ResourceTypes.ApiVersions[0]
    }
    else
    {
        $provider.ResourceTypes.ApiVersions | Where-Object {
            $_ -notmatch '-preview'
        } | Select-Object -First 1
    }
}

You can now determine the latest API Version using either the full type:

Get-AzureRmResourceProviderLatestApiVersion -Type Microsoft.Storage/storageAccounts

Or by passing the Resource Provider and Resource Type:

Get-AzureRmResourceProviderLatestApiVersion -ResourceProvider Microsoft.Storage -ResourceType storageAccounts

Finally, to include preview versions you can append the -IncludePreview switch:

Get-AzureRmResourceProviderLatestApiVersion -Type Microsoft.Storage/storageAccounts -IncludePreview

You can download the script from my GitHub repository.

Additional information: Resource providers and types

Things you may consider when choosing a Microsoft Azure Region.

With over 50 regions worldwide, Azure has more global regions than any other cloud provider. When you have to choose a region for your Azure resources, consider these four factors:

  1. Not all services are available in all regions
    You have to ensure all resources you want to deploy are available within the desired region. Here is a nice overview of Products available by region.
  2. Prices vary by region
    Not all Azure Regions have the same pricing for the same resources. While most of the time you are looking for the closest Azure Region to your company / customers, you may consider choosing another Region due to the lower pricing model. A good resource to determine the price for a resource for a specific region is the Azure Pricing Calculator.
  3. Latency from your customer to the Azure datacenter
    For performance reason you want to choose the Azure region with the least latency to your company / customers. To measure the latency you can perform a Azure Latency Test on AzureSpeed.com
  4. Location of your customer data (data residency)
    You may have specific compliance or data-residency requirements that will force you to use a specific region. Get data residency details.

Additional information: Learn more about Azure Regions