Maciej & Code

Deploying .NET application to Azure App Service using Azure CLI - Troubleshooting

Maciej Zwierzchlewski -

So you deployed your application to Azure App Service and are waiting for the 503 Service Unavailable status to go away, and it finally does. But instead of your app, you see an error that tells you nothing. It happens to everybody, and it might not be your fault at all!

Below are some of the error screens you might see.

HTTP Error 500.30 - ASP.NET Core app failed to start
502 - Web server received an invalid response while acting as a gateway or proxy server.

These don't really reveal much about what went wrong and how to solve the underlying issues. Thankfully there is a way to get more information. Go to Azure Portal and select your broken App Service. Then select the Diagnose and solve problems option in the sidebar menu.

This tab is a good place for finding out different issues with your application. To find out the root cause of a startup problem, it's best to select the Diagnostic Tools option and then check the Application Event Logs.

Hopefully, the event log contains information that will point you in the right direction. When it doesn't, it's always a good idea to check Microsoft's documentation at Common error troubleshooting for Azure App Service and IIS with ASP.NET Core because the answer might be there.

I hope the Azure Portal UI doesn't change by the time you'll need this information. If it does and you can't find the event logs this way, go to the Advanced Tools tab. Then you can open the Debug console and navigate to the LogFiles directory. It should contain an eventlog.xml file which is the source for the information shown in the troubleshooting UI.

Below you will find solutions to a few problems. These have troubled my app services recently.

Issue #1

Excerpt from eventlog.xml:

<Event>
    <System>
        <Provider Name="IIS AspNetCore Module V2"/>
        <EventID>1018</EventID>
        <Level>1</Level>
        <Task>0</Task>
        <Keywords>Keywords</Keywords>
        <TimeCreated SystemTime="2022-05-16T09:34:39Z"/>
        <EventRecordID>-1264572671</EventRecordID>
        <Channel>Application</Channel>
        <Computer>webwk000003</Computer>
        <Security/>
    </System>
    <EventData>
        <Data>Application '/LM/W3SVC/675285524/ROOT' with physical root 'C:\home\site\wwwroot\' hit unexpected managed exception, exception code = '0xe0434352'. Please check the stderr logs for more information.</Data>
        <Data>Process Id: 22284.</Data>
        <Data>File Version: 15.0.22048.15. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 6939d9ab90aa1e57bb0619bb28819f7bcbfdbb54</Data>
    </EventData>
</Event>

The exception code (0xe0434352) is described in Microsoft's troubleshooting guide mentioned above. According to that, the issue is:

An x86 app is deployed but the app pool isn't enabled for 32-bit apps

But that couldn't be the case as the project was built explicitly with the win-x64 Runtime ID. But that did suggest there was a problem with the bitness of the application.

So why was the project built for the 64-bit runtime? I assumed that would be the default for any freshly created Azure App Service. It turns out that was not the correct assumption:

After rebuilding the project, this time for the win-x86 runtime and redeploying it, the error changed! So the documentation was half-right - the bitness was wrong, but it was the opposite of what was described.

Issue #2

This problem was directly caused by first deploying the application built for the wrong CPU bitness, but it persisted after redeploying the rebuilt project.

Excerpt from eventlog.xml:

<Event>
    <System>
        <Provider Name=".NET Runtime"/>
        <EventID>1026</EventID>
        <Level>1</Level>
        <Task>0</Task>
        <Keywords>Keywords</Keywords>
        <TimeCreated SystemTime="2022-05-17T08:28:53Z"/>
        <EventRecordID>-1182119390</EventRecordID>
        <Channel>Application</Channel>
        <Computer>webwk000003</Computer>
        <Security/>
    </System>
    <EventData>
        <Data>
            Application: w3wp.exe
            CoreCLR Version: 5.0.1522.11506
            .NET Version: 5.0.15
            Description: The process was terminated due to an unhandled exception.
            Exception Info: System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.TdsParser' threw an exception.
            ---> System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SNILoadHandle' threw an exception.
            ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)
            at Microsoft.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize(IntPtr pmo)
            at Microsoft.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize()
            at Microsoft.Data.SqlClient.SNILoadHandle..ctor()
            at Microsoft.Data.SqlClient.SNILoadHandle..cctor()
                ...
            at MyApp.Program.Main(String[] args) in /builds/myapp/API/Program.cs:line 26
            at MyApp.Program.<Main>(String[] args)
        </Data>
    </EventData>
</Event

This issue is better than the first problem since the Provider node points out that the problem happened when the .NET runtime was starting the application. Previously it crashed before that had happened - on the IIS level.

Googling the exception message (An attempt was made to load a program with an incorrect format. (0x8007000B)) once again pointed in the direction of wrong bitness. I used the sigcheck utility to verify the deployed files, but all seemed in order. 

Sigcheck v2.82 - File version and signature viewer
Copyright (C) 2004-2021 Mark Russinovich
Sysinternals - www.sysinternals.com

.\sni.dll:
        Verified:       Signed
        Signing date:   8:23 PM 7/12/2017
        Publisher:      Microsoft Corporation
        Company:        Microsoft Corporation
        Description:    n/a
        Product:        Microsoft« .NET Framework
        Prod version:   4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
        File version:   4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
        MachineType:    32-bit
        
.\Microsoft.Data.SqlClient.SNI.dll:
        Verified:       Signed
        Signing date:   6:43 PM 11/17/2021
        Publisher:      Microsoft Corporation
        Company:        Microsoft Corporation
        Description:    Microsoft.Data.SqlClient.SNI
        Product:        n/a
        Prod version:   4.0.0.0
        File version:   4.0.0.0
        MachineType:    32-bit

After deploying the application a couple more times without any improvements, I wanted to verify that everything was getting deployed correctly. I downloaded the two assemblies from the App Service (using Kudu aka Advanced Tools).

Sigcheck v2.82 - File version and signature viewer
Copyright (C) 2004-2021 Mark Russinovich
Sysinternals - www.sysinternals.com

.\Microsoft.Data.SqlClient.SNI.dll:
        Verified:       Signed
        Signing date:   6:43 PM 11/17/2021
        Publisher:      Microsoft Corporation
        Company:        Microsoft Corporation
        Description:    Microsoft.Data.SqlClient.SNI
        Product:        n/a
        Prod version:   4.0.0.0
        File version:   4.0.0.0
        MachineType:    64-bit
        
.\sni.dll:
        Verified:       Signed
        Signing date:   8:23 PM 7/12/2017
        Publisher:      Microsoft Corporation
        Company:        Microsoft Corporation
        Description:    n/a
        Product:        Microsoft« .NET Framework
        Prod version:   4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
        File version:   4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
        MachineType:    64-bit

Those were not the deployed files. It turns out that the deployment uses a tool called KuduSync.NET, which doesn't copy the files it deems unchanged, and somehow it couldn't figure out that those files differed.

What finally fixed the problem was manually removing the two 64-bit dll files and redeploying the application. This time it finally started and was running correctly!

I was using the Azure CLI to deploy the project to Azure Web Service (described in detail in my previous post). The az webapp deploy accepts a --clean true parameter that is supposed to wipe the deploy target directory before copying the new files, but that didn't change anything in this case. This looks like a bug that will hopefully be fixed in the future.

Issue #3

Excerpt from eventlog.xml:

 <Event>
        <System>
            <Provider Name=".NET Runtime"/>
            <EventID>1026</EventID>
            <Level>1</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2022-05-24T06:31:01Z"/>
            <EventRecordID>213793516</EventRecordID>
            <Channel>Application</Channel>
            <Computer>DW0SDWK0002IE</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>
                Application: w3wp.exe
                CoreCLR Version: 6.0.322.12309
                .NET Version: 6.0.3
                Description: The process was terminated due to an unhandled exception.
                Exception Info: System.PlatformNotSupportedException: System.Data.SqlClient is not supported on this platform.
                at System.Data.SqlClient.SqlConnection.Dispose(Boolean disposing)
                at System.ComponentModel.Component.Finalize()
            </Data>
    </EventData>
</Event>

This issue happened after deploying a project which had been upgraded from .NET 5 to .NET 6 with a simultaneous update to the newest Entity Framework Core version. The first action was checking for differences with sigcheck:

Sigcheck v2.82 - File version and signature viewer
Copyright (C) 2004-2021 Mark Russinovich
Sysinternals - www.sysinternals.com

.\System.Data.SqlClient (Build).dll:
        Verified:       Signed
        Signing date:   6:54 PM 7/19/2017
        Publisher:      Microsoft Corporation
        Company:        Microsoft Corporation
        Description:    System.Data.SqlClient
        Product:        Microsoft« .NET Framework
        Prod version:   4.6.25519.03 built by: dlab-DDVSOWINAGE013. Commit Hash: 8321c729934c0f8be754953439b88e6e1c120c24
        File version:   4.6.25519.03
        MachineType:    32-bit
        
.\System.Data.SqlClient (App Service).dll:
        Verified:       Signed
        Signing date:   6:52 PM 7/19/2017
        Publisher:      Microsoft Corporation
        Company:        Microsoft Corporation
        Description:    System.Data.SqlClient
        Product:        Microsoft« .NET Framework
        Prod version:   4.6.25519.03 built by: dlab-DDVSOWINAGE013. Commit Hash: 8321c729934c0f8be754953439b88e6e1c120c24
        File version:   4.6.25519.03
        MachineType:    32-bit

Bitness is fine, the file version is the same. But as the files were next to each other in the same directory, the difference between them was very noticeable:

Hint: It's the size

Once again, removing the file from the App Service manually and redeploying the project made the problem go away.

More in-depth information about application deployment using Kudu can be found in this fantastic blog post: Azure Web Apps Continuous Deployment.

Update (23.09.2022): A follow-up to this post featuring further problems with Issue #3 was published:

Deploying .NET application to Azure App Service using Azure CLI - Troubleshooting Part II
Deploying a .NET application using Azure CLI sounds like an easy job. And it usually is. As with everything, sometimes problems occur, but with the help of Google (well, StackOverflow, mostly), you should be able to solve them.
Maciej & CodeMaciej Zwierzchlewski

Cover photo by Jack Douglass on Unsplash