So you deployed your application to Azure App Service and are waiting for the 503 Service Unavailable status to go away, and it finally does. But instead of your app, you see an error that tells you nothing. It happens to everybody, and it might not be your fault at all!
Below are some of the error screens you might see.
These don't really reveal much about what went wrong and how to solve the underlying issues. Thankfully there is a way to get more information. Go to Azure Portal and select your broken App Service. Then select the Diagnose and solve problems option in the sidebar menu.
This tab is a good place for finding out different issues with your application. To find out the root cause of a startup problem, it's best to select the Diagnostic Tools option and then check the Application Event Logs.
Hopefully, the event log contains information that will point you in the right direction. When it doesn't, it's always a good idea to check Microsoft's documentation at Common error troubleshooting for Azure App Service and IIS with ASP.NET Core because the answer might be there.
I hope the Azure Portal UI doesn't change by the time you'll need this information. If it does and you can't find the event logs this way, go to the Advanced Tools tab. Then you can open the Debug console and navigate to the
LogFilesdirectory. It should contain aneventlog.xmlfile which is the source for the information shown in the troubleshooting UI.
Below you will find solutions to a few problems. These have troubled my app services recently.
Issue #1
Excerpt from eventlog.xml:
<Event>
<System>
<Provider Name="IIS AspNetCore Module V2"/>
<EventID>1018</EventID>
<Level>1</Level>
<Task>0</Task>
<Keywords>Keywords</Keywords>
<TimeCreated SystemTime="2022-05-16T09:34:39Z"/>
<EventRecordID>-1264572671</EventRecordID>
<Channel>Application</Channel>
<Computer>webwk000003</Computer>
<Security/>
</System>
<EventData>
<Data>Application '/LM/W3SVC/675285524/ROOT' with physical root 'C:\home\site\wwwroot\' hit unexpected managed exception, exception code = '0xe0434352'. Please check the stderr logs for more information.</Data>
<Data>Process Id: 22284.</Data>
<Data>File Version: 15.0.22048.15. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 6939d9ab90aa1e57bb0619bb28819f7bcbfdbb54</Data>
</EventData>
</Event>The exception code (0xe0434352) is described in Microsoft's troubleshooting guide mentioned above. According to that, the issue is:
An x86 app is deployed but the app pool isn't enabled for 32-bit apps
But that couldn't be the case as the project was built explicitly with the win-x64 Runtime ID. But that did suggest there was a problem with the bitness of the application.
So why was the project built for the 64-bit runtime? I assumed that would be the default for any freshly created Azure App Service. It turns out that was not the correct assumption:
After rebuilding the project, this time for the win-x86 runtime and redeploying it, the error changed! So the documentation was half-right - the bitness was wrong, but it was the opposite of what was described.
Issue #2
This problem was directly caused by first deploying the application built for the wrong CPU bitness, but it persisted after redeploying the rebuilt project.
Excerpt from eventlog.xml:
<Event>
<System>
<Provider Name=".NET Runtime"/>
<EventID>1026</EventID>
<Level>1</Level>
<Task>0</Task>
<Keywords>Keywords</Keywords>
<TimeCreated SystemTime="2022-05-17T08:28:53Z"/>
<EventRecordID>-1182119390</EventRecordID>
<Channel>Application</Channel>
<Computer>webwk000003</Computer>
<Security/>
</System>
<EventData>
<Data>
Application: w3wp.exe
CoreCLR Version: 5.0.1522.11506
.NET Version: 5.0.15
Description: The process was terminated due to an unhandled exception.
Exception Info: System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.TdsParser' threw an exception.
---> System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SNILoadHandle' threw an exception.
---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)
at Microsoft.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize(IntPtr pmo)
at Microsoft.Data.SqlClient.SNINativeMethodWrapper.SNIInitialize()
at Microsoft.Data.SqlClient.SNILoadHandle..ctor()
at Microsoft.Data.SqlClient.SNILoadHandle..cctor()
...
at MyApp.Program.Main(String[] args) in /builds/myapp/API/Program.cs:line 26
at MyApp.Program.<Main>(String[] args)
</Data>
</EventData>
</EventThis issue is better than the first problem since the Provider node points out that the problem happened when the .NET runtime was starting the application. Previously it crashed before that had happened - on the IIS level.
Googling the exception message (An attempt was made to load a program with an incorrect format. (0x8007000B)) once again pointed in the direction of wrong bitness. I used the sigcheck utility to verify the deployed files, but all seemed in order.
Sigcheck v2.82 - File version and signature viewer
Copyright (C) 2004-2021 Mark Russinovich
Sysinternals - www.sysinternals.com
.\sni.dll:
Verified: Signed
Signing date: 8:23 PM 7/12/2017
Publisher: Microsoft Corporation
Company: Microsoft Corporation
Description: n/a
Product: Microsoft« .NET Framework
Prod version: 4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
File version: 4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
MachineType: 32-bit
.\Microsoft.Data.SqlClient.SNI.dll:
Verified: Signed
Signing date: 6:43 PM 11/17/2021
Publisher: Microsoft Corporation
Company: Microsoft Corporation
Description: Microsoft.Data.SqlClient.SNI
Product: n/a
Prod version: 4.0.0.0
File version: 4.0.0.0
MachineType: 32-bitAfter deploying the application a couple more times without any improvements, I wanted to verify that everything was getting deployed correctly. I downloaded the two assemblies from the App Service (using Kudu aka Advanced Tools).
Sigcheck v2.82 - File version and signature viewer
Copyright (C) 2004-2021 Mark Russinovich
Sysinternals - www.sysinternals.com
.\Microsoft.Data.SqlClient.SNI.dll:
Verified: Signed
Signing date: 6:43 PM 11/17/2021
Publisher: Microsoft Corporation
Company: Microsoft Corporation
Description: Microsoft.Data.SqlClient.SNI
Product: n/a
Prod version: 4.0.0.0
File version: 4.0.0.0
MachineType: 64-bit
.\sni.dll:
Verified: Signed
Signing date: 8:23 PM 7/12/2017
Publisher: Microsoft Corporation
Company: Microsoft Corporation
Description: n/a
Product: Microsoft« .NET Framework
Prod version: 4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
File version: 4.6.25512.01 built by: dlab-DDVSOWINAGE016. Commit Hash: d0d5c7b49271cadb6d97de26d8e623e98abdc8db
MachineType: 64-bitThose were not the deployed files. It turns out that the deployment uses a tool called KuduSync.NET, which doesn't copy the files it deems unchanged, and somehow it couldn't figure out that those files differed.
What finally fixed the problem was manually removing the two 64-bit dll files and redeploying the application. This time it finally started and was running correctly!
I was using the Azure CLI to deploy the project to Azure Web Service (described in detail in my previous post). Theaz webapp deployaccepts a--clean trueparameter that is supposed to wipe the deploy target directory before copying the new files, but that didn't change anything in this case. This looks like a bug that will hopefully be fixed in the future.
Issue #3
Excerpt from eventlog.xml:
<Event>
<System>
<Provider Name=".NET Runtime"/>
<EventID>1026</EventID>
<Level>1</Level>
<Task>0</Task>
<Keywords>Keywords</Keywords>
<TimeCreated SystemTime="2022-05-24T06:31:01Z"/>
<EventRecordID>213793516</EventRecordID>
<Channel>Application</Channel>
<Computer>DW0SDWK0002IE</Computer>
<Security/>
</System>
<EventData>
<Data>
Application: w3wp.exe
CoreCLR Version: 6.0.322.12309
.NET Version: 6.0.3
Description: The process was terminated due to an unhandled exception.
Exception Info: System.PlatformNotSupportedException: System.Data.SqlClient is not supported on this platform.
at System.Data.SqlClient.SqlConnection.Dispose(Boolean disposing)
at System.ComponentModel.Component.Finalize()
</Data>
</EventData>
</Event>This issue happened after deploying a project which had been upgraded from .NET 5 to .NET 6 with a simultaneous update to the newest Entity Framework Core version. The first action was checking for differences with sigcheck:
Sigcheck v2.82 - File version and signature viewer
Copyright (C) 2004-2021 Mark Russinovich
Sysinternals - www.sysinternals.com
.\System.Data.SqlClient (Build).dll:
Verified: Signed
Signing date: 6:54 PM 7/19/2017
Publisher: Microsoft Corporation
Company: Microsoft Corporation
Description: System.Data.SqlClient
Product: Microsoft« .NET Framework
Prod version: 4.6.25519.03 built by: dlab-DDVSOWINAGE013. Commit Hash: 8321c729934c0f8be754953439b88e6e1c120c24
File version: 4.6.25519.03
MachineType: 32-bit
.\System.Data.SqlClient (App Service).dll:
Verified: Signed
Signing date: 6:52 PM 7/19/2017
Publisher: Microsoft Corporation
Company: Microsoft Corporation
Description: System.Data.SqlClient
Product: Microsoft« .NET Framework
Prod version: 4.6.25519.03 built by: dlab-DDVSOWINAGE013. Commit Hash: 8321c729934c0f8be754953439b88e6e1c120c24
File version: 4.6.25519.03
MachineType: 32-bit
Bitness is fine, the file version is the same. But as the files were next to each other in the same directory, the difference between them was very noticeable:
Once again, removing the file from the App Service manually and redeploying the project made the problem go away.
More in-depth information about application deployment using Kudu can be found in this fantastic blog post: Azure Web Apps Continuous Deployment.
Update (23.09.2022): A follow-up to this post featuring further problems with Issue #3 was published:
Cover photo by Jack Douglass on Unsplash