Skip to main content
Matrix42 Self-Service Help Center

Troubleshoot Error occurred while sending request to APNS WEB API

Overview

This article is intended to analyze problems with Apple device enrollment after a Silverback installation, which may occur due to a faulty network configuration or system requirements not being met. For this guide, we have captured all possible steps in the troubleshooting process for a scenario where Apple device enrollment problems persist on a recently installed Silverback instance. In the specific situation, the message Errors with the APNS WEB API appeared in the Silverback logs every minute, and once the maintenance service was stopped, the messages also stopped. This indicated that Silverback was trying to talk to the APNS every minute (while idle) and this is where the problem occurred. Silverback was not able to connect to the APNS at all, which ended up being due to an incorrect server and network configuration.

Error occurred while sending request to APNS WEB API: Newtonsoft.Json.JsonReaderException: 
Unexpected character encountered while parsing value: <. Path '', line 0, position 0.

The following chapters should help you to analyze and identify communication issues between Silverback and the Apple Push Notification Service. 

Check Network Connectivity

  • Ping api.push.apple.com from the Silverback server and check if its trying to connect to some IP from Apple starting from 17. 

clipboard_e8e81e5d61ebc683bfb162b7b4cf4577e.png

  • Connect with telnet to api.push.apple.com with Port 443 and check that the connection is successful
Ensure to have Telnet Client Feature installed on the server and enter telnet api.push.apple.com 443 This indiciates that you can access the target server via port 443
clipboard_e6862dbaa2b15596b1dc7a1b0b29006ae.png clipboard_ea01dc592cba469d118d50fd6ecb989c5.png

clipboard_e2122f61b32646b2beffd6d2aa9c51e09.png

Check Application Pool

Review if the Silverback.apns application pool is Started and has the same settings like in the screenshot.

clipboard_e358c1d8a38b37820fb58e855c7fc3c70.png

Check APNS Service

After the Network Connectivity and Application Pool tests, review the second component that is involved. This is the .Net core component.

clipboard_ea8bbde2f257d0fd15916a625b659e3a6.png

  • Normally it should look like this:

clipboard_e732159d62a2cf5f57c310b1908e2da1e.png

  • If your current Hosting Bundle is installed from Version 3 and you missed the updated prerequisites introduced with Silverback 23.0 and missed to update it to version 6 or to a non-supported version 7 or newer, you will find the following error message for the /apnssender/sendpush page

clipboard_e0aca3dcf43be712eed9f588bd720ac13.png

Review .Net Core Hosting Bundle Installation

  • Look in the registry and in the control panel to see if the (correct) Hosting Bundle is installed at all.
    • HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Updates\.NET Core
    • Probably the wrong Hosting Bundle is installed, like in the following example: Microsoft .Net Core Runtime - 3.1.22 (x64) or Microsoft .Net Core Runtime - 6.0.27 (x64)  is not the correct one. It should be the named as Windows Server Hosting as shown in the section below. 

clipboard_ee2564a105761facc3ca0d4db388311cf.png

Uninstall and Re-Install .Net Core Hosting Bundle

clipboard_e1cb27db1c778642fcac334402b5ea995.png

  • Afterwards check the system controls again and review if the  Microsoft Net Core 6.0.XX - Windows Server Hosting and several additional ASP.Net Core components are installed.

clipboard_ea204a9f5e2b9ac072c332ed462ae9c16.png

  • Restart your Internet Information Service

Review APNS Sender

  • Open the /apnssender/sendpush page again and it should look like this

clipboard_e7bf8228b042401cfa5693db30f002491.png

  • This is exactly what we want to see.

Enroll Device

  • Enrolled the device again and review if it's going now successfully to managed:

clipboard_e7591a9bd557025782285461de60e56b8.png

Analyze Network Traffic

In case your device can still not be enrolled, please focus on your network traffic. 

  • Install and Run Wireshark 
  • Shortly before you install the MDM Profile on the start, start capturing packets
  • Review after the profile installation if the server is starting to try to communicate at all
  • To granularly filter traffic to Apple, add the following filter: ip.addr == 17.0.0.0/8
  • The output should look similar to this:

clipboard_e826c7f7b31466d0d7643e754f4cf97c3.png

  • In case your output is empty, please review the incoming traffic after installing the MDM Profile on the device
  • On your device, remove your MDM profile and get your public IP address, e.g. by opening whatismyip.com 
  • Now open the Silverback SSP page, login and download a new MDM Profile
  • Close now all Silverback related tabs in Safari and close Safari
  • Right before installing the MDM Profile on the device, start capturing packets
  • Now filter by your public IP-Address and review if the device is communicating to your server: ip.src == 89.247.127.244
  • The output should look similar to this: 

clipboard_e2839a1da170598b84d38df22b3f2786c.png

In case you have a Reverse Proxy in place, check the incoming traffic from the device from the Reverse Proxy. 

Check Traffic on your Firewall / Proxy

  • When Silverback is at the enrollment point communication to Apple, please review your outgoing network point in your environment and check if you can see Network Traffic in this direction on your Firewall or on your Proxy. If no traffic can be seen, check all the communication and check where the traffic might be blocked or gets lost. 

Check Telnet for Network Service

  • Install on your Silverback Server the sysinternals tool psexec 
  • Open your command with elevated privileges
    • Navigate to your installation folder that contains psexec.exe (e.g. C:\Tools\PsTools)
    • Now execute the following command to start cmd.exe as Network Service 
    • psexec -i -u "nt authority\network service" cmd.exe
    • Check if you are running under the Network with executing the whoami command
    • The answer should be nt authority\network service
    • Now execute the following command: api.push.apple.com 443 
    • The output should look like this:
    • clipboard_efc05d0460042bea42703b469c81860db.png
  • If you can connect, close this windows

Check Installed APNS Certificates for Network Service

  • Now execute the following command to start mmc.exe as Network Service
    • psexec -i -u "nt authority\network service" mmc.exe
  • Add the certificates snap-in
  • Check if under personal your APNS certificate is present (review Thumbprints with the Certificates section in Silverback)
  • In case you have multiple APNS certificates in the store, note down the serial number from the currently used one
  • Review if you have under Trusted Root Certification Authorities, the correct AAA Certificate Services certificate present
    • Thumbprint: d1eb23a46d17d68fd92564c2f1f1601764d8e349
  • Close the MMC

Open Apple API via Chrome as Network Service

  • Now execute the following command to start Chrome as Network Service
    • psexec -i -u "nt authority\network service" "C:\Program Files\Google\Chrome\Application\chrome.exe" 
  • Open https://api.push.apple.com
  • You should be prompted with a certificate prompt 
  • Select now your current APNS certificate, e.g. APSP:77d6963f-22b5-4a89-8fb6-083563e5d373
  • After selecting your correct APNS certificate, the output should look like this

clipboard_e0de1777dde0e6866d3bae25e6a3f32f8.png

  • Close Google Chrome

Check Invoke Command as Network Service

  • Now execute the following command to start Powershell as Network Service
    • psexec -i -u "nt authority\network service" powershell.exe
  • Enter the following commands step by step: 
$Body = @{
title = 'foo'
body = 'bar'
userId = 1 }
$JsonBody = $Body | ConvertTo-Json
$Params = @{
Method = "Post"
Uri = "https://api.push.apple.com"
Body = $JsonBody
ContentType = "application/json"
}
Invoke-RestMethod @Params

Review now the Server server answer and compare to the screenshot. With this answer you can verify if the server is able to receive an answer from the API. As we don't use HTTP/2 in this case, the answer should be a protocol violation. 

Push Notification Message Logging

As the last step, you can enable the Push Notification Message Logging in Silverback and review the communication between Silverback and the APNS Sender application pool:

  • Login to the Silverback Management Console as Settings Administrator
  • Navigate to APNS
  • Enable the checkbox for Enable Push Notification Message Logging and press Save 
  • Now Logout as Settings Administrator and Login as Administrator 
  • Re-enroll your device and navigate to the Log Section and review if you have the similar entries like below

clipboard_ed4f8d001fe88cd2d9f3d5ed0fee108a8.png

This means that Silverback is sending out messages to the apnssender application on IIS, which is responsible to receive the Push Notifications from Silverback and transfer them into an HTTP2 request to Apple Push Notification Service. The next step is to create a Logs folder on your apnssender location on your Server.

  • Open Windows Explorer and navigate to C:\inetpub\Silverback\apnssender
  • Create a new folder named Logs
  • Right Click the folder and click Properties
  • Navigate to Security and click Edit
  • Press Add and select as Location your Silverback Server
  • Type in Enter the object names to select Network Service and press Check Names
  • Click OK
  • Assign Write Permissions and Press OK
  • Close with OK
  • Now edit the appsettings.Production.json with a Text Editor
  • Change the Serilog Minimum Level from 6 to Information and save the file
clipboard_ed66f616dcf124cf36f10fa04388949ae.png clipboard_e56711c5347c1e2bdaf6412b00b3ff620.png
  • Now restart your Internet Information Service
  • Re-enroll your Device and review the ApnsSender.log File in your recently created folder
  • Compare the Log entry from the Silverback Management Console with the entry in the log file
    • {"DeviceToken":"C5CD0D6E3D54D5EC493311B9A10EB7DBEAB4AAA5E324EA546075475408879ED5","PushMessage":"{\"mdm\":\"938EA16E-CF4B-4ADC-8339-D3FE41CF5DD3\"}","CertificateThumbprint":"937A5967CEE67DBBE4B2CB28F512A76941664344","UseSandBox":false,"UseReservePort":false}
    • 2022-10-12T13:55:41.6897583+00:00 [INF] (Matrix42.SilverbackMDM.ApnsSender.Controllers.SendPushController) {"DeviceToken":"C5CD0D6E3D54D5EC493311B9A10EB7DBEAB4AAA5E324EA546075475408879ED5","PushMessage":"{\"mdm\":\"938EA16E-CF4B-4ADC-8339-D3FE41CF5DD3\"}","CertificateThumbprint":"937A5967CEE67DBBE4B2CB28F512A76941664344","UseSandBox":false,"UseReservePort":false}

If you receive something different, then something on the server is influencing the communication on the server itself. This could be a Proxy Server or any other faulty network configuration for example. If everything is fine here, then probably the problem is located in sending out the Notifications from the apnssender to the Apple Push Notification Service which should be already visible through the Network Traffic analysis. 

Do not forget to revert the Serilog Level and restart IIS after your investigation as it might produce in the future a big amount of log entries.

Additional Notes

Since Microsoft constantly provides new versions for .Net Core Version 6, there can be errors under certain conditions or combinations (e.g. IIS version, Windows Server Version, Hosting Bundle version), which Microsoft then (hopefully) fixes again. So it may well be that you have to do a bit of experimentation with the different minor versions of the hosting bundles.