Troubleshoot Error occurred while sending request to APNS WEB API
Overview
This article is intended to analyze problems with Apple device enrollment after a Silverback installation, which may occur due to a faulty network configuration or system requirements not being met. For this guide, we have captured all possible steps in the troubleshooting process for a scenario where Apple device enrollment problems persist on a recently installed Silverback instance. In the specific situation, the message Errors with the APNS WEB API appeared in the Silverback logs every minute, and once the maintenance service was stopped, the messages also stopped. This indicated that Silverback was trying to talk to the APNS every minute (while idle) and this is where the problem occurred. Silverback was not able to connect to the APNS at all, which ended up being due to an incorrect server and network configuration.
Error occurred while sending request to APNS WEB API: Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
The following chapters should help you to analyze and identify communication issues between Silverback and the Apple Push Notification Service.
Check Network Connectivity
- Ping api.push.apple.com from the Silverback server and check if its trying to connect to some IP from Apple starting from 17.
- Connect with telnet to api.push.apple.com with Port 443 and check that the connection is successful
Ensure to have Telnet Client Feature installed on the server and enter telnet api.push.apple.com 443 | This indiciates that you can access the target server via port 443 |
- Open a browser and open https://api.push.apple.com and review if you receive the following message, which is positive
Check Application Pool
Review if the Silverback.apns application pool is Started and has the same settings like in the screenshot.
Check APNS Service
After the Network Connectivity and Application Pool tests, review the second component that is involved. This is the .Net core component.
- For this you can go to the following adjusted URL of your Silverback Server: e.g., https://silverback.imagoverum.com/apnssender/sendpush
- You will see here the following in case of an error
- Normally it should look like this:
- If your current Hosting Bundle is installed from Version 3 and you missed the updated prerequisites introduced with Silverback 23.0 and missed to update it to version 6 or to a non-supported version 7 or newer, you will find the following error message for the /apnssender/sendpush page
Review .Net Core Hosting Bundle Installation
- Look in the registry and in the control panel to see if the (correct) Hosting Bundle is installed at all.
- HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Updates\.NET Core
- Probably the wrong Hosting Bundle is installed, like in the following example: Microsoft .Net Core Runtime - 3.1.22 (x64) or Microsoft .Net Core Runtime - 6.0.27 (x64) is not the correct one. It should be the named as Windows Server Hosting as shown in the section below.
Uninstall and Re-Install .Net Core Hosting Bundle
- Uninstall the current used .Net Core Hosting bundle
- Install the hosting bundle from this page: https://dotnet.microsoft.com/en-us/download/dotnet/6.0
- Afterwards check the system controls again and review if the Microsoft Net Core 6.0.XX - Windows Server Hosting and several additional ASP.Net Core components are installed.
- Restart your Internet Information Service
Review APNS Sender
- Open the /apnssender/sendpush page again and it should look like this
- This is exactly what we want to see.
Enroll Device
- Enrolled the device again and review if it's going now successfully to managed:
Analyze Network Traffic
In case your device can still not be enrolled, please focus on your network traffic.
- Install and Run Wireshark
- Shortly before you install the MDM Profile on the start, start capturing packets
- Review after the profile installation if the server is starting to try to communicate at all
- To granularly filter traffic to Apple, add the following filter: ip.addr == 17.0.0.0/8
- The output should look similar to this:
- In case your output is empty, please review the incoming traffic after installing the MDM Profile on the device
- On your device, remove your MDM profile and get your public IP address, e.g. by opening whatismyip.com
- Now open the Silverback SSP page, login and download a new MDM Profile
- Close now all Silverback related tabs in Safari and close Safari
- Right before installing the MDM Profile on the device, start capturing packets
- Now filter by your public IP-Address and review if the device is communicating to your server: ip.src == 89.247.127.244
- The output should look similar to this:
In case you have a Reverse Proxy in place, check the incoming traffic from the device from the Reverse Proxy.
Check Traffic on your Firewall / Proxy
- When Silverback is at the enrollment point communication to Apple, please review your outgoing network point in your environment and check if you can see Network Traffic in this direction on your Firewall or on your Proxy. If no traffic can be seen, check all the communication and check where the traffic might be blocked or gets lost.
Check Telnet for Network Service
- Install on your Silverback Server the sysinternals tool psexec
- Open your command prompt with elevated privileges
- Navigate to your installation folder that contains psexec.exe (e.g. C:\Tools\PsTools)
- Now execute the following command to start cmd.exe as Network Service
-
psexec -i -u "nt authority\network service" cmd.exe
- Check if you are running under the Network with executing the whoami command
- The answer should be nt authority\network service
- Now execute the following command: api.push.apple.com 443
- The output should look like this:
- If you can connect, close this windows
Check Installed APNS Certificates for Network Service
- Now execute the following command to start mmc.exe as Network Service
-
psexec -i -u "nt authority\network service" mmc.exe
-
- Add the certificates snap-in
- Check if under personal your APNS certificate is present (review Thumbprints with the Certificates section in Silverback)
- In case you have multiple APNS certificates in the store, note down the serial number from the currently used one
- Review if you have under Trusted Root Certification Authorities, the correct AAA Certificate Services certificate present
- Thumbprint: d1eb23a46d17d68fd92564c2f1f1601764d8e349
- Close the MMC
Open Apple API via Chrome as Network Service
- Now execute the following command to start Chrome as Network Service
- psexec -i -u "nt authority\network service" "C:\Program Files\Google\Chrome\Application\chrome.exe"
- Open https://api.push.apple.com
- You should be prompted with a certificate prompt
- Select now your current APNS certificate, e.g. APSP:77d6963f-22b5-4a89-8fb6-083563e5d373
- After selecting your correct APNS certificate, the output should look like this
- Close Google Chrome
Check Invoke Command as Network Service
- Now execute the following command to start Powershell as Network Service
-
psexec -i -u "nt authority\network service" powershell.exe
-
- Enter the following commands step by step:
$Body = @{ title = 'foo' body = 'bar' userId = 1 }
$JsonBody = $Body | ConvertTo-Json
$Params = @{ Method = "Post" Uri = "https://api.push.apple.com" Body = $JsonBody ContentType = "application/json" }
Invoke-RestMethod @Params
Review now the Server server answer and compare to the screenshot. With this answer you can verify if the server is able to receive an answer from the API. As we don't use HTTP/2 in this case, the answer should be a protocol violation.
Push Notification Message Logging
As the last step, you can enable the Push Notification Message Logging in Silverback and review the communication between Silverback and the APNS Sender application pool:
- Login to the Silverback Management Console as Settings Administrator
- Navigate to APNS
- Enable the checkbox for Enable Push Notification Message Logging and press Save
- Now Logout as Settings Administrator and Login as Administrator
- Re-enroll your device and navigate to the Log Section and review if you have the similar entries like below
This means that Silverback is sending out messages to the apnssender application on IIS, which is responsible to receive the Push Notifications from Silverback and transfer them into an HTTP2 request to Apple Push Notification Service. The next step is to create a Logs folder on your apnssender location on your Server.
- Open Windows Explorer and navigate to C:\inetpub\Silverback\apnssender
- Create a new folder named Logs
- Right Click the folder and click Properties
- Navigate to Security and click Edit
- Press Add and select as Location your Silverback Server
- Type in Enter the object names to select Network Service and press Check Names
- Click OK
- Assign Write Permissions and Press OK
- Close with OK
- Now edit the appsettings.Production.json with a Text Editor
- Change the Serilog Minimum Level from 6 to Information and save the file
- Now restart your Internet Information Service
- Re-enroll your Device and review the ApnsSender.log File in your recently created folder
- Compare the Log entry from the Silverback Management Console with the entry in the log file
- {"DeviceToken":"C5CD0D6E3D54D5EC493311B9A10EB7DBEAB4AAA5E324EA546075475408879ED5","PushMessage":"{\"mdm\":\"938EA16E-CF4B-4ADC-8339-D3FE41CF5DD3\"}","CertificateThumbprint":"937A5967CEE67DBBE4B2CB28F512A76941664344","UseSandBox":false,"UseReservePort":false}
- 2022-10-12T13:55:41.6897583+00:00 [INF] (Matrix42.SilverbackMDM.ApnsSender.Controllers.SendPushController) {"DeviceToken":"C5CD0D6E3D54D5EC493311B9A10EB7DBEAB4AAA5E324EA546075475408879ED5","PushMessage":"{\"mdm\":\"938EA16E-CF4B-4ADC-8339-D3FE41CF5DD3\"}","CertificateThumbprint":"937A5967CEE67DBBE4B2CB28F512A76941664344","UseSandBox":false,"UseReservePort":false}
If you receive something different, then something on the server is influencing the communication on the server itself. This could be a Proxy Server or any other faulty network configuration for example. If everything is fine here, then probably the problem is located in sending out the Notifications from the apnssender to the Apple Push Notification Service which should be already visible through the Network Traffic analysis.
Do not forget to revert the Serilog Level and restart IIS after your investigation as it might produce in the future a big amount of log entries.
Additional Notes
Since Microsoft constantly provides new versions for .Net Core Version 6, there can be errors under certain conditions or combinations (e.g. IIS version, Windows Server Version, Hosting Bundle version), which Microsoft then (hopefully) fixes again. So it may well be that you have to do a bit of experimentation with the different minor versions of the hosting bundles.