On Monday morning, 11/13/2023, users within the HCA and Lifepoint networks started to experience connectivity issues with the Dr. Know product. This quickly became a critical issue impacting the entire organization.
Update - 11/13/2023
Problem Summary:
- The Dr. Know application was randomly unable to communicate with its API endpoint. - Many users could not login to the product, while those who could, had random connection issues while using the product. - Users had some success by switching to different computers. - The issue seems random and does not behave in a predictable or easily reproducible manner. - HCA local IT support was unable to help. - Speed tests showed significant variance in speeds at the NCMC location. Not sure if related. - The Dr. Know product is installed via HCA's 'Software Center.' The product does not appear for some users inside Software Center. Not sure if this is related. HCA IT said they have to manually install the product in those scenarios and that this is a known issues that is likely related to a Group Policy configuration. - Incidentally, the timeline for this issue perfectly coincides with an HCA training at TNHV facilities for an e-Prescribing transition from Care360 to eCW. Not sure if related.
Update - 11/14/2023
- The issue seems to be growing. - More users are unable to login to Dr. Know or have random connection issues while logged in. - This problem exists only inside the HCA / Lifepoint networks. All outside testing is normal. - Dr. Know contacted other customers, who verified they are having no issues.
Update - 11/15/2023 @ 11:15 AM
- The problem was successfully identified and resolved. - HCA opened a critical, all hands-on-deck ticket, to help resolve the issue. - A WebEx call with 13 senior participants helped identify and resolve the issue.
Resolution Summary:
- Unknown to us, HCA did an update to their central proxy servers on Monday (updated to v.12.5) in order to support TLS 1.3 - The Dr. Know API endpoint, hosted in Azure, currently supports TLS 1.2 - Unknown to HCA, the newer proxy version required an added 'SSL Passthrough' rule be applied in order to allow TLS 1.2 - The issue appeared random, but growing over the last 72 hours, because the proxy change was a staggered rollout to user PCs. When connected to the new proxy, they had the issue. When connected to the old proxy, they were fine. - Even though Dr. Know is 'proxy unaware,' HCA utilizes several proxy servers to help minimize the outbound connections to their main firewall. - HCA made, tested, and deployed the necessary proxy change everywhere. - HCA indicated that the Lifepoint network should be resolved by this change as well as they share the same infrastructure. - Impacted users seem to have their normal connectivity and operations restored.