This page contains all current information on the status of this incident.
| Incident Details | |
|---|---|
| Incident Status | closed |
| Services Impacted |
Smarthub
iVUE Connect Service |
| Impact Level | N/A |
| Start Time | 2025-01-09 07:00 CST |
| End Time | 2025-01-09 16:00 CST |
| Description | We are investigating issues impacting iVUE Connect Service and SmartHub. Reports of users being unable to log into SmartHub and intermittent slowness/errors in Connect Services. |
| NISC Community post | Click here for more details |
| Incident Updates | |
|---|---|
| Time | Detail |
| 2025-01-09 09:25:31 CST | We have determined root cause and found that this only affects a subset of Members. Determining next steps for a fix. We will post another update in the next 30 minutes. |
| 2025-01-09 09:59:37 CST | We have determined root cause and found that this only affects a small subset of Members. We are working on a fix now. We will post another update in the next 30 minutes. |
| 2025-01-09 10:35:55 CST | An underlying subsystem was updated as part of the iVUE Connect Service upgrade last night. This subsystem introduced a poor performing SQL against the billing history table for a small subset of Members. We are working directly with those Members affected and will have a fix available soon. |
| 2025-01-09 11:24:19 CST | To clarify, the current issue results in slowness in iVUE Connect and SmartHub for a small subset of Members. We are currently working on a fix. If you are experiencing any other support issues, please contact NISC Support and we apologize any inconveniences. |
| 2025-01-09 12:16:40 CST | We continue to work on the fix for the Members impacted. |
| 2025-01-09 12:56:00 CST | We are currently testing the fix. After that our next step will be to deploy the fix to our iVUE Connect Beta channel and then to the remaining Members directly impacted |
| 2025-01-09 13:34:04 CST | As we are working on the fix, we have identified that additional Members are impacted with SmartHub slowness. We are close to having the fix deployed to the iVUE Connect Beta channel as well as additional testing sites. We are working on this very urgently |
| 2025-01-09 14:21:04 CST | NISC has taken steps to eliminate the slowness the majority of our Members were experiencing in SmartHub. Most sites would have started to see improvements around 2:10pm CST. We continue to work on the original issue that only impacted a small subset of our Members. That change will be going to beta sites in the next hour or so. |
| 2025-01-09 15:28:30 CST | We continue to see things running well for SmartHub. For the original issue we have deployed the fix to 4 sites and are monitoring. Results are promising and the areas causing concern are corrected. We will continue to monitor for a short period of time and then deploy the changes to the remaining sites impacted. |
| 2025-01-09 16:07:03 CST | We have deployed the fix to the remaining sites who were impacted by the original issue and all monitoring of SmartHub and iVUE Connect look good. The change will be sent to all other sites after 11:59 PM tonight. |
| 2025-01-09 17:02:13 CST | All monitoring continues to show the system performing as expected. Both the original issue and any residual impacts have been corrected. We will close this event but continue to monitor the system tonight and tomorrow. |