Adnan Hamid, Director, IT Services
Henry Mayo Newhall Memorial Hospital, Valencia, CA
At the International MUSE Conference, I facilitated an
education session on my hospital’s experience as a C/S 5.66 early adopter. With the delay of MU certification for 5.66,
our go-live was scheduled after the session for Tuesday, June 18. I promised the attendees in the session that
I would chronicle our experiences with the upgrade and share via the MUSE
website. In this article, you will find major upgrade activities along with
approximate start times. I have also included any of the major issues we
experienced during the upgrade.
We planned for a 12-hour downtime given our previous history
with upgrades and the uncertainty of being the early adopter. As early adopters, we requested to have MEDITECH
staff onsite to assist with any issues. MEDITECH
sent us three staff members. An Update
Programmer Manager, Update Specialist Supervisor, and our HCIS Coordinator. Their presence and ability to communicate
with MEDITECH during the upgrade was a critical success factor.
We delayed the start of the upgrade by two hours due to a
full Emergency Room. We started the upgrade at 3:00 am PDT. MEDITECH completed
the upgrade by 6:35 am. At this point,
our hospital validate team began reviewing each module for basic functionality
and access. We completed our validation
at 8:00 am and planned to lift the downtime flags at 8:30 am allowing our
Admission team to register the remaining patients that came through our doors
during the downtime.
Shortly after 8:30 am, we started getting calls from users
that the system was extremely slow and in some cases would not allow to enter
their login information. We were able to
replicate this issue in our command center and show to the MEDITECH staff
onsite. Quickly, the MEDITECH staff were
on the phone with the Technical team at MEDITECH. They were able to diagnose
the problem to the MIS server running two instances of the CSMagic.exe file. Once
they were able to remove one instance, we began to see stability in the system.
For caution sake, we decided to allow our users by departments to monitor
system stability. This process took about two hours and by 12:30 pm, we were
confident to allow all users back to the system.
For the rest of the day, we did get calls from users about
unusual menu freezes and not able to access certain routines. In all cases, we were able to resolve the
issue by uninstalling and reinstalling the MEDITECH Client on their
workstation. We kept the command center open until 11:00 pm the same day.
Towards the end, we did experience brief outages with FS applications (EMR,
PCS, EDM) due to issues with HUB Proxy Server. We continue to see random
outages since the upgrade. MEDITECH is well aware of the issue and working on
installing code on the Proxy Server to fix the issue.
Overall, the upgrade was a success! This was truly a TEAM effort by hospital and MEDITECH
team members. While we could not have predicted the duration of the outage and
subsequent issues, the time spent on planning for support and preparing our
users for downtime were the primary reasons for our success. I hope that you find this article helpful in
your planning efforts. If you have any
additional questions, please feel free to contact me at hamidae@henrymayo.com
|
DATE
|
TIME (PDT)
|
Upgrade Activity
|
Comments
|
|
6/11
|
9:00 AM
|
Completed PP#2 Validation Testing; ensure testing in
Citrix environment
|
|
|
6/11
|
3:30 PM
|
Go/No Go Decision
|
|
|
6/14
|
5:00 PM
|
Finalized Support Schedule for Upgrade
|
Expanded staff schedules to cover 12 hr shifts
|
|
6/17
|
8:53 AM
|
The UPT application reconciliation report and all
worklist items
|
Even though MT asks to close UPT 3 business days in
advance; waited until the day before to add RXM dTS
|
|
6/17
|
11 AM
|
MEDITECH Upgrade Status Meeting;
|
Meeting with Core Teams, Hospital Management to review
timeline and open issues; final go/no-go for upgrade
|
|
6/17
|
11:50 AM
|
Received Go from Hospital Team
|
|
|
6/17
|
12:02
|
MEDITECH starts pre-Live process. MEDITECH move the code
from TEST in a renamed status to the current LIVE ring, because the code is
renamed it will not affect the LIVE ring. Once Pre-LIVE is complete, MEDITECH
will NOT move changes to LIVE or translate code until after the update is
complete.
Do not retranslate custom reports, menus and rules.
Do not refile the EDM Tracker/Access dictionaries in LIVE
after the Pre-LIVE is complete, if they are refiled they will crash and
cannot be corrected till after the Go Live is complete.
|
|
|
6/17
|
12:40
|
MEDITECH Pre-Live process is complete
|
|
|
6/17
|
3 PM
|
Completed client update for workstations in non-clinical
areas
|
Utilize Citrix workstations in Nursing areas
|
|
6/17
|
3:30 PM
|
Finalized downtime
users
|
Still able to add downtime users even with downtime flag
set
|
|
6/17
|
6 PM
|
Confirmed Downtime Binders are available on Nursing Units
|
|
|
6/17
|
11 PM
|
Open up Upgrade Command Center in IT Trailer #1
|
|
|
6/17
|
11:30 P
|
Validated Patient Census Reports
|
|
|
6/18
|
12 AM
|
Verified Medical Record number procedure in Place
|
|
|
6/18
|
12 AM
|
Printed out all downtime reports
MAR
Patient Census
OR Schedules
|
|
|
6/18
|
12:09 AM
|
Called MEDITECH to postpone upgrade until 3 AM due to full
Emergency Room
|
|
|
6/18
|
12:30 A
|
Completed full system back-up
|
|
|
6/18
|
3:00 A
|
Authorized MEDITECH to move forward with upgrade
|
|
|
6/18
|
3:00 A
|
MEDITECH Software Support Technician (SST) began the
update
|
|
|
6/18
|
3:05 A
|
MT proceeded with the Go-LIVE process, FS is also now
starting their side
of the update as well.
|
|
|
6/18
|
3:24 A
|
The 5.66 FS services are now running and I confirmed EMR
sign-on on HMA-PXL01. I will now work on setting up the rest of the CS side
and updating the
CSproxyservers.
|
|
|
6/18
|
3:53 A
|
The object code rename has finished, we are proceeding
with the conversions and replays.
|
|
|
6/18
|
4:34 A
|
The CFSs are currently running
|
|
|
6/18
|
4:41 A
|
We have encountered an issue with the MIS CFSs.
Programming is currently reviewing and we will update once we have more
information
|
|
|
6/18
|
5:20 A
|
The MIS CFSs have been completed, we are now proceeding
with the Replays
|
|
|
6/18
|
5:30 A
|
Hospital Validation team reports to the Command Center
|
|
|
6/18
|
5:34 A
|
The replays are currently running
|
|
|
6/18
|
5:43 A
|
The Replays have now finished, we are now proceeding with
updating the LIVE bkgs and restarting LIVE jobs.
|
|
|
6/18
|
6:28 A
|
The HUB jobs are up and running and the CSproxy servers
are up to date.
|
|
|
6/18
|
6:35 A
|
The LIVE update is complete and downtime users have been
entered. We are now at restricted signon.
|
|
|
6/18
|
6:35A – 8:00 A
|
Hospital Validation team begins checking modules for
access, functionality, parameters.
Initially, system very slow and unable to emulate. MEDITECH discovers
the following:
There were no parameters set to replay to LIVE for MIS
The MIS CFS's are in a cancelled status. It may be related
to the configuration issue reported earlier
The MIS job XI MSG FILR: ERX.XI is running with minutes
since; NMI is reviewing
|
|
|
6/18
|
8:00 A
|
Validation Team 1st Roll Call; all modules report
functional; decide to lift downtime flags by 8:30 AM after last downtime patient
entered into the system
|
|
|
6/18
|
8:30 A
|
Downtime Flags lifted
|
|
|
6/18
|
8:57 A
|
After lifting downflags, all applications are reporting
slowness. Users are
experiencing timeouts when trying to log into the system
using single signon and then experience delays when trying to navigate the
menus. After accessing the applications, all routines are displaying response
issues.
|
|
|
6/18
|
10:42 A
|
There were two csmagic.exe running on the MIS server. This
was terminated and performance has improved. Units are being asked to start
using the system gradually and we are continuing to monitor response times.
This issue may have been caused from refiling the permissions on HMA-FS1
|
|
|
6/18
|
10:45 – 12:30 P
|
Gradually allowing users onto the system starting
|
|
|
6/18
|
12:30 P
|
All users back onto the system; no slowness experienced
|
|
|
6/18
|
1:00 P
|
Downtime Status Meeting
|
|
|
6/18
|
1:30 P
|
Send out master list notification to users
|
|
|
6/18
|
8:30 P
|
The MEDITECH HUBSproxy-HMA55_L " service terminate
unexpectedly
|
Caused outage for EMR, PCS,
|
|
6/18
|
8:45 P
|
Restarted Service
|
|
|
6/18
|
11:00 P
|
Close Command Center
|
|
|
6/19
|
12:00 A
|
The MEDITECH HUBSproxy-HMA55_L " service terminate
unexpectedly
|
Caused outage for EMR, PCS,
|
|
6/19
|
12:15 a
|
Restarted Service
|
|
|
6/19
|
12:27 P
|
Data Repository Available
|
According to MEDITECH, Data Repository module and database
will also experience a downtime but the timeframe is longer and will be down
between 8 to 24 hours
|
Issues:
|
Module
|
Description
|
Status
|
|
TECH
|
TCP proxy for HUBS Proxy server
|
Contains a few minor corrections
in addition to a new debugging feature that will allow MEDITECH to view any
issues that may occur and better identify what those issues may be
|
|
PCM
|
Physician Rounds Report Format
|
|
|
EDM
|
Setting up default printers for controlled
Medication Rx
|
Closed - users are unable to see
the listing of printers and there is no default defined.
|
|
PHA
|
Multiple interface errors coming
from your dispensing machines into PHA - Pyxis refills; Invalid Transaction
Type errors that are displaying from the Process Interface Errors screen in
PHA
|
CS PHA 10963 will correct this
issue
|
|
BBK
|
The codabar units will not scan
the expiration date, and the ISBT 128 units are giving a "Max # is 14
digits" message
|
Reload the system dictionaries in
LIVE
|
|
SCA
|
Cannot Transfer scans in SCA
from folder
|
MRM DTS 3175
|
|
NMI
|
Mass amount of ADT messages
since 5.66 go LIVE
|
It looks like the ADT generator
urns were reset to 6/14; needed to reset for current date
|
Adnan Hamid
Director, IT Services
Henry Mayo Newhall Memorial Hospital, Valencia, CA
Next Post ... What are Webinars Worth?
No comments:
Post a Comment