MotioCI Saves Corrupted IBM Cognos Content Store at DaVita HealthCare

Executive Summary

DaVita previously relied on a laborious method of deploying BI content between IBM Cognos environments that did not have any real rollback or versioning capabilities of content store objects. This method put DaVita at risk of losing a lot of BI development work. DaVita implemented MotioCI to improve deployment and mitigate such risks. In addition, MotioCI enabled DaVita to restore their entire Cognos content store database, which was corrupted. About DaVita DaVita HealthCare Partners Inc. is a Fortune 500® company that provides a variety of health care services to patient populations throughout the United States and abroad. A leading provider of dialysis services in the United States, DaVita Kidney Care treats patients with chronic kidney failure and end stage renal disease. DaVita Kidney Care strives to improve patient quality of life by innovating clinical care, and by offering integrated treatment plans, personalized care teams and convenient health management services.

DaVita’s IBM Cognos Implementation

IBM Cognos is one of several applications within DaVita’s IT infrastructure. Five years ago, DaVita installed Cognos version 8.4 in their BI environment, which includes a Dev, Test/QA, and Production server. DaVita’s IT infrastructure team members are located at their Denver headquarters and throughout the country. Within DaVita’s IT infrastructure department is a BI operations team, consisting of a primary IT administrator, 3 employees who have admin and promotion capabilities, and 10 report authors. Outside of the IT team, there are 9,000 named Cognos users, who are primarily report consumers. Several standalone subsidiaries of DaVita can develop their own, separate BI reports and host them on the shared Cognos environment. DaVita’s Cognos content store comprises of thousands of objects.

DaVita’s BI Challenges

DaVita’s process of deploying BI content was time consuming, tedious, and error-prone. They also faced the daily risk of losing development work by not having a version control system in place.

DaVita’s BI Challenges

DaVita’s original deployment process consisted of exporting content from Dev to Test to Prod.

1. First, they would create the export archive in Dev and check it into a version control system.
2. They would then import it into the Test environment and deploy.

This process created an “artificial safety net.” In other words, the process felt good, but it was not very functional or reliable. If a user needed to recover a report, an administrator would need to retrieve the correct version of the deployment archive from the repository and import it to a sandbox to retrieve the report spec of an individual report. That spec would then need to be placed in the target environment, which could potentially be out of sync with its package. In addition, the report spec may or may not be the version that the user requested. Besides its complexity, the problem with this deployment model was that it did not provide any real rollback capability nor did it offer any versioning of the objects in the content store. The absence of versioning objects in the content store also put DaVita at high risk of losing a large amount of work in the Dev environment. The DaVita BI operations team wanted to improve and automate some of their Cognos-related work processes. They wanted to reduce risk and have the ability to roll back to previous versions of BI content if needed. They also wanted to safely transfer deployment responsibilities from one person to multiple people so that developers could reduce their cycle time.

How MotioCI Saved DaVita’s Content Store

Four months after DaVita installed MotioCI, their Cognos implementation needed to be rebooted as required when services are renewed. When they attempted to reboot Cognos, nothing happened, it would not come back up. The version control capabilities of MotioCI were used to pinpoint the cause of the reboot failure and restore the content store database. In performing a root cause analysis, Motio and DaVita discovered that DaVita’s Cognos Content Store got into an unstable state because of a “perfect storm.” The combination of events which led to the unusable content store was the innocent actions of one user and an esoteric bug in a specific version of Cognos, which has since been corrected. In Cognos 10.1.1, it was possible to create a folder, say “Folder A” in Public Folders, cut it, navigate into “Folder A” and paste it there. In essence you’re moving a folder under itself. A Cognos error CMREQ4297 was logged, but the issue could not be corrected from within Cognos Connection. It got worse. When the Cognos service was recycled, it would not restart. Cognos displayed this message: “CMSYS5230 Content Manager found circular CMIDs internally. The circular CMIDs are {xxxxxx}. These bad child-parent CMIDs are causing Content Manager to malfunction.” They were stuck in that state. The Motio support team was able to walk DaVita through the process of recovering the corrupted reports and packages.