Is Sensitive Data Secure at Your Organization? Stay Safe with PII & PHI Compliance Testing

by Jan 7, 2020Blog, Cognos Analytics, MotioCI0 comments

If your organization regularly handles sensitive data, you must implement data security compliance strategies to protect not only the individuals that the data belongs to but also your organization from violating any federal laws (e.g. HIPPA, GDPR, etc). This affects organizations in industries such as healthcare, banking, government, legal…really any organization who handles sensitive data.

We are talking about PII (Personally Identifiable Information) and PHI (Protected Health Information)Examples of PII

  • Social security numbers
  • Bank accounts
  • Full names
  • Passport numbers, etc.

Examples of PHI

  • Health records
  • Lab results
  • Medical bills and the like, that include individual identifiers

Methods for Protecting Sensitive Data

Some customers have described their methods as scenes you might imagine in some movie you’ve watched…visualize a group of folks armed with their required security clearances being huddled in a locked room, without windows, to manually check report printouts to ensure that sensitive information is not included. While this makes for a dramatic movie scene, it’s not the most foolproof nor the most efficient way to test reports for sensitive information. And with remote Covid-19 workforce requirements, this is simply not doable at this time.

We’ve helped several of our customers implement the power of automated testing to dynamically test their Cognos report outputs. This testing strategy catches the reports early, as soon as they fall out of compliance, and before they end up in production to wind up in the wrong hands. It’s always a good idea to know where the nearest social security office to you is, like the Social Security offices in Nevada, should the worst should happen, as the team at your local office will know how to manage the situation.

Value of Testing Early in Development Cycles

Detecting data security vulnerabilities early in the development stage can help avoid any future government imposed fines and sanctions. According to the U.S. Department of Health and Human Services, to date, the Office for Civil Rights (OCR) “has settled or imposed a civil money penalty in 75 cases resulting in a total dollar amount of $116,303,582.00.” That’s over $1.5M per case! And according to the HIPAA Journal the “failure to perform an organization-wide risk analysis is one of the most common HIPAA violations to result in a financial penalty.”

Aside from avoiding government imposed penalties, it is generally important to detect errors early on in the development cycle since this is the stage where issues are much easier and cheaper to fix. As a result, the main goal of this exercise is to use MotioCI’s power of regression testing to easily identify such mistakes and therefore prevent them early on in the development cycle.

Let’s take a look at how to set up testing. We’ll start with setting up our Cognos environment and then we explain how to set up automated testing for PHI and PII data for our example. We will also be using these same test cases in the production environment for an added level of compliance and security checking.

PHI & PII Cognos Environment Set Up

Our sample Cognos environment (Figure 1) consists of several reports, each containing a mix of PII and PHI sensitive data (eg. diagnosis code, prescription, social security number, patient last name and etc.) and minimally sensitive data (eg. patient first name, date of visit, and etc.).

Cognos Analytics Sample Patient Data Folder

Figure 1: Our sample Cognos environment.

There are two Cognos roles, AllowPII and AllowPHI, that determine whether any of the sensitive data are rendered when reports are executed. (Table 1)

Cognos Roles



Members of this role are able to view all PII (ie. social security number, and patient last name) data in Cognos reports.


Members of this role are able to view all PHI (eg. ICD10 diagnosis codes, detailed diagnosis description, and etc.) data in Cognos reports.

Table 1: Cognos roles controlling the rendition of sensitive data.

For example, a user that lacks both of our Cognos roles, their “Patient Daily Intake” report should look like this (Figure 2):

Cognos Report Patient Intake Sample

Figure 2: Report output produced by a user who lacks both AllowPII and AllowPHI roles.

As you can see, all PHI and PII data are fully obfuscated from the user lacking membership in both “AllowPHI/PII” roles.

Now, let’s run the report with a user that is a member of the “AllowPII” role, meaning we expect this user to be able to only view PII data (Figure 3):

Cognos Report AllowPII Role

Figure 3: Report output produced by a user who is a member of the AllowPII role and NOT the AllowPHI role.

And you can see here that both the Social Security Number and Last Name columns are showing up appropriately without any redaction.

So far we’ve taken a glimpse at our mythical clinic’s Cognos environment and all we’ve seen so far is Cognos role-based data security that many of you might already have implemented in your own Cognos environments. This would then bring us to the main question that bearers of sensitive data hope would never have to face:

What if, say after some heavy development effort, some sensitive data slips through and starts showing up for users that aren’t supposed to see it?

Mistakes are certainly unavoidable, so later on in the blog we will use MotioCI’s power of regression testing to keep a watchful eye on our reports to make sure private data never gets exposed to the unintended audience.

Understanding Compliance Testing for Cognos

As mentioned in the previous section, simple mistakes in report authoring or modeling could induce unwanted behavior in the output of the reports in your Cognos environment. And if these changes are uncaught, they have the potential to sneak their way into your Production environment. What would be even more disastrous is that if these unwanted changes include exposure of private data to an unintended audience.

For example, a user without being a member of either AllowPII or AllowPHI Cognos roles is not supposed to see either PII or PHI private data in our sample Cognos environment. However, as you can see below (Figure 4), a simple change in the FM model has caused the diagnosis description and patient SSN number to be exposed to such a user, which is a HUGE violation of the federal HIPAA Security Rule.

Exposed PII/PHI Data in Cognos Report

Figure 4: A user without AllowPII and AllowPHI role membership is somehow exposed to HIPAA sensitive data.

Before moving things to MotioCI, we will first create three test users in our Cognos environment and assign them to our two roles in the following manner (Table 2):

Users Role Membership Notes
TestUserA AllowPII All PHI data must be hidden from this user
TestUserB AllowPHI All PII data must be hidden from this user
TestUserC None User is expected to NOT see either PHI or PII

Table 2: Testing Cognos user accounts with their assigned roles.

These test user accounts will later be utilized in MotioCI for regression testing of our reports that contain sensitive PII and PHI data. Our test results will depend on visibility of sensitive data to each user according to their role membership.

Now that we have set up our test users, we’re ready to configure our regression testing in MotioCI.

MotioCI Environment Setup

Our sample environment consists of Development, UAT, and Production Cognos instances. Even though MotioCI allows us to login to all three at the same time, we will begin our setup of regression testing in the Development environment in three different phases.

MotioCI Login Screen

Figure 5: MotioCI login screen.

MotioCI Home Screen with Cognos instances

Figure 6: MotioCI home screen, showing the Cognos instances.

With regard to regression testing in MotioCI, an assertion is an individual check or “test” that a test case performs on an object in your MotioCI instance, such as a report, folder, or package. The assertion that will be doing the job of testing the report outputs for sensitive data is called Sensitive Data Compliance Testing (Figure 7). This is a custom assertion that we have put together for this exercise. Below you can see the assertion type which basically acts as the main template that is copied to test cases throughout our MotioCI environment. More on this later.

Compliance Data Test Assertion in MotioCI

Figure 7: “Sensitive Data Compliance Testing” assertion type. Copies of this assertion are deployed to the testing environment.

Some assertions provide for some user adjustable functionality through a prompt window. Here you could change how you would like a given assertion to test any given Cognos report. Figure 8 below shows the prompt window of our assertion that we will be using for testing our Cognos reports containing sensitive data.

User Options Window for Compliance Testing in MotioCI

Figure 8: Prompt window of the “Sensitive Data Compliance Testing” assertion, revealing all the user adjustable testing options.

The top highlighted section in Figure 8 shows the testing options for PII and PHI sensitive data. This allows you to have the assertion test whether the report must either show or hide its PII or PHI data. We will be making changes to these two options as we start creating test cases for each of our three test users.

The middle highlighted section in Figure 8 shows the names of the columns that contain PHI sensitive data in our reports. Although our sample environment consists of columns by the names of ICD10 Diag Code, Diagnosis Description, Procedure, and Rx, you could certainly modify this list to suit your needs.

Finally, the bottom highlighted section in Figure 8 shows the email options. In the case of a failure, this assertion will send a detailed email message to the recipient configured in this section.

Phase I: Reports Displaying PII Only

Let’s create a project under the Development instance in MotioCI and call it Allow PII Only. We can do that by first right-clicking on the Development instance node in the MotioCI navigation tree and selecting the Add Project option (Figure 9).

Add Testing Project Prompt in MotioCI

Figure 9: Creating a new project. In MotioCI each project acts as a testing ground for a predefined section of the content store.

The Add Project Wizard will take you through some steps to select the paths needed for your project. In our example, all the reports containing PII and PHI sensitive data exist under the Patient Data folder. Checking this parent folder will automatically include all the underlying reports (Figures 10 & 11).

Parent Folder Selects all Cognos Reports to Test

Figure 10: Determining the scope of the project in MotioCI by selecting the paths from the Cognos environment.

All Cognos Objects Associated with Parent Folder Selected for Testing

Figure 11: Showing all the Cognos objects selected for the MotioCI project.

Since all the reports in this project are expected to allow all PII data to be displayed and all PHI to be obfuscated, we will need to configure our assertion type with the correct settings before adding any test cases (Figure 12). That means setting the two testing options on the same assertion prompt window that we saw before in Figure 8.

Set Testing options for allow PII data to show and hide PHI data

Figure 12: PII and PHI testing options of the “Sensitive Data Compliance Testing” assertion.

Now we’re ready to add our test cases to our reports. To do that, right click on the project node (ie. the Allow PII Only project) in MotioCI and select the Generate Test Cases option (Figure 13). This will start the generate test case wizard which will allow us to create a large number of test cases for all the reports within the project.

Generate Test cases for your project in MotioCI

Figure 13: MotioCI can automatically generate all the necessary test cases at any level from within the project.

The Generate Test Case wizard will also allow us to select the output formats for the test case that we would like to perform tests on. For our sample environment I chose the CSV output. The wizard will also let us pick the assertions that each test case will utilize for the actual job of testing. And for us that would be the Sensitive Data Compliance Testing assertion. You can see both of these options highlighted below (Figure 14).

Select output types and assertions for MotioCI PII compliance test project

Figure 14: The options revealed during the “Generate Test Cases” wizard.

After clicking “OK” you will be taken back to the MotioCI home screen, where you will be able to see all of our reports each containing a single test case and each containing our single assertion (Figure 15).

MotioCI navigation shows Cognos objects with their test case and assertion for compliance testing

Figure 15: MotioCI navigation tree showing all the Cognos objects now each containing a test case and the underlying assertion.

Finally, we need to configure all of the test cases to execute their parent reports using the correct Cognos user (e.g. one of the three test users that we configured in Cognos before setting things up in MotioCI). And since for this project we are testing to ensure that PHI content is not displayed to users who are only permitted to view PII data, we will need to set all the test cases to run with TestUserA (see table 2).

At first this might sound like a tedious task, but lucky for us we can set the user at the project level which would then be inherited by ALL the underlying test cases within that project. To do that, on the left navigation tree, we’re going to click on the project node ( Allow PII Only project) and then select the Project Settings in the middle of the screen. Then, under the Testing section, we will see an option to change the credentials (Figure 16):

Set the user credentials for the MotioCI project settings at the parent level

Figure 16: Setting the user credentials on a project will cause all the test cases to execute the parent Cognos report in Cognos with that user. This can be overwritten by each individual test case.

After clicking on the Edit button located in front of the Credentials option, we will be presented with the Edit Credentials window. We will go ahead and enter the credentials for TestUserA (Figure 17).

Set user credentials window for the MotioCI project

Figure 17: The “Edit Credentials” window allows you to set a new user credentials, or utilize the parent credentials set at the Cognos instance level, also known as the system credentials.

We now see the new user reflected in the Testing section of the Project Settings tab (Figure 18).

New user credentials are set.

Figure 18: The new user credentials are now set on the project.

Now we’re all set and ready to execute all of our test cases.

To do that, we will click on the Allow PII Only project and in the middle we will be presented with the Test Cases tab which displays all of the test cases located within the project. Since we still haven’t executed anything we would see the Status showing as No Results. To execute all the test cases, we will click on the tiny arrow by the Run button and select the Run All option (Figure 19).

Select Run All to execute the MotioCI test cases

Figure 19: The “Test Cases” tab provides a number of actions that could be performed on all or part of the test cases in bulk. Here we are just executing all the test cases.

MotioCI will now execute all the test cases and present us with the results when they’re all done (Figure 20).

Status of MotioCI test cases after execution

Figure 20: The “Test Cases” tab displays the execution status of each test case including outputs, if any.

As you can see, all of our test cases succeeded with the exception of the Inpatient report. So, let’s take a look at the results. To do that we will click on the blue timestamp located under the Result column and look at the details in Figure 21.

Details from the MotioCI test case result

Figure 21: The “Test Case Result” panel shows the detailed results of the execution of the test case including the path of the tested object, assertion results, and any outputs produced by the report.

Under the Assertion Results section we can now see that our report is in the violation of the PHI compliance requirements. We can download the CSV report output from the Test Case Outputs section by clicking on the CSV icon (Figure 21).

compliance test failure report shows PHI procedures that should be hidden

Figure 22: The CSV report output showing a displayed “Procedure” column that must have been obfuscated for the test user.

As you can see in our report (Figure 22), in addition to the PII data which is okay for TestUserA to have access to, we are able to see the PHI procedure data which puts the report in violation of the federal HIPAA Security Rule.

If you remember from the assertion settings window, we were also supposed to receive an email notification for this failure. Let’s see what that looks like (Figure 23):

Email alert notification of Cognos compliance test failure

Figure 23: Email message sent by the assertion of the failed test case, showing a breach of sensitive data compliance, probably due to a recent change in the report.

At this point we’re finished testing to make sure PHI data is hidden from users without the required AllowPHI Cognos role. Now we’re ready to extend our testing to PII data being hidden from users lacking the required AllowPII Cognos role.

Phase II: Reports Displaying PHI Only

Before creating the new project, first let’s edit our master assertion’s options to ensure that it now tests for all PII to be hidden and all PHI to be shown (Figure 24).

Hide PII data set up for cognos compliance test

Figure 24: PII and PHI testing options of the “Sensitive Data Compliance Testing” assertion being set for TestUserB.

With our assertion now all configured, we’re now ready to create the new project and our test cases. For that we will just follow the same steps as in “Phase I” and create a project called Allow PHI Only. Also, let’s not forget to add the credentials of TestUserB as the project user.

When we’re done with all the configuration steps, we will execute all the test cases like we did in Phase I. In our sample environment, this time we have a different report that seems to be in HIPAA violation (Figure 25).

Hide PII data test results

Figure 25: The “Test Cases” tab displaying the execution status of each test case including outputs, if any.

Further investigation into the test case results of the Patient Daily Intake report shows that our report is displaying patient social security numbers to the unintended audience (Figure 26).

Failure indicates PII data is shown to unintended recipients

Figure 26: The test case result showing a violation of the SSN PII compliance requirement.

Downloading and opening the CSV file will further confirm the results of our test (Figure 27):

failure report shows PII data violations

Figure 27: The CSV output shows the revealed patient SSN where it should’ve been obfuscated.

As you can see in Figure 27, however, our report is properly masking the patient last name column (also a PII) by displaying only the initial.


Repeat the same steps for TestUserC which lacks both the AllowPII and AllowPHI roles, meaning that they’re not supposed to be seeing either PII or PHI data when they execute any of our reports.

By this point our environment should have achieved full regression testing of both PHI and PII sensitive data utilizing Cognos’ role based data security. Our test cases will each execute their parent report and analyze the output according to the testing configuration set within their underlying assertions and tell us if any of the reports fall out of line.

Certainly one of the most important distinctions between our test environment and what you might have in your environment is size. A typical Cognos environment most likely has an upwards of hundreds or even thousands of reports and executing those all at the same time, as we have been doing in our tiny sample environment, could take a toll on Cognos’ performance. With MotioCI’s test scripts, however, you can schedule your test cases to run in smaller batches during off hours, therefore ensuring optimal performance of your Cognos environment during the high traffic hours.

A Good testing practice during development

In between the scheduled run times however, you could still manually run as many individual test cases as you wish. A good example would be while developing a report, you could run the test case to make sure your changes haven’t created any HIPAA violations.

Automating Cognos Test Cases

Back to MotioCI, on the navigation tree, we expand one of the projects that we created to reveal its content. This should reveal a node called Test Scripts. Expanding it will show a set of test scripts that were automatically created when you first created your project (Figure 28).

Test scripts associates with your MotioCI test project

Figure 28: Test scripts can be created to display only a limited number of test cases matching a certain criteria set by the administrator user.

By definition, a test script is a component of a project that selects test cases that belong to a project based on specified criteria. You can schedule test scripts or run them manually. When you run a test script, MotioCI runs all test cases that adhere to the script criteria.

In our case we would like to set all the test cases on a schedule. So in order to do that we click on the All test script from the navigation tree and then click on the Test Script Settings tab found in the middle of the screen (Figure 29).

Schedule Cognos Test cases to run automatically on a specific schedule

Figure 29: The “Test Script Settings” tab allows you to add a schedule for all the test cases.

Next, we select the Add Schedule option. Here we can now set a schedule for our test script. I will go ahead and have our test cases run daily Monday through Friday at 3:00am (Figure 30).

Set your automate cognos testing schedule to run after hours

Figure 30: In addition to the daily and weekly schedule, you could also set a minute frequency on a schedule.

That’s it! We can now check our email inbox every morning to find out if any of our reports have fallen out of compliance. We can also see all of the failed reports by simply clicking on the Changed or Failed test script and all the failed test cases will be presented to us under the Test Cases panel (Figure 31).

Changed/Failed test script in MotioCI shows any failures

Figure 31: The included “Changed or Failed” test script showing the single test case that has failed in the latest test case batch run.


Falling out of compliance with HIPPA, GDPR, and other federal regulations around sensitive information and privacy can be quite costly, around $1.5M per case found in violation, in fact.

By implementing an automated testing strategy to handle the compliance testing, you will have that extra layer of security as well as peace of mind that you are adhering to the laws. Beyond privacy data mandates, automated testing can benefit all types of industries and any sort of testing requirements your organization would like to put in place.

How Can We Help?

If you would like to watch the webinar about this blog topic, access it here. Or, contact us to discuss your Cognos testing questions further.

BlogBusiness AnalyticsBusiness Intelligence Qlik
From Excel to Qlik Sense: Our Analytics Journey in the Insurance Industry

From Excel to Qlik Sense: Our Analytics Journey in the Insurance Industry

n this special guest blog post, Chirag Shukla, a CITO in the insurance industry, will be our guide for all of the adventures, discoveries, and milestones his company encountered on their analytics journey. We’ll start with Excel and end at our final destination,...

Read More