Post: Is Sensitive Data Secure at Your Organization? PII & PHI Compliance Testing

If your organization regularly handles sensitive data, you must implement data security compliance strategies to protect not only the individuals that the data belongs to but also your organization from violating any federal laws (e.g. HIPPA, GDPR, etc). This affects organizations in industries such as healthcare, banking, government, legal…really any organization who handles sensitive data.

We are talking about PII (Personally Identifiable Information) and PHI (Protected Health Information)Examples of PII

  • Social security numbers
  • Bank accounts
  • Full names
  • Passport numbers, etc.

Examples of PHI

  • Health records
  • Lab results
  • Medical bills and the like, that include individual identifiers

Methods for Protecting Sensitive Data

Some customers have described their methods as scenes you might imagine in some movie you’ve watched…visualize a group of folks armed with their required security clearances being huddled in a locked room, without windows, to manually check report printouts to ensure that sensitive information is not included. While this makes for a dramatic movie scene, it’s not the most foolproof nor the most efficient way to test reports for sensitive information. And with remote Covid-19 workforce requirements, this is simply not doable at this time.

We’ve helped several of our customers implement the power of automated testing to dynamically test their Cognos report outputs. This testing strategy catches the reports early, as soon as they fall out of compliance, and before they end up in production to wind up in the wrong hands. It’s always a good idea to know where the nearest social security office to you is, like the Social Security offices in Nevada, should the worst should happen, as the team at your local office will know how to manage the situation.

Value of Testing Early in Development Cycles

Detecting data security vulnerabilities early in the development stage can help avoid any future government imposed fines and sanctions. According to the U.S. Department of Health and Human Services, to date, the Office for Civil Rights (OCR) “has settled or imposed a civil money penalty in 75 cases resulting in a total dollar amount of $116,303,582.00.” That’s over $1.5M per case! And according to the HIPAA Journal the “failure to perform an organization-wide risk analysis is one of the most common HIPAA violations to result in a financial penalty.”

Aside from avoiding government imposed penalties, it is generally important to detect errors early on in the development cycle since this is the stage where issues are much easier and cheaper to fix. As a result, the main goal of this exercise is to use MotioCI’s power of regression testing to easily identify such mistakes and therefore prevent them early on in the development cycle.

Let’s take a look at how to set up testing. We’ll start with setting up our Cognos environment and then we explain how to set up automated testing for PHI and PII data for our example. We will also be using these same test cases in the production environment for an added level of compliance and security checking.

PHI & PII Cognos Environment Set Up

Our sample Cognos environment (Figure 1) consists of several reports, each containing a mix of PII and PHI sensitive data (eg. diagnosis code, prescription, social security number, patient last name and etc.) and minimally sensitive data (eg. patient first name, date of visit, and etc.).

Sample IBM Cognos Analytics Environment

Figure 1: Our sample Cognos environment.

There are two Cognos roles, AllowPII and AllowPHI, that determine whether any of the sensitive data are rendered when reports are executed. (Table 1)

Cognos Roles



Members of this role are able to view all PII (ie. social security number, and patient last name) data in Cognos reports.


Members of this role are able to view all PHI (eg. ICD10 diagnosis codes, detailed diagnosis description, and etc.) data in Cognos reports.

Table 1: Cognos roles controlling the rendition of sensitive data.

For example, a user that lacks both of our Cognos roles, their “Patient Daily Intake” report should look like this (Figure 2):

PII, PHI, Cognos Roles

Figure 2: Report output produced by a user who lacks both AllowPII and AllowPHI roles.

As you can see, all PHI and PII data are fully obfuscated from the user lacking membership in both “AllowPHI/PII” roles.

Now, let’s run the report with a user that is a member of the “AllowPII” role, meaning we expect this user to be able to only view PII data (Figure 3):

Cognos Report Output, PII, PHI

Figure 3: Report output produced by a user who is a member of the AllowPII role and NOT the AllowPHI role.

And you can see here that both the Social Security Number and Last Name columns are showing up appropriately without any redaction.

So far we’ve taken a glimpse at our mythical clinic’s Cognos environment and all we’ve seen so far is Cognos role-based data security that many of you might already have implemented in your own Cognos environments. This would then bring us to the main question that bearers of sensitive data hope would never have to face:

What if, say after some heavy development effort, some sensitive data slips through and starts showing up for users that aren’t supposed to see it?

Mistakes are certainly unavoidable, so later on in the blog we will use MotioCI’s power of regression testing to keep a watchful eye on our reports to make sure private data never gets exposed to the unintended audience.

Understanding Compliance Testing for Cognos

As mentioned in the previous section, simple mistakes in report authoring or modeling could induce unwanted behavior in the output of the reports in your Cognos environment. And if these changes are uncaught, they have the potential to sneak their way into your Production environment. What would be even more disastrous is that if these unwanted changes include exposure of private data to an unintended audience.

For example, a user without being a member of either AllowPII or AllowPHI Cognos roles is not supposed to see either PII or PHI private data in our sample Cognos environment. However, as you can see below (Figure 4), a simple change in the FM model has caused the diagnosis description and patient SSN number to be exposed to such a user, which is a HUGE violation of the federal HIPAA Security Rule.

PII and PHI role membership, HIPAA

Figure 4: A user without AllowPII and AllowPHI role membership is somehow exposed to HIPAA sensitive data.

Before moving things to MotioCI, we will first create three test users in our Cognos environment and assign them to our two roles in the following manner (Table 2):

Users Role Membership Notes
TestUserA AllowPII All PHI data must be hidden from this user
TestUserB AllowPHI All PII data must be hidden from this user
TestUserC None User is expected to NOT see either PHI or PII

Table 2: Testing Cognos user accounts with their assigned roles.

These test user accounts will later be utilized in MotioCI for regression testing of our reports that contain sensitive PII and PHI data. Our test results will depend on visibility of sensitive data to each user according to their role membership.

Now that we have set up our test users, we’re ready to configure our regression testing in MotioCI.

MotioCI Environment Setup

Our sample environment consists of Development, UAT, and Production Cognos instances. Even though MotioCI allows us to login to all three at the same time, we will begin our setup of regression testing in the Development environment in three different phases.

MotioCI login screen

Figure 5: MotioCI login screen.

MotioCI home screen showing Cognos instances

Figure 6: MotioCI home screen, showing the Cognos instances.

With regard to regression testing in MotioCI, an assertion is an individual check or “test” that a test case performs on an object in your MotioCI instance, such as a report, folder, or package. The assertion that will be doing the job of testing the report outputs for sensitive data is called Sensitive Data Compliance Testing (Figure 7). This is a custom assertion that we have put together for this exercise. Below you can see the assertion type which basically acts as the main template that is copied to test cases throughout our MotioCI environment. More on this later.

sensitive data compliance testing assertion type

Figure 7: “Sensitive Data Compliance Testing” assertion type. Copies of this assertion are deployed to the testing environment.

Some assertions provide for some user adjustable functionality through a prompt window. Here you could change how you would like a given assertion to test any given Cognos report. Figure 8 below shows the prompt window of our assertion that we will be using for testing our Cognos reports containing sensitive data.

sensitive data compliance testing assertion type prompt window

Figure 8: Prompt window of the “Sensitive Data Compliance Testing” assertion, revealing all the user adjustable testing options.

The top highlighted section in Figure 8 shows the testing options for PII and PHI sensitive data. This allows you to have the assertion test whether the report must either show or hide its PII or PHI data. We will be making changes to these two options as we start creating test cases for each of our three test users.

The middle highlighted section in Figure 8 shows the names of the columns that contain PHI sensitive data in our reports. Although our sample environment consists of columns by the names of ICD10 Diag Code, Diagnosis Description, Procedure, and Rx, you could certainly modify this list to suit your needs.

Finally, the bottom highlighted section in Figure 8 shows the email options. In the case of a failure, this assertion will send a detailed email message to the recipient configured in this section.

Phase I: Reports Displaying PII Only

Let’s create a project under the Development instance in MotioCI and call it Allow PII Only. We can do that by first right-clicking on the Development instance node in the MotioCI navigation tree and selecting the Add Project option (Figure 9).

create a new project in MotioCI

Figure 9: Creating a new project. In MotioCI each project acts as a testing ground for a predefined section of the content store.

The Add Project Wizard will take you through some steps to select the paths needed for your project. In our example, all the reports containing PII and PHI sensitive data exist under the Patient Data folder. Checking this parent folder will automatically include all the underlying reports (Figures 10 & 11).

selecting paths from Cognos environment in MotioCI

Figure 10: Determining the scope of the project in MotioCI by selecting the paths from the Cognos environment.

showing all Cognos objects selected in MotioCI project

Figure 11: Showing all the Cognos objects selected for the MotioCI project.

Since all the reports in this project are expected to allow all PII data to be displayed and all PHI to be obfuscated, we will need to configure our assertion type with the correct settings before adding any test cases (Figure 12). That means setting the two testing options on the same assertion prompt window that we saw before in Figure 8.

PII and PHI testing options of the Sensitive Data Compliance Testing assertion.

Figure 12: PII and PHI testing options of the “Sensitive Data Compliance Testing” assertion.

Now we’re ready to add our test cases to our reports. To do that, right click on the project node (ie. the Allow PII Only project) in MotioCI and select the Generate Test Cases option (Figure 13). This will start the generate test case wizard which will allow us to create a large number of test cases for all the reports within the project.

MotioCI generate test cases screen

Figure 13: MotioCI can automatically generate all the necessary test cases at any level from within the project.

The Generate Test Case wizard will also allow us to select the output formats for the test case that we would like to perform tests on. For our sample environment I chose the CSV output. The wizard will also let us pick the assertions that each test case will utilize for the actual job of testing. And for us that would be the Sensitive Data Compliance Testing assertion. You can see both of these options highlighted below (Figure 14).

generate test cases options wizard

Figure 14: The options revealed during the “Generate Test Cases” wizard.

After clicking “OK” you will be taken back to the MotioCI home screen, where you will be able to see all of our reports each containing a single test case and each containing our single assertion (Figure 15).

MotioCI navigation tree showing all Cognos objects

Figure 15: MotioCI navigation tree showing all the Cognos objects now each containing a test case and the underlying assertion.

Finally, we need to configure all of the test cases to execute their parent reports using the correct Cognos user (e.g. one of the three test users that we configured in Cognos before setting things up in MotioCI). And since for this project we are testing to ensure that PHI content is not displayed to users who are only permitted to view PII data, we will need to set all the test cases to run with TestUserA (see table 2).

At first this might sound like a tedious task, but lucky for us we can set the user at the project level which would then be inherited by ALL the underlying test cases within that project. To do that, on the left navigation tree, we’re going to click on the project node ( Allow PII Only project) and then select the Project Settings in the middle of the screen. Then, under the Testing section, we will see an option to change the credentials (Figure 16):

Setting the user credentials on a project will cause all the test cases to execute the parent Cognos report in Cognos with that user

Figure 16: Setting the user credentials on a project will cause all the test cases to execute the parent Cognos report in Cognos with that user. This can be overwritten by each individual test case.

After clicking on the Edit button located in front of the Credentials option, we will be presented with the Edit Credentials window. We will go ahead and enter the credentials for TestUserA (Figure 17).

edit credentials window MotioCI

Figure 17: The “Edit Credentials” window allows you to set a new user credentials, or utilize the parent credentials set at the Cognos instance level, also known as the system credentials.

We now see the new user reflected in the Testing section of the Project Settings tab (Figure 18).

new user credentials MotioCI

Figure 18: The new user credentials are now set on the project.

Now we’re all set and ready to execute all of our test cases.

To do that, we will click on the Allow PII Only project and in the middle we will be presented with the Test Cases tab which displays all of the test cases located within the project. Since we still haven’t executed anything we would see the Status showing as No Results. To execute all the test cases, we will click on the tiny arrow by the Run button and select the Run All option (Figure 19).

Select Run All to execute the MotioCI test cases

Figure 19: The “Test Cases” tab provides a number of actions that could be performed on all or part of the test cases in bulk. Here we are just executing all the test cases.

MotioCI will now execute all the test cases and present us with the results when they’re all done (Figure 20).

he Test Cases tab displays the execution status of each test case including outputs

Figure 20: The “Test Cases” tab displays the execution status of each test case including outputs, if any.

As you can see, all of our test cases succeeded with the exception of the Inpatient report. So, let’s take a look at the results. To do that we will click on the blue timestamp located under the Result column and look at the details in Figure 21.

MotioCi test case result panel

Figure 21: The “Test Case Result” panel shows the detailed results of the execution of the test case including the path of the tested object, assertion results, and any outputs produced by the report.

Under the Assertion Results section we can now see that our report is in the violation of the PHI compliance requirements. We can download the CSV report output from the Test Case Outputs section by clicking on the CSV icon (Figure 21).

CSV Report Output

Figure 22: The CSV report output showing a displayed “Procedure” column that must have been obfuscated for the test user.

As you can see in our report (Figure 22), in addition to the PII data which is okay for TestUserA to have access to, we are able to see the PHI procedure data which puts the report in violation of the federal HIPAA Security Rule.

If you remember from the assertion settings window, we were also supposed to receive an email notification for this failure. Let’s see what that looks like (Figure 23):

Email message sent by the assertion of the failed test case

Figure 23: Email message sent by the assertion of the failed test case, showing a breach of sensitive data compliance, probably due to a recent change in the report.

At this point we’re finished testing to make sure PHI data is hidden from users without the required AllowPHI Cognos role. Now we’re ready to extend our testing to PII data being hidden from users lacking the required AllowPII Cognos role.

Phase II: Reports Displaying PHI Only

Before creating the new project, first let’s edit our master assertion’s options to ensure that it now tests for all PII to be hidden and all PHI to be shown (Figure 24).

PII and PHI testing options of the “Sensitive Data Compliance Testing” assertion being set for TestUserB

Figure 24: PII and PHI testing options of the “Sensitive Data Compliance Testing” assertion being set for TestUserB.

With our assertion now all configured, we’re now ready to create the new project and our test cases. For that we will just follow the same steps as in “Phase I” and create a project called Allow PHI Only. Also, let’s not forget to add the credentials of TestUserB as the project user.

When we’re done with all the configuration steps, we will execute all the test cases like we did in Phase I. In our sample environment, this time we have a different report that seems to be in HIPAA violation (Figure 25).

Test Cases tab displaying the execution status of each test case including outputs

Figure 25: The “Test Cases” tab displaying the execution status of each test case including outputs, if any.

Further investigation into the test case results of the Patient Daily Intake report shows that our report is displaying patient social security numbers to the unintended audience (Figure 26).

test case result showing a violation of the SSN PII compliance requirement

Figure 26: The test case result showing a violation of the SSN PII compliance requirement.

Downloading and opening the CSV file will further confirm the results of our test (Figure 27):

CSV output

Figure 27: The CSV output shows the revealed patient SSN where it should’ve been obfuscated.

As you can see in Figure 27, however, our report is properly masking the patient last name column (also a PII) by displaying only the initial.


Repeat the same steps for TestUserC which lacks both the AllowPII and AllowPHI roles, meaning that they’re not supposed to be seeing either PII or PHI data when they execute any of our reports.

By this point our environment should have achieved full regression testing of both PHI and PII sensitive data utilizing Cognos’ role based data security. Our test cases will each execute their parent report and analyze the output according to the testing configuration set within their underlying assertions and tell us if any of the reports fall out of line.

Certainly one of the most important distinctions between our test environment and what you might have in your environment is size. A typical Cognos environment most likely has an upwards of hundreds or even thousands of reports and executing those all at the same time, as we have been doing in our tiny sample environment, could take a toll on Cognos’ performance. With MotioCI’s test scripts, however, you can schedule your test cases to run in smaller batches during off hours, therefore ensuring optimal performance of your Cognos environment during the high traffic hours.

A Good testing practice during development

In between the scheduled run times however, you could still manually run as many individual test cases as you wish. A good example would be while developing a report, you could run the test case to make sure your changes haven’t created any HIPAA violations.

Automating Cognos Test Cases

Back to MotioCI, on the navigation tree, we expand one of the projects that we created to reveal its content. This should reveal a node called Test Scripts. Expanding it will show a set of test scripts that were automatically created when you first created your project (Figure 28).

test scripts

Figure 28: Test scripts can be created to display only a limited number of test cases matching a certain criteria set by the administrator user.

By definition, a test script is a component of a project that selects test cases that belong to a project based on specified criteria. You can schedule test scripts or run them manually. When you run a test script, MotioCI runs all test cases that adhere to the script criteria.

In our case we would like to set all the test cases on a schedule. So in order to do that we click on the All test script from the navigation tree and then click on the Test Script Settings tab found in the middle of the screen (Figure 29).

MotioCI test script settings tab

Figure 29: The “Test Script Settings” tab allows you to add a schedule for all the test cases.

Next, we select the Add Schedule option. Here we can now set a schedule for our test script. I will go ahead and have our test cases run daily Monday through Friday at 3:00am (Figure 30).

MotioCI test script schedule

Figure 30: In addition to the daily and weekly schedule, you could also set a minute frequency on a schedule.

That’s it! We can now check our email inbox every morning to find out if any of our reports have fallen out of compliance. We can also see all of the failed reports by simply clicking on the Changed or Failed test script and all the failed test cases will be presented to us under the Test Cases panel (Figure 31).

MotioCI changed or failed test script

Figure 31: The included “Changed or Failed” test script showing the single test case that has failed in the latest test case batch run.


Falling out of compliance with HIPPA, GDPR, and other federal regulations around sensitive information and privacy can be quite costly, around $1.5M per case found in violation, in fact.

By implementing an automated testing strategy to handle the compliance testing, you will have that extra layer of security as well as peace of mind that you are adhering to the laws. Beyond privacy data mandates, automated testing can benefit all types of industries and any sort of testing requirements your organization would like to put in place.

How Can We Help?

If you would like to watch the webinar about this blog topic, access it here. Or, contact us to discuss your Cognos testing questions further.

Scroll to Top
As the BI space evolves, organizations must take into account the bottom line of amassing analytics assets.
The more assets you have, the greater the cost to your business. There are the hard costs of keeping redundant assets, i.e., cloud or server capacity. Accumulating multiple versions of the same visualization not only takes up space, but BI vendors are moving to capacity pricing. Companies now pay more if you have more dashboards, apps, and reports. Earlier, we spoke about dependencies. Keeping redundant assets increases the number of dependencies and therefore the complexity. This comes with a price tag.
The implications of asset failures differ, and the business’s repercussions can be minimal or drastic.
Different industries have distinct regulatory requirements to meet. The impact may be minimal if a report for an end-of-year close has a mislabeled column that the sales or marketing department uses, On the other hand, if a healthcare or financial report does not meet the needs of a HIPPA or SOX compliance report, the company and its C-level suite may face severe penalties and reputational damage. Another example is a report that is shared externally. During an update of the report specs, the low-level security was incorrectly applied, which caused people to have access to personal information.
The complexity of assets influences their likelihood of encountering issues.
The last thing a business wants is for a report or app to fail at a crucial moment. If you know the report is complex and has a lot of dependencies, then the probability of failure caused by IT changes is high. That means a change request should be taken into account. Dependency graphs become important. If it is a straightforward sales report that tells notes by salesperson by account, any changes made do not have the same impact on the report, even if it fails. BI operations should treat these reports differently during change.
Not all reports and dashboards fail the same; some reports may lag, definitions might change, or data accuracy and relevance could wane. Understanding these variations aids in better risk anticipation.

Marketing uses several reports for its campaigns – standard analytic assets often delivered through marketing tools. Finance has very complex reports converted from Excel to BI tools while incorporating different consolidation rules. The marketing reports have a different failure mode than the financial reports. They, therefore, need to be managed differently.

It’s time for the company’s monthly business review. The marketing department proceeds to report on leads acquired per salesperson. Unfortunately, half the team has left the organization, and the data fails to load accurately. While this is an inconvenience for the marketing group, it isn’t detrimental to the business. However, a failure in financial reporting for a human resource consulting firm with 1000s contractors that contains critical and complex calculations about sickness, fees, hours, etc, has major implications and needs to be managed differently.

Acknowledging that assets transition through distinct phases allows for effective management decisions at each stage. As new visualizations are released, the information leads to broad use and adoption.
Think back to the start of the pandemic. COVID dashboards were quickly put together and released to the business, showing pertinent information: how the virus spreads, demographics affected the business and risks, etc. At the time, it was relevant and served its purpose. As we moved past the pandemic, COVID-specific information became obsolete, and reporting is integrated into regular HR reporting.
Reports and dashboards are crafted to deliver valuable insights for stakeholders. Over time, though, the worth of assets changes.
When a company opens its first store in a certain area, there are many elements it needs to understand – other stores in the area, traffic patterns, pricing of products, what products to sell, etc. Once the store is operational for some time, specifics are not as important, and it can adopt the standard reporting. The tailor-made analytic assets become irrelevant and no longer add value to the store manager.