Coverage Reporting in Angular

Angular: The Full Gamut

Charlie Greenman
September 14, 2020
7 min read

Anecdote on Coverage Reporting

Coverage reporting for an Angular application using the Angular CLI is objectively easy to implement in this day of age. Albeit easy to generate coverage reporting, it’s interesting, because as a software engineer, who develops exclusively using TDD/BDD, coverage reporting becomes of less importance. I find myself spontaneously checking up on coverage reporting from time to time (usually every week), to make sure the team is hitting the numbers we want.


Testivus On Test Coverage

There is a funny, yet honestly, largely impactful founding post by a one Alberto Savioa. It is entitled, “Testivus On Test Coverage”. I would like to post the entire entry here because it is pervasive in unit test coverage:

*“I am ready to write some unit tests. What code coverage should I aim for?”

The great master replied:

“Don’t worry about coverage, just write some good tests.” The programmer smiled, bowed, and left. … Later that day, a second programmer asked the same question.

The great master pointed at a pot of boiling water and said:

“How many grains of rice should put in that pot?”

The programmer, looking puzzled, replied:

“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”

“Exactly,” said the great master.

The second programmer smiled, bowed, and left.

Toward the end of the day, a third programmer came and asked the same question about code coverage. “Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table. The third programmer smiled, bowed, and left.

After this last reply, a young apprentice approached the great master: “Great master, today I overheard you answer the same question about code coverage with three different answers. Why?” The great master stood up from his chair:

“Come get some fresh tea with me and let’s talk about it.”

After they filled their cups with smoking hot green tea, the great master began to answer:

“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.”

“The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”

“I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?”

The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.

“The third programmer wants only simple answers — even when there are no simple answers … and then does not follow them anyway.” The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.*


The Impact of Testivus

Testivus’s impact on unit testing cannot be overrated. At most organizations that I have worked at, 80% is indeed the golden number. The question is, what is the validity of the number 80%? First, and foremost, I think that culturally 80% is intuitively considered above average. Particularly, because 70% from an academic perspective is considered as passing by many.

However, I would also like to assess the number 80%cobjectively. 80% seems to coincide with the Pareto principle. The Alfred Pareto principle, if not already familiar, claims that 80 percent of consequences, come from 20% of the causes. In a super interesting article, Microsoft learned that, “…that 80 percent of the errors and crashes in Windows and Office are caused by 20 percent of the entire pool of bugs detected”.

So, you might ask why don’t we keep the rule of 20% to the most important features. Well this might just be an impossible task. For a software engineer to be aware of potential bugs before they happen, is once again, impossible. doInstead, what we should do is focus on 80%. This means that we have a 4/5 chance of catching a 20% bug.


The Phenomenon of the Pareto Rule

It is also important to realize, that within the Pareto rule, there is a 20% within the 80%. That is, the difficulty of completing a meaningful test(i.e. one that solves 80% of uses cases)increases.

The interesting thing about the Pareto rule, is that one can make the argument, that the 20% features that make an 80% impact, are those that are hardest to test. So arguably, those tend to overlooked. However, similar to what Testivus said, if we want a blanket rule, let’s apply the Pareto Priority Index(PPI):

PPI = savings × probability of success / cost × time of completion


Different Types of Coverage Reporting

The following are the four different types of coverage reporting:

  1. Statements — Has each statement in the program been executed?
  2. Branches — Has each branch of each control structure been executed? For instance, have all case statements, or if/else statements been called
  3. Functions — Has each function (or subroutine) in the program been called?
  4. Lines — has each executable line in the source file been executed?

Distinguish Between Statements + Branches

I generally find it difficult to distinguish between a statement, and a branch. The following is a great example. Let’s say we have the following scenario:

determineStatement(a: any, b: any): boolean {
  if(a){
     if(b){
       statement2 = true;
     }
     statement1 = true
  }
}

If we test a = true, and that b = true, that will give us 100% statement coverage, as we have tested all statements. However, we have not tested the hidden else statements. Therefore while the statement coverage within our function is 100%, the branch coverage is only at 50%.


Understanding Letters on Side of Code Coverage

  1. “E” stand for, “else path not taken”. Meaning that if path has been test, but not the else path.
  2. “I” stand for, “if path not taken”.
  3. “xN” in left column is the amount of times the line has been executed.

Understanding Colors

  1. Orange: Functions not covered.
  2. Pink: Statements not covered.
  3. Red: Non executed lines, or pieces of code.
  4. Yellow: Branches not covered.

Code Coverage Enforcement

In line with our decided 80% coverage, we can go into our karma.conf.js file and add the following in the coverageIstanbulReport: key.

coverageIstanbulReporter: {
 reports: [ ‘html’, ‘lcovonly’ ],
 fixWebpackSourcePaths: true,
 thresholds: {
   statements: 80,
   lines: 80,
   branches: 80,
   functions: 80
 }
}

Now we have made it that our unit testing will fail whenever we reach a threshold below 80%. This is useful for development, and useful for devops, as we can keep a consistent way of keeping our unit tests at a certain level.


Running a Coverage Report in Angular

My preferred method is to add a package.json script for running a --code-coverage report. It would look something like:

{
  ...,
  test-coverage: ng test --code-coverage
}

and then in your terminal run:

npm run test test-coverage

Walla! Just like that, the CLI will take care of emitting code-coverage for you.


Opening up Coverage Reporting

To open up a coverage report, after a coverage report has been generated, navigate to your coverage/src folder, and click on the index.html file.

I’ve found that a script that’s available to automatically open up the coverage reporting index.html has made me more prone to paying attention to coverage reporting. However, as time has gone on, and I realize the value of TDD, I think that coverage reporting is less relevant. At this point in time, I feel it is more relevant as a DevOps tool and a way of making sure that unit testing stays at a certain caliber.


Trap of Coverage Reporting

In a perfect world, all functions would be pure, and they would solve one use case. However, that is frequently not the case. In addition, even if it was the case, a unit test should still be tested against numerous use cases. So Coverage reporting while on paper makes seems like it is 80% it might be 10% to 20% less than that.

I hope you enjoyed!

More articles similar to this

footer

Razroo is committed towards contributing to open source. Take the pledge towards open source by tweeting, #itaketherazroopledge to @_Razroo on twitter. One of our associates will get back to you and set you up with an open source project to work on.