Add markdown output support

[beck3905]

Summary

I understand that pytest-cov uses coverage.py under the hood and coverage.py supports --format=markdown as an output option. Could this be added as an output format in pytest-cov?

This would be helpful adding the test report to a GitHub Actions Job Summary, which requires markdown format.

[nedbat]

You can run coverage report --format=markdown as a second command and get the output without changing pytest-cov at all.

[alelom]

We definitely could use this feature -- getting a markdown output would make it fast to display the coverage report from github, as a fast result from a github action. At the moment, we're forced to publish the html to gihub pages, but that's more complex

[nedbat]

You already have this feature today. Instead of adding an option to pytest-cov, you simply have to add one line to your GitHub action workflow.

[marcosboger]

@nedbat could I try implementing this as a first issue? I've started using pytest-cov recently and willing to collaborate as possible. If any specific reason not to implement, let me know 🙂

[webknjaz]

I think there's no harm in having this setting. Though, it's most useful within GHA when appending to the output file, not rewriting it. So --cov-report markdown:"${GITHUB_STEP_SUMMARY}" doesn't feel enough — it's unclear how one could specify that the file in the env var should be open in append mode?

[nedbat]

@marcosboger I'm not a maintainer here so it's not my decision. And as I mentioned in my comment, I'm generally not in favor of jamming more features into this tool when the same thing can already be done in a simple way without adding maintenance burden here.

[marcosboger]

@webknjaz I believe in that case, the user should probably leverage Multiline Markdown content capturing the output from the command and >> that to $GITHUB_STEP_SUMMARY. But that should be done directly by the user configuring this in GHA, so I think no different behavior from pytest-cov needed. What do you think?

[beck3905]

I was able to accomplish what I needed using the following composite action that leverages some other community actions:

runs:
  using: composite
  steps:
    - name: Run Pytest with Coverage
      shell: bash
      run: |
        poetry run pytest --junitxml=test-report/test-report.xml --cov=. --cov-report=html:coverage/htmlcov --cov-report=xml:coverage/coverage.xml --cov-report=term-missing

    - name: Publish Test Report
      uses: ctrf-io/github-test-reporter@v1
      if: always()
      with:
        report-path: test-report/test-report.xml
        github-report: true
        integrations-config: '{"junit-to-ctrf": {"enabled": true, "action": "convert"}}'

    - name: Build Pytest Report
      id: test-report
      if: always()
      uses: MishaKav/pytest-coverage-comment@main
      with:
        pytest-xml-coverage-path: coverage/coverage.xml
        junitxml-path: test-report/test-report.xml
        hide-comment: true

    - if: always()
      shell: bash
      run: |
        echo "---" >> $GITHUB_STEP_SUMMARY
        echo "# Coverage Report" >> $GITHUB_STEP_SUMMARY
        echo -e ${{ steps.test-report.outputs.summaryReport }} >> $GITHUB_STEP_SUMMARY

[webknjaz]

@webknjaz I believe in that case, the user should probably leverage Multiline Markdown content capturing the output from the command and >> that to $GITHUB_STEP_SUMMARY. But that should be done directly by the user configuring this in GHA, so I think no different behavior from pytest-cov needed. What do you think?

pytest is rarely invoked directly. In all my workflows, it's wrapped with tox. The interception tricks are rather inconvenient. I'm trying to output any $GITHUB_STEP_SUMMARY content from within tox so it doesn't leak elsewhere. But it's possible to pass CLI args to pytest from GHA which would enable --cov-report markdown:"${GITHUB_STEP_SUMMARY}" on that level, except that it'd break anything added into that file when overwriting. Hence the need for an appending mechanism.

[ionelmc]

I think there's no harm in having this setting. Though, it's most useful within GHA when appending to the output file, not rewriting it. So --cov-report markdown:"${GITHUB_STEP_SUMMARY}" doesn't feel enough — it's unclear how one could specify that the file in the env var should be open in append mode?

I am not against such a feature. Perhaps have --cov-report=markdown:path and --cov-report=markdown-append:path would solve your append issue?

[webknjaz]

Yeah, I had something like this in mind (with a single-letter suffix, though). Just wanted to see what others would come up with..

[nedbat]

I guess I'm really missing something. Why invent strange ways to append from within the pytest command-line? There's already a command you can run that generates the markdown, and you can use it to append to the summary.

@webknjaz said:

pytest is rarely invoked directly. In all my workflows, it's wrapped with tox.

Right: there's no need for everything to be jammed into one command. It's already part of a multi-step process. Keep it simple.