Security Testing: Everything but the pentest

When most think of a security test, it conjures images of Hollywood-style hacking. For those who work in the software industry, it still conjures images of an employee or contractor attempting to hack “for good” so that the flaws can be fixed before a bad actor finds them. I want to talk about the other stuff. The rest of the work that goes into testing for security that isn’t a tool-assisted investigation of a live (or live-like) deployment of the software.

The Application Under Test

Let’s start by describing a deployment that we’d like to test.

|     Application Container      |
|                                |
|  |--------------------------|  |
|  |       Application        |  |
|  |  |------------|          |  |
|  |  | |------------|        |  |
|  |  |-| |------------|      |  |
|  |    |-| |------------|    |  |
|  |      |-| Dependency |    |  |
|  |        |------------|    |  |
|  |                          |  |
|  |  |--------------------|  |  |
|  |  |  Application Code  |  |  |
|  |  |--------------------|  |  |
|  |--------------------------|  |
|                                |
|  |--------------------------|  |
|  |     Operating System     |  |
|  |--------------------------|  |

This is a single application container, based on an upstream base OS container plus our application under test. The application is bespoke code that exposes and API, and has dependencies. Here’s an example repository:

Knowing the Risks

Looking at the architecture above, there are a few obvious risks:

  • The application code is vulnerable
  • A dependency of the application is vulnerable
  • The operating system of the container is vulnerable

Note that this precludes the operating system of the container runner, or the version of the container runtime - those are part of the production environment, and whilst they could and should be considered before production deployment, are only truly testable in deployment.

Each of these identified risks are independently security testable and the risks addressable as pre-deployment activities. This doesn’t guarantee that a pentest won’t find problems, but it does mean it won’t find anything stupid.

Testing Application Dependencies

OWASP Dependency Check

Plenty of tooling exists to test dependencies. One of the most popular of these is OWASP Dependency-Check.

This can either be downloaded from their webpage, or installed via a package manager (e.g. brew install dependency-check).

For our example repository, we can run:

git clone [email protected]:EBTPT/spring-boot-hello-world.git
cd spring-boot-hello-world
mvn package
dependency-check -s .

This will generate a file named dependency-check-report.html which lists all of the vulnerable dependencies.

OWASP Dependency Check in Maven

Instead of relying on a local tool being downloaded or installed, you can add the dependency check to be run via maven.

Add the following to the build plugins within the POM:


Then run mvn verify to see the same page generated in the “target” directory. There are further configuration options in the Dependency-Check Documentation.

GitHub Dependabot Alerts

Rather than keep dependencies local, moving the scanning into CI has value. You don’t need active development or humans running commands. CI means that every build will include a dependency scan. In the previous step we added dependency scanning to Maven, and we could easily have added the mvn verify command as an additional step (or in lieu of an imaginary mvn package).

GitHub offers a further level of ease to this, where instead of requiring either active development or scheduled builds to enable the scan to occur, it constantly monitors dependencies in the background and offers alerts and solutions.

To switch it on, visit the Security settings of a project and enable it. To try this on the example project, you’ll need to fork it first.

Screenshot of Github Security settings

Screenshot of Github Security settings

Enable all three to get alerts and automated fixes.

Remediations for dependency problems

Except when using Dependabot in GitHub to generate PRs for updates, remediation for these is often just a case of bumping numbers in the POM or other dependency management file in your source code. The trickiest part can be finding the correct dependency to bump, since the vulnerability may be in a dependency of a dependency of a dependency.

It’s worth noting that just because a vulnerability is known, it might not always have a fix. Perhaps the fix is still in progress, or the software could be abandoned. It’s also possible that the vulnerability is “by design”, for example “eval” style tools intentionally allow execution of arbitrary strings, and it’s how they’re used that lead to whether it’s a genuine vulnerability.

Testing the Container

Given that the container image contains a lot of components that are seen in a full operating system (e.g. bash, curl), it has a lot of components, any of which could introduce its own vulnerabilities to the running application, allowing an attacker to exert some control.

Like with dependencies, there are a number of tools and methods available to scan your container for vulnerabilities.

Anchore Grype

Anchore has made a name for itself of as one of the de-facto container scanning tools. You can install and run this locally with our image like so:

    git clone [email protected]:EBTPT/spring-boot-hello-world.git
    cd spring-boot-hello-world
    mvn package
    docker build . -t spring-boot-hello-world
    brew tap anchore/grype
    brew install grype
    grype spring-boot-hello-world

This will print a table of known vulnerabilities with the components of this image.

$ grype ebtpt/hello-world-spring-boot
 ✔ Vulnerability DB        [updated]
 ✔ Loaded image
 ✔ Parsed image
 ✔ Cataloged packages      [176 packages]
 ✔ Scanned image           [149 vulnerabilities]

NAME                         INSTALLED      FIXED-IN  VULNERABILITY        SEVERITY
h2                           1.4.197                  CVE-2018-10054       High
h2                           1.4.197                  CVE-2018-14335       Medium
hibernate-validator          6.0.11.Final   6.0.18    GHSA-m8p2-495h-ccmh  Medium
hibernate-validator          6.0.11.Final             CVE-2020-10693       Medium
jackson-databind             2.9.6          2.9.7     GHSA-645p-88qh-w398  High
jackson-databind             2.9.6          2.9.8     GHSA-f9hv-mg5h-xcw9  High

Take a look at the Grype docs to see other output formats, especially if you’re using this in CI.

Docker Scan command

Little known fact: Docker includes container vulnerability scanning out of the box.

Instead of installing additional tooling, you can use the docker scan command (backed by the Snyk vulnerability database) to get a similar list of vulnerabilities as provided by Anchore.

Similar to above:

    git clone [email protected]:EBTPT/spring-boot-hello-world.git
    cd spring-boot-hello-world
    mvn package
    docker build . -t spring-boot-hello-world
    docker scan spring-boot-hello-world

Which outputs:

$ docker scan ebtpt/hello-world-spring-boot

Testing ebtpt/hello-world-spring-boot...

✗ Low severity vulnerability found in openssl/libcrypto1.1
  Description: Inadequate Encryption Strength
  Introduced through: openssl/[email protected], openssl/[email protected], apk-tools/[email protected], libtls-standalone/    [email protected], ca-certificates/ca-certificates@20190108-r0, krb5-conf/[email protected]
  From: openssl/[email protected]
  From: openssl/[email protected] > openssl/[email protected]
  From: apk-tools/[email protected] > openssl/[email protected]
  and 7 more...
  Fixed in: 1.1.1j-r0

<...lots more here...>

✗ High severity vulnerability found in bzip2/libbz2
  Description: Out-of-bounds Write
  Introduced through: bzip2/[email protected], freetype/[email protected]
  From: bzip2/[email protected]
  From: freetype/[email protected] > bzip2/[email protected]
  Fixed in: 1.0.6-r7

Package manager:   apk
Project name:      docker-image|ebtpt/hello-world-spring-boot
Docker image:      ebtpt/hello-world-spring-boot
Platform:          linux/amd64

Tested 55 dependencies for known vulnerabilities, found 87 vulnerabilities.

Alpine 3.9.4 is no longer supported by the Alpine maintainers. Vulnerability detection may be affected by a lack of security updates.

For more free scans that keep your images secure, sign up to Snyk at

Docker Hub (Paid plans)

Those with a paid Docker Hub plan can enable container scanning as images are pushed into the repository. This surfaces the same information as the docker scan command, and uses the same Snyk datasource. For some, this might be to late in the flow, but that’s entirely dependent on context.

Screenshot of Docker Hub scanning settings

GitHub Actions

Where the Docker Image is being built as part of the CI pipeline, adding scanning to the pipeline is easy, especially in GitHub where the GitHub Actions marketplace has many 3rd party tools packaged for easy inclusion in pipelines. GitHub Actions appears to have innovated upon the CircleCI concept of Orbs, and brought an enormous ecosystem of tools to bear for easy inclusion in CI, largely through its position of “being GitHub”. For example, here’s Anchore Grype. Results can be emitted as a file, as well as configured to fail the build.

Remediations for container problems

The vulnerabilities might be introduced by something in the local Dockerfile, one of the application dependencies, or something from the upstream image. The Dockerfile lists the upstream image as openjdk:8-jdk-alpine. Depending on the environment in which you’re developing, there’s a trade-off here between silent updates and reproducible builds. Docker tags are not immutable. This tag will have been updated many times through its lifetime. Each time a downstream image like ours was built, the latest image associated with the tag will have been fetched, including any new fixes in the upstream image. This has the downside of not being able to guarantee the same behaviour between two container image builds with no application code changes. This isn’t just a problem of reproducible builds, but a problem of unrecorded unintentional changes. This can be resolved by pinning the SHA256 of the image, e.g. changing FROM openjdk:8-jdk-alpine to FROM openjdk:8-jdk-alpine@sha256:a3562aa0b991a80cfe8172847c8be6dbf6e46340b759c2b782f8b8be45342717. Once this is done, you can treat this is as a dependency like any other, cognisantly upgrading when needed. If you use Dependabot for automated updates, this supports updates of Dockerfiles too.

The reason there are so many vulnerabilities on this particular tag is that it’s no longer maintained. Alpine 3.9 was end of life some time ago, and this image was last updated in May 2019.

To solve this instance, we could switch the base image to openjdk:8-jdk-oracle which is still maintained, and at time of writing lists 0 vulnerabilities. We could also switch to openjdk:8-jdk-slim but whilst this is still maintained, at time of writing it boasts more vulnerabilities than the unmaintained alpine image (not that “count” is necessary a subjective measure of quality or security).

Testing the Application Code

Given the information we’ve surfaced so far, we might now trust the container and trust our dependencies (insofar as they contain no known vulnerabilities), so we now need to turn our focus to our own code.

These examples are somewhat lighter on content, as at the time of writing, the repo I’ve picked contains no detectable vulnerabilities. If I updated the repo to introduce some, the examples would still be very contextual to the code and its issues, and wouldn’t likely aid you fair readers.

GitHub CodeQL

Turn it on to get static analysis as one of your CI pipeline checks on your PRs, and to get enhanced alerting in the Security tab in GitHub.

  • Go to Security -> Code scanning, and click the ‘Configure scanning tool’ button.
  • Next to CodeQL analysis, pick Set up, then Default
  • Tick Java / Kotlin
  • Press the Enable CodeQL button

Screenshot of CodeQL config

This will kick off a Github Action run, which will be slow on its first run, as it needs to scan the entire repository, and deliver results to Security -> Code scanning, ready for your review.

CodeQL alternatives in GitHub Actions

If you’d rather not rely more heavily on GitHub and its internal tooling (perhaps because you want to be ready to migrate away at a moment’s notice), the GitHub Actions marketplace absolutely has off-the-shelf alternatives that will be portable to other CI systems. Take a look at

Snyk in the IDE

Snyk has a hosted option (which makes sense, given the number of times we’ve mentioned the rich Snyk data throughout this article) but did you know that multiple IDEs have Snyk plugins to enhance the static analysis that happens as you type or open files?

As well as highlighting as you go, local full scans are possible to get a full set of problems, not just in code, but in dependencies and Infrastructure-as-Code as well.

Screenshot of Snyk in IntelliJ

SpotBugs and FindSecBugs

This is only for Java, but alternatives for other languages are listed in the OWASP Wiki.

Full System Solutions

Rather than apply effort layer-by-layer, some solutions want to reduce that burden by offering all layers wrapped in a single package.


Mentioned above as an IDE plugin, Snyk offer a hosted platform that will cover every layer of the application.

They’ll tackle container vulnerabilities, dependencies, code, and even look at code quality and peek at your IaC to check your deployment security.


GitLab have had a proper Security Center in their offering since before GitHub offered Actions. They’ll do all the scans, including more advanced application security scanning like DAST. All vulnerabilities generate a detection which can be triaged (issues created, dismissed as false positives, etc), which is lovely auditor fodder, if you care about such things.


I’ve mentioned a bunch of tools, some of which would love to have some of your money. I’ve mentioned tools that I’ve used and like, but I am in no way affiliated, remunerated, compensated or related to any of them.