Trusted private container registry for EGI

(Bruce Becker) #1

Continuing the discussion from Where should we push our images?:

So, could we have a container registry in EGI, for use by members of federation and customers of EGI. We already have two “homes” for software artefacts:

There has been a convention until now that infrastructure components[1] go into the UMD repo, whilst the AppDB contained community and user applications. This convention has been a bent a bit by a few FedCloud appliances[2]. Also, neither of these artifact repositories exposes the Docker Registry API, allowing container images to be pushed to and pulled from them[3].

Are we missing a piece of infrastructure

So, we have OS-package sources and virtual appliance sources for use in cloud sites. Are we missing a piece of infrastructure for the Cloud Container Compute service? Perhaps.

Given the very good services already available for free via Docker Hub, and Red Hat Quay. The major cloud providers Google Cloud Platform, Azure and AWS provide enticing options. GitLab has one. Puppet has one. This space is pretty well covered…

A very good justification should be made for EGI building and offering this service.

We already deliver container images to Quay and Docker Hub – actually via (at least) two separate organisations: EGI FedCloud and EGI Foundation. Those are all public organisations on these hosted providers, meaning that the EGI Federation has no control over who uses these applications, and the rights to push to them is not managed by the EGI Attribute Management service, but rather done using indvidual credentials internal to those platforms.

Clearly, we need a container registry, but do we need to operate it? If we were to do so, what would be the business case:

  1. What gap would this service be filling, and to what kind of customer?
  2. What would be the cost of such as service, and how would that cost be recovered?

Gaps

We have discussed the use cases before. The current offerings have a a few gaps. The ability to protect repositories according to specific criteria is one. These criteria could be e.g.

  • Access from specific domains (cloud site, EGI infrastructure, etc)
  • Allow access only to specific identities or groups

The key issues are typically the confidentiality or proprietary nature of the application.

Apart from these, some other issues, may be the discoverability and trustworthiness of the application. These could be solved by ensuring consistency of images across all registries and signing the image layers with the same key, to prove provenance.

Another gap could be better co-ordination with the EGI Software Vulnerability Group and the EGI CSIRT, in order to better trust the artifacts delivered to EGI infrastructure. Again, thanks to the massive uptake of the technology, there is a very mature ecosystem of tools[4], from which components – even FOSS components – could be picked off the shelf. It is also entirely feasible that by using these components, we could enhance existing efforts to monitor and secure the production infrastructure as well as arbitrary user applications.

I can’t tell if this has been proposed or done before in the EGI ecosystem – either by federation members or projects within the e-Infrastructure domain, which is somewhat surprising to me.

At a first glance then, either the service is not really needed, or it does not make business sense to build it. Another possibility is that there is a latent opportunity here.

A container registry operated by EGI could be built to fill this gap, if it were properly designed to integrate as a component of other EGI Services.

Costs and business model

Let’s assume that such a service is actually desired, that a true gap exists and that there is a latent opportunity for EGI here to deliver some value. How would the container registry service be built?

The first consideration is of course to study the landscape and see what options we have. We could then decide what components we could re-use and which we’d have to build from scratch. Given a few realistic scenarios, we could come up with a few business models.

There are already production-grade components for the service available (which I’ll flesh out below), even entire products which can be bought[5].

Building the service

If we had to build this service from scratch, it might look something like this…

Some details described below – (:moneybag: :arrow_right: incurs some cost[6] , :free: :arrow_right: free as in :beers: and free as in :statue_of_liberty: )

  • Container Registry Service:
    • Function: Store and deliver container images and keep track of image layers
    • Product: Docker Registry :free:
  • Container Registry Web Server:
    • Function: Provide a user-friendly web interface to browse containers and their metadata.
    • Product: Reg from Genuine Tools :free:
  • Metadata Service:
    • Function: Produce relevant object metadata for sharing with other services (AppDB, OpenAIRE).
    • Product: In-House :moneybag:
  • Image Store:
    • Function: Provide the persistent backing store for the image layers in the registry
    • Product: various cloud providers, possibly free from EGI FedCloud providers. :free:
  • Continuous Integration Service:
    • Function: Perform testing and build functions upon changes to respective source repositories.
    • Product: In-house Jenkins :free: or hosted CI :moneybag:[7]
  • Security Scanning Service:
    • Function: scan the layers for known vulnerabilities and alert maintainers or operations
    • Product: Clair :free:
  • Image/Layer signing service:
    • Function: Sign images, layers or metadata with trusted cryptographic keys, to propagate trust.
    • Product: In house :moneybag: or Docker Notary
  • Signing Key Database:
    • Function: Store the trusted signing keys for the Signing Service
    • Product: keybase :free: , Hashicorp Vault :free: or in-house :moneybag:
  • CVE DB:
    • Function: Store relevant common vulnerabilities and exposures.
    • Product: EGI CSIRT and other CVE products. :free:

I haven’t drawn any flow lines here, although I have put in other parts of the EGI infrastructure. The service will be accessed as usual via EGI AAI, taking advantage of the group and attribute authorities, protecting the services appropriately. The link to the EGI CSIRT and SVG activities is also importantly present. Some external sources are displayed - source control repositories which may contain the applications themselves and container build definitions from :github:, GitLab or BitBucket.

Artefacts typically flow from commits to the repository to the image store, while metadata is generated internally to the service and is synchronised with the AppDB via the metadata service.

Filling the gaps : Summary and conclusion

What is not shown here is the interface to the actual infrastructure where the images would be executed in containers. It is implied that access to the individual images would be protected using the same AAI as the rest of the ecosystem (EGI Check-In for example), but I admit that actually ensuring that such protections are in place could be a non-trivial task. Several strategies could be imagined, up to and including the encryption of individual layers with personal certificates from RCAuth. We could thus cover the confidentiality and protection of private images.

At the very least, we would be able to adequately address the provenance issue, by allowing users of images in the repositories offered by the service to check whether they originated from the delivery pipeline and who signed them.

The question remains how much this would actually cost to build and operate. There is an abundance of readily-available open source tooling to build the main components, although a few would have to be built in-house. Although this would incur some sort of cost, I don’t think it would be very significant. The operation of the service could be done on existing EGI cloud infrastructure, thereby reducing the need for recurring expenditure.

Recovering costs from communities would be a tough sell though, because the moment we start charging for usage of the service, they will likely be led to competing offerings – and we’ve already shown that there is an abundance of choice.

Verdict

Jury’s out… but @UCST should definitely engage with communities around this. Else, we’ll end up doing a Jurassic Park.

I wouldn’t be surprised if this came in the form of an EGI Innovation Fund proposal.


  1. Actually, this is even more restricted to just the “grid infrastructure” components. ↩︎

  2. In order for a FedCloud resource provider to be integrated, certain infrastructure components must be present, e.g. the cASO accounting conversion appliance. These are not kept UMD repo, but in the AppDB. ↩︎

  3. Sure, you could always export your image as a tarball and push that to AppDB instead. ↩︎

  4. Just under 30 options according to this list from SysDig for an idea. ↩︎

  5. See this Slant.co comparison of options ↩︎

  6. These costs may be capital or development costs. ↩︎

  7. This would incur costs because hosted CI platforms are typically paid, for private repositories. The whole point of this post is to describe a solution for this use case, so we are considering this default scenario. Notable exceptions would be GitLab runners, or GitHub Actions. ↩︎