• Blog
  • Info Support
  • Career
  • Training
  • International Group
  • Info Support
  • Blog
  • Career
  • Training
  • International Group
  • Search
logo InfoSupport
  • Latest blogs
  • Popular blogs
  • Experts
      • All
      • Bloggers
      • Speakers
  • Meet us
  • About us
    • nl
    • en
    • .NET
    • 3D printing
    • Advanced Analytics
    • Agile
    • Akka
    • Alexa
    • Algorithms
    • Api's
    • Architectuur
    • Artificial Intelligence
    • ATDD
    • Augmented Reality
    • AWS
    • Azure
    • Big Data
    • Blockchain
    • Business Intelligence
    • Chatbots
    • Cloud
    • Code Combat
    • Cognitive Services
    • Communicatie
    • Containers
    • Continuous Delivery
    • CQRS
    • Cyber Security
    • Dapr
    • Data
    • Data & Analystics
    • Data Science
    • Data Warehousing
    • Databricks
    • DataOps
    • Developers life
    • DevOps
    • Digital Days
    • Digital Twin
    • Docker
    • eHealth
    • Enterprise Architecture
    • Event Sourcing
    • Hacking
    • Infrastructure & Hosting
    • Innovatie
    • Integration
    • Internet of Things
    • Java
    • Machine Learning
    • Microservices
    • Microsoft
    • Microsoft Bot Framework
    • Microsoft Data Platform
    • Mobile Development
    • Mutation Testing
    • Open source
    • Pepper
    • Power BI
    • Privacy & Ethiek
    • Python
    • Quality Assistance & Test
    • Quality Assurance & Test
    • Requirements Management
    • Scala
    • Scratch
    • Security
    • SharePoint
    • Software Architecture
    • Software development
    • Software Factory
    • SQL Server
    • SSL
    • Start-up
    • Startup thinking
    • Stryker
    • Test Quality
    • Testing
    • TLS
    • TypeScript
    • Various
    • Web Development
    • Web-scale IT
    • Xamarin
    • All
    • Bloggers
    • Speakers
Home » GitOps: Storing secrets safely in Git
  • GitOps: Storing secrets safely in Git

    • By Martin Kanters
    • Cyber Security 1 year ago
    • Cyber Security 0 comments
    • Cyber Security Cyber Security
    GitOps: Storing secrets safely in Git

    Secret management in IT is a complex problem. Secrets should be stored safely, versioned and easy to use. The least amount of people should be able to manage or even see them. This article explains how to solve this problem using encryption in a GitOps fashion.

    External secrets vault

    Let’s look at two techniques often used for secret management. Most of the time they are stored in external services, such as Hashicorp Vault. This is a product that is installed and running somewhere your application can access. It usually provides very refined access control mechanisms, like RBAC. Applications can read these secrets on runtime and use them any way they like. Some of these services even offer versioning capabilities of these secrets.

    Kubernetes Secrets

    In case your application is running on Kubernetes, there is another way to externalize secrets. Kubernetes offers a specific resource type called a Secret. This is a key-value map, where the value is base64 encoded (read: not encrypted). A Kubernetes pod can then read the value from this secret into an environment variable or mount it onto the filesystem. RBAC rules can protect the Secrets from being read by unwanted eyes.

    Problems

    There are some issues with both type of solutions:

    • Versioning: Both Kubernetes Secrets and external vaults have some support of versioning, but often work very different from version control systems that are already used in the company. Developers need to understand both ways of versioning and make it work together somehow.
    • Independence: Your application is not one contained package of deployment once external secrets are referenced from within. If the secret your application uses has been deleted in your Vault, the package that worked yesterday, doesn’t work anymore today.
    • Process: Application development and secret management are two separate processes often done by separate parties. A developer is not always able to manage secrets himself, so it’s possible that he deploys the application to production, while the required secret is not available on production yet.

    Introducing GitOps

    Because of the reasons listed above and other things, it is becoming more and more common to store everything in Git. First only application code was stored in Git, but it was later followed by database evolution scripts and even CI pipelines. GitOps is an upcoming way of storing even the desired state of your application architecture and infrastructure in Git. Imagine a Git repository including all Kubernetes (and for example Terraform) definitions. You get all the benefits of a great source control system and you can easily attach a pipeline to it which automatically deploys your desired state to Kubernetes. Or even better, you can use a tool, like Argo CD, for that!

    Sounds nice, but as most people know, secrets are one of the things that should never be stored in Git. Once stored, everyone who can access the repository can see them, so they are compromised and will forever stay in the history (unless you dare to rewrite the history – nineteen eighty-four anyone?).

    Encryption

    One way to overcome this problem is to encrypt the secrets before pushing them to your Git repository. By using asymmetric encryption every developer can encrypt secrets using a (public) certificate and upload them safely to Git. The private key is the only means of decrypting the secret, so this should obviously be kept hidden. It should be installed closest to the place where the secrets will be used. So how do we put this to practice?

    Bitnami SealedSecrets

    SealedSecrets by Bitnami is an open-source tool which can be used in Kubernetes environments which works exactly like explained above. It extends Kubernetes with a custom resource definition (CRD). Once installed, a Kubernetes controller starts running. It will generate a private key and a certificate. The certificate should be made public to the developers of the company. The process then works as follows:

    1. The developer installs the CLI tool that comes with SealedSecrets, called “kubeseal”.
    2. The developer creates a regular Kubernetes Secret, exactly like how it should end up in the Kubernetes cluster.
    3. The developer runs kubeseal against the Secret to encrypt the contents and to generate a SealedSecret.
    4. The developer pushes the SealedSecret to their GitOps repository.
    5. The GitOps repository pipeline deploys the SealedSecret to Kubernetes.
    6. The SealedSecret controller notices the SealedSecret and decrypts it into a regular Kubernetes secret.
    7. Now it can be used like any other regular Kubernetes secret.
    Showing how kubeseal CLI can be used to encrypt a regular Kubernetes Secret.
    The developer creates a SealedSecret from a regular Kubernetes Secret using kubeseal.

    This sounds simple and it really is. Both the Kubernetes and CLI installation are done in a matter of minutes and the usage is simple as well.

    Keep this in mind, though…

    • Only the private key can decrypt these SealedSecrets. Once the private key is lost, the secret is lost as well. That’s why it’s smart to back up the private key. Next to that, writing down how to regenerate secrets and backing up non-regeneratable secrets will help you to recover in emergency situations.
    • Even though the exposure of the secret is now limited to one developer, you can think of ways to let the secret have no exposure at all. For example, a shared secret between two services can easily be generated, encrypted and committed to the GitOps repository by a script.
    • Make sure to limit the number of people who are able to see the unencrypted Secrets or the private key on Kubernetes. Additionally, things like auditing and 4-eyes processes can be put in place to ensure even those people cannot access them without permission.
    • Specific to SealedSecrets: Only one private key is created over a whole cluster, but the name of the secret and the namespace are used when encrypting it. So, when you are changing either the namespace or the name, make sure to encrypt it again, otherwise it cannot be decrypted.

    Alternatives

    If you are running on Kubernetes and making use of Hashicorp Vault already, you can also make use of Vault sidecar injectors. Then you can inject secrets into your container using annotations on your Pod templates. This is definitely interesting, but the annotation template syntax can be quite cumbersome and error prone sometimes. You also need a sidecar next to every Pod. While this does not have to be a problem, it certainly increases the complexity of your application architecture.

    I don’t know of any similar tools on non-Kubernetes environments, but I can imagine they exist, because the essence of such a tool is not something new. Writing a tool like this yourself is something I would never recommend – security is always hard to do right – but if there is no alternative it is always a possibility.

    Conclusion

    Secrets are historically a hassle to do right. Products like Hashicorp Vault or Kubernetes Secrets can be used to store secrets, but then secret management is not integrated in application development, which results in an error-prone system. GitOps aims to store desired state in Git. Plaintext secrets cannot be stored in a repository, but they can when they are encrypted asymmetrically. A tool to use for this is Bitnami SealedSecrets. It is easy to install and use, but some processes should be put in place to use this safely.

    Share this

Martin Kanters

View profile

Data Discovery Channel

  • Modern Data Platform

  • Gartner Data & Analytics Summit 2022

  • De Data Architecture ®Evolution

Nieuwsbrief

* verplichte velden

Contact

  • Head office NL
  • Kruisboog 42
  • 3905 TG Veenendaal
  • T +31 318 552020
  • Call
  • Mail
  • Directions
  • Head office BE
  • Generaal De Wittelaan 17
  • bus 30 2800 Mechelen
  • T +32 15 286370
  • Call
  • Mail
  • Directions

Follow us

  • Twitter
  • Facebook
  • Linkedin
  • Youtube

Newsletter

Sign in

Extra

  • Media Library
  • Disclaimer
  • Algemene voorwaarden
  • ISHBS Webmail
  • Extranet
Beheer cookie toestemming
Deze website maakt gebruik van Functionele en Analytische cookies voor website optimalisatie en statistieken.
Functioneel Always active
De technische opslag of toegang is strikt noodzakelijk voor het legitieme doel het gebruik mogelijk te maken van een specifieke dienst waarom de abonnee of gebruiker uitdrukkelijk heeft gevraagd, of met als enig doel de uitvoering van de transmissie van een communicatie over een elektronisch communicatienetwerk.
Voorkeuren
De technische opslag of toegang is noodzakelijk voor het legitieme doel voorkeuren op te slaan die niet door de abonnee of gebruiker zijn aangevraagd.
Statistieken
De technische opslag of toegang die uitsluitend voor statistische doeleinden wordt gebruikt. De technische opslag of toegang die uitsluitend wordt gebruikt voor anonieme statistische doeleinden. Zonder dagvaarding, vrijwillige naleving door uw Internet Service Provider, of aanvullende gegevens van een derde partij, kan informatie die alleen voor dit doel wordt opgeslagen of opgehaald gewoonlijk niet worden gebruikt om je te identificeren.
Marketing
De technische opslag of toegang is nodig om gebruikersprofielen op te stellen voor het verzenden van reclame, of om de gebruiker op een website of over verschillende websites te volgen voor soortgelijke marketingdoeleinden.
Manage options Manage services Manage vendors Read more about these purposes
Voorkeuren
{title} {title} {title}