Goodbye ENV Files

We learned a lot when we stopped using ENV files as the default export option in the Doppler CLI. This change yielded several benefits, including supporting multi-line variables and a deterministic schema. Along the way, we identified several reasons why developers should want to trend away from storing secrets in ENV files and want to share our experience. However, before going too deep on the technical choices we made, let's first go over what ENV files are and how they're used in software development.

What are ENV files?

ENV files are plain text files that store variables and secrets you would not want hardcoded in your codebase. These environment variables could be a port number or a database URL and may change depending on where your code is deployed. For example, when developing locally, you may use port 3000, but when deployed to Heroku, your application will need to use the port it's dynamically assigned. An example ENV file, when developing locally, could look something like this with the schema of KEY=VALUE storing your env variables:

Reasons to use an ENV file

So, what are some of the reasons for using an ENV file? These files live on your local machine, meaning you do not need a network connection to fetch your secrets. The schema is also quite simple, so it's easy to go into a file and add a new variable. Lastly, everyone knows this format, so there is a ton of support from the open-source community for parsers and managers.

Why you shouldn't use ENV files

From working with ENV files and adding support for various use cases, we have found that there isn't a standardized schema all libraries use.

Issues with parsing ENV files

For example, take a look at the sample ENV file below:

Notice that this ENV file contains a space between the KEY and the VALUE. If we were to use bash to inject the variable into the environment with the source command, we would get an error for this parameter:

Now, if we use another tool like Foreman, we would see it parse without an error. This is because each library decides the schema of an ENV file instead of strictly following an open standard. These inconsistencies cause other problems to arise as well, such as parsing multi-line secrets. In this example, a variable uses encoded newlines through \n.

This newline is treated differently depending on which tool you use. Using the bash source command, the \n in the string would not be converted to newline characters. This is the same with Node.js, where the process.env global variable does not convert the \n to newlines at runtime automatically. On the other hand, using Python's most popular ENV library dotenv will convert the \n to newlines automatically. Now let's look at the inverse:

In this example, we have the same cert but with newline characters. Surprisingly, the bash source command respects the newline character, and the Node dotenv library from NPM does not. More interestingly is how the Node library breaks. It parses the value as "-----BEGIN RSA PRIVATE KEY-----" and disregards all the other lines.

Other Issues to consider when parsing ENV files:

• Differences in how container providers handle ENV files, such as Docker and Kubernetes.
• Making env variables available to frontend parts of an application for web development.
• Debugging on different systems, such as from Linux command line and Windows command-line shells.

Problems Sharing ENV files

Developers generally exclude ENV files in a .gitignore file to prevent Git from pushing the files to Github. Often, the existence of this file is the only thing preventing developers from accidentally publishing secrets to a code repo. Depending on the security settings, env vars could easily be exposed online. This is a DevOps nightmare for startups looking to earn customer trust. Even a small window of exposure could be a disaster.

Developers often share secrets in ways that may not be secure or desirable. For example, sharing unencrypted secrets in .env files over Slack when secrets change, or new developers join a team risks breaking the principle of least privilege by exposing secrets to potentially unauthorized users. There’s also no version control, and developers often get out of sync.

Alternatives to ENV files

After realizing ENV files are problematic, we started looking at alternative formats we could export to. We wanted a universally accepted schema with no room for interpretation and a large community for support. The two data formats we focused on were YAML and JSON.

Let's start with YAML. One of the primary advantages of YAML is that it is straightforward to read and write. It uses indentation and nesting as a way to designate structure. Let's look at a sample YAML file:

At first glance, the syntax looks very similar to the ENV format, but when we look closer, we see subtle differences. The YAML syntax uses colons instead of equal signs and has native support for multi-line strings. The one downside when using multi-line secrets is that indentation matters. The fabled debate of how many spaces equals a tab comes into play. With developers each having their own style, YAML files can be prone to parsing errors when sharing.

A JSON file, on the other hand, has a wildly different syntax than YAML. Wikipedia has an accurate description of the language:

JavaScript Object Notation is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value).

Let's take a look at the same config of variables in JSON format:

One of the prominent beauties of JSON is that it is strictly enforced, and there is only one way of accomplishing each task. For example, when we look at the variable PORT, we can see the value is wrapped in quotes to state it is a string. Unlike YAML, which will guess if the line should be cast to a string or number, JSON only has one way of notating strings and numbers. Another stark difference between YAML and JSON is how they handle multi-line variables. In JSON, we can see it uses the encoded newline characters \n, which is a safer bet than trusting humans with indentation.

JSON versus YAML for ENV Variables

We went with JSON because it has a far stricter schema and strong native support in most languages. After making the switch, we saw our customers' issues with parsing downloaded config files flat line. Since the Doppler CLI creates a fallback of your secrets by default when running your application, we decided to go one step further by enabling encryption by default.

We strongly believe that you will always be worse off having secrets on disk, but if you are going to, they must be encrypted and not left in plain English.

Let Doppler Manage Environment Variables

Tired of managing a fleet of secrets by hand using copy/paste? Want an end-to-end managed secrets manager that vaults all your secrets in one place, has built-in versioning and access control? Try out Doppler. It works great in local development and in production. It effortlessly scales with you as your team and products grow. Take a look for free to see if it is a fit for your team.