Engineering
6 min read

Goodbye ENV Files

Why JSON is a better format than .env files for app secrets and configuration.

Feb 19, 2020
Brian Vallelunga Avatar
Brian Vallelunga
Founder & CEO
Goodbye ENV Files
Back to the blog
Goodbye ENV Files
Share
Engineering

We learned a lot when we stopped using ENV files as the default export option in the Doppler CLI. This change yielded several benefits, including supporting multi-line variables and a deterministic schema. Along the way, we identified several reasons why developers should want to trend away from storing secrets in ENV files and want to share our experience. However, before going too deep on the technical choices we made, let's first go over what ENV files are and how they're used in software development.

What are ENV files?

ENV files are plain text files that store variables and secrets you would not want hardcoded in your codebase. These environment variables could be a port number or a database URL and may change depending on where your code is deployed. For example, when developing locally, you may use port 3000, but when deployed to Heroku, your application will need to use the port it's dynamically assigned. An example ENV file, when developing locally, could look something like this with the schema of KEY=VALUE storing your env variables:

1PORT="3000" DATABASE_URL="psql://postgres@localhost/db_name" AWS_KEY="rfiunb34fu93n49iufgn3o2o0ini2ef" BUGSNAG_API_KEY="33dasdk34bsf23f35871as0fa27" INTERCOM_KEY="ndeiovnkofrnij30490398u39RIBF39IOENIOWF3ENWKEWENVIO" MAILGUN_KEY="fdjnbvjoenvlmswsdnjoelfv" NEW_RELIC_LICENSE_KEY="b583d4eab955872122843a067faca9db5d4202af" NEW_RELIC_LOG="stdout" SALESFORCE_TOKEN="feojkvndfjkovnskfn3eni32one" STRIPE_KEY="sk_test_ejidsfnvoiuebnfvoi3enjkdNCJSB"

Reasons to use an ENV file

So, what are some of the reasons for using an ENV file? These files live on your local machine, meaning you do not need a network connection to fetch your secrets. The schema is also quite simple, so it's easy to go into a file and add a new variable. Lastly, everyone knows this format, so there is a ton of support from the open-source community for parsers and managers.

Why you shouldn't use ENV files

From working with ENV files and adding support for various use cases, we have found that there isn't a standardized schema all libraries use.

Issues with parsing ENV files

For example, take a look at the sample ENV file below:

1PORT = 8080

Notice that this ENV file contains a space between the KEY and the VALUE. If we were to use bash to inject the variable into the environment with the source command, we would get an error for this parameter:

1source spaces.env
2# >> bash: PORT: command not found

Now, if we use another tool like Foreman, we would see it parse without an error. This is because each library decides the schema of an ENV file instead of strictly following an open standard. These inconsistencies cause other problems to arise as well, such as parsing multi-line secrets. In this example, a variable uses encoded newlines through \n.

1CERT="-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE\nlw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk\nqDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr\nR3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr\n9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG\nBO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP\nUvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm\n4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO\n2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY\ndqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf\nzq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf\nSidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE\nCCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB\nVLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6\ndBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU\nOOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL\ndX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva\n/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh\n2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K\n2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV\nZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM\nUyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc\nO/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F\n6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa\nNxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=\N-----END RSA PRIVATE KEY-----\n"

This newline is treated differently depending on which tool you use. Using the bash source command, the \n in the string would not be converted to newline characters. This is the same with Node.js, where the process.env global variable does not convert the \n to newlines at runtime automatically. On the other hand, using Python's most popular ENV library dotenv will convert the \n to newlines automatically. Now let's look at the inverse:

1CERT="-----BEGIN RSA PRIVATE KEY-----
2MIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE
3lw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk
4qDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr
5R3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr
69S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG
7BO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP
8UvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm
94KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO
102UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY
11dqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf
12zq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf
13SidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE
14CCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB
15VLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6
16dBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU
17OOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL
18dX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva
19/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh
202splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K
212iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV
22ZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM
23Uyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc
24O/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F
256kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa
26NxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=
27-----END RSA PRIVATE KEY-----"

In this example, we have the same cert but with newline characters. Surprisingly, the bash source command respects the newline character, and the Node dotenv library from NPM does not. More interestingly is how the Node library breaks. It parses the value as "-----BEGIN RSA PRIVATE KEY-----" and disregards all the other lines.

Other Issues to consider when parsing ENV files:

  • Differences in how container providers handle ENV files, such as Docker and Kubernetes.
  • Making env variables available to frontend parts of an application for web development.
  • Debugging on different systems, such as from Linux command line and Windows command-line shells.

Problems Sharing ENV files

Developers generally exclude ENV files in a .gitignore file to prevent Git from pushing the files to Github. Often, the existence of this file is the only thing preventing developers from accidentally publishing secrets to a code repo. Depending on the security settings, env vars could easily be exposed online. This is a DevOps nightmare for startups looking to earn customer trust. Even a small window of exposure could be a disaster.

Developers often share secrets in ways that may not be secure or desirable. For example, sharing unencrypted secrets in .env files over Slack when secrets change, or new developers join a team risks breaking the principle of least privilege by exposing secrets to potentially unauthorized users. There’s also no version control, and developers often get out of sync.

Alternatives to ENV files

After realizing ENV files are problematic, we started looking at alternative formats we could export to. We wanted a universally accepted schema with no room for interpretation and a large community for support. The two data formats we focused on were YAML and JSON.

Let's start with YAML. One of the primary advantages of YAML is that it is straightforward to read and write. It uses indentation and nesting as a way to designate structure. Let's look at a sample YAML file:

1PORT: 3000
2DATABASE_URL: "psql://postgres@localhost/db_name"
3AWS_KEY: "rfiunb34fu93n49iufgn3o2o0ini2ef"
4CERT: |
5  -----BEGIN RSA PRIVATE KEY-----
6  MIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE
7  lw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk
8  qDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr
9  R3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr
10  9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG
11  BO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP
12  UvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm
13  4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO
14  2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY
15  dqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf
16  zq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf
17  SidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE
18  CCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB
19  VLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6
20  dBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU
21  OOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL
22  dX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva
23  /YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh
24  2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K
25  2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV
26  ZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM
27  Uyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc
28  O/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F
29  6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa
30  NxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=
31  -----END RSA PRIVATE KEY-----

At first glance, the syntax looks very similar to the ENV format, but when we look closer, we see subtle differences. The YAML syntax uses colons instead of equal signs and has native support for multi-line strings. The one downside when using multi-line secrets is that indentation matters. The fabled debate of how many spaces equals a tab comes into play. With developers each having their own style, YAML files can be prone to parsing errors when sharing.

A JSON file, on the other hand, has a wildly different syntax than YAML. Wikipedia has an accurate description of the language:

JavaScript Object Notation is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value).

Let's take a look at the same config of variables in JSON format:

1{
2  "PORT": "3000",
3  "DATABASE_URL": "psql://postgres@localhost/db_name",
4  "AWS_KEY": "rfiunb34fu93n49iufgn3o2o0ini2ef",
5  "CERT": "-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE\nlw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk\nqDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr\nR3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr\n9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG\nBO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP\nUvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm\n4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO\n2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY\ndqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf\nzq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf\nSidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE\nCCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB\nVLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6\ndBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU\nOOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL\ndX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva\n/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh\n2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K\n2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV\nZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM\nUyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc\nO/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F\n6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa\nNxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=\n-----END RSA PRIVATE KEY-----\n"
6}

One of the prominent beauties of JSON is that it is strictly enforced, and there is only one way of accomplishing each task. For example, when we look at the variable PORT, we can see the value is wrapped in quotes to state it is a string. Unlike YAML, which will guess if the line should be cast to a string or number, JSON only has one way of notating strings and numbers. Another stark difference between YAML and JSON is how they handle multi-line variables. In JSON, we can see it uses the encoded newline characters \n, which is a safer bet than trusting humans with indentation.

JSON versus YAML for ENV Variables

We went with JSON because it has a far stricter schema and strong native support in most languages. After making the switch, we saw our customers' issues with parsing downloaded config files flat line. Since the Doppler CLI creates a fallback of your secrets by default when running your application, we decided to go one step further by enabling encryption by default.

We strongly believe that you will always be worse off having secrets on disk, but if you are going to, they must be encrypted and not left in plain English.

Let Doppler Manage Environment Variables

Tired of managing a fleet of secrets by hand using copy/paste? Want an end-to-end managed secrets manager that vaults all your secrets in one place, has built-in versioning and access control? Try out Doppler. It works great in local development and in production. It effortlessly scales with you as your team and products grow. Take a look for free to see if it is a fit for your team.

Stay up to date with new platform releases and get to know the team of experts behind them.

Related Content

Explore More