Standardizing Observability at Procore

Implementing observability into all of your services can be a large undertaking. At Procore, we developed observability libraries to lower the barrier of implementing standardized observability across services, while also reducing dependency on specific vendors.

Standardizing Observability at Procore
Standardizing Observability at Procore

At Procore, we want to implement standardized observability across all of our services with the least burden on our internal developer community. The easier it is for developers to include observability within their services, the more quickly we could get every service covered. We created an observability library for our most commonly used languages: Ruby, JavaScript/TypeScript, and Java to accomplish this.

Our Observability Libraries are the Golden Path of Success

Lowered barrier-of-entry
If you asked all of your development teams to add logging, metrics, and traces into their services, most of them would likely install three new packages. That would be three new packages that they would have to learn, maintain, and configure. So to lower the barrier-to-entry and cognitive overload, our language-specific libraries allow developers to install just one package that includes logging, metrics, and traces. Additionally, we abstracted the default configuration of these modules away from the developer so they will not have to worry about creating/maintaining/duplicating service specific configurations. Developers will get all of this for free, out-of-the-box, from one package.

Standardized and Simplified
Where each team can implement their own configuration for observability data, they will likely end up with different standards. Since all telemetry metrics, emitted logs, and traces will go through these libraries, we can standardize the data by shaping and tagging the data before passing it to our 3rd party services. All developers have to do is import our observability library and call the module they are trying to use. Here is an example of how we're making it easy to set up tracing for our Procore developers using our JavaScript/TypeScript observability library:

const { initTracing } = require(‘our-ts-observability-library');
initTracing('example-service');

With these two lines of code, this service will have traces sent to an observability platform vendor, for example, New Relic. The developer does not have to worry about setting up the correct configuration file. We handle all of that in our library.

Having our observability configuration defined in one location simplifies our management for secrets and endpoints. Our teams spend less time figuring out what API endpoint to send their metrics to and more time developing features.

By standardizing all of the data produced from our services, we are removing the complexities from monitoring and troubleshooting. Doing this will create searchable and filterable logs, metrics, and traces.

Eliminate Vendor Dependency (i.e Vendor Lock)
One of the biggest freedoms our libraries give us is removing vendor lock. Since we are pushing for all services in Procore to use the observability library, we have wrapped the calls to 3rd party metrics and tracing services into our library methods. Remember the `setupTracing` method we called in the example above, well here is the code in our JavaScript/TypeScript library:

export function initTracing(serviceName: string): void {
  process.env.NEW_RELIC_APP_NAME = serviceName;
  const { env, stage } = standardTags;
  process.env.NEW_RELIC_LABELS = `env:${env};stage:${stage};`
  require('newrelic');
}

Say we don’t want to send traces to New Relic anymore but instead wanted to implement Open Telemetry. That is an easy switch for us. In our JavaScript/TypeScript library in the setupTracing code, we swap out the call to NewRelic with the call to Otel like this:

const { startTracing } = require('@splunk/otel');
export function initTracing(serviceName: string): void {
  startTracing({
      serviceName,
  });
}

All of these changes have been abstracted away from the development teams using our libraries and are handled within our observability libraries. The development teams don't even have to know that something has changed under the hood, they can keep calling the same methods. Once our team has finalized the code change above, the next step is to release a new version of our observability library. The development teams can update to the newest version of the observability library.  And without any changes to their own codebase, our teams will be switched over to our new 3rd party platform for traces.

Conclusion
If you are looking for the fastest way to standardize and deploy observability across all your services, I would highly suggest investing your time into an observability library. Not only does it give you uniform data in your 3rd party observability tools, it gives you peace-of-mind not being locked into a vendor that multiple services depend on individually.

If challenges like Observability excite you, come join us!