The dangers of logging and other adventures Nodeland
Learn about the dangers of logging and how Pino handles slow destinations and lack of memory, along with module releases and Platformatic updates.
111
Hi Folks! The last few weeks have been incredibly busy. In this edition I include a brief explanation of Logging - the process of creating log lines as a byproduct of an application. Read up!
I’ve included all last releases for modules, barred Platformatic… which we released multiple times per week, working hard to fix all bugs for old and upcoming features!
The dangers of Logging
One of the biggest misconceptions about logging is that it is a safe operation. It's far from safe: it's an unbounded memory leak. Whenever you log a line, you allocate memory released when the line is written to its destination. What happens if your destination cannot keep up with the number of log lines your application produces? This is primarily theoretical, as most extreme logging scenarios would resolve themselves given enough time (and memory), but it's good to consider what happens in those cases. This situation usually comes as an EBUSY error code, meaning the disk is full. There are three options to handle this situation:
- slow down your application
- drop log lines if the destination is busy
- crash for an out-of-memory error
The first option is to slow down your application, typically by synchronously waiting for the EBUSY or EAGAIN to be resolved. This usually degrades the end-user experience and is also very tricky to debug.
Dropping log lines if the destination is busy is a radical option to preserve end-user experience at the cost of observing what is currently happening, degrading the operator experience.
Crashing for an out-of-memory error is destructive, losing all ready-to-be-delivered logs. However, it is very noisy and provides the best approach for the operator, who now knows they should handle this case.
How Pino deals with this case
In Pino, writing the logline can happen synchronously or asynchronously. In synchronous logging, the only possible option to deal with a slow hard disk is to wait for it to be available again. In asynchronous logging, pino will buffer the lines and retry at given intervals.
We chose to log asynchronously by default, as it provides the best performance for your application. However, it will crash your application in case of a slow destination and a lack of available memory.
While this case is uncommon, a few companies at a massive scale had this issue. Mary Marchini had this issue and contributed the fix: implement the retryEAGAIN() option is sonic-boom to drop the bytes that the destination could not absorb, as slowing things down was not option.
Videos
I've published the videos of me adding ARRAY columns support to Platformatic DB. Who knew PostgreSQL supported arrays as a column type? This would have drastically changed most of my SQL exams at University! Watch Part 1 and Part 2.
Follow along with the progress at https://github.com/platformatic/platformatic/pull/1275.
Releases
- pino-pretty v10.2.0 adds suport for ifs in
messageFormat. - fastify v4.21.0 adds a hint on the error
ERR_REP_ALREADY_SENTthat a route may be missingreturn reply; it also fixes a few TS bugs. - fluent-json-schema fixes object schema carry properties added by raw to
$ref - @fastify/oauth2 v7.2.2 fixes
generateAuthorizationUri()typedef signature. - @fastify/secure-session v6.2.0 passes the secret to fastify cookie if directly registered.
- @fastify/cookie v9.0.0 does not send the
Set-Cookieheader multiple times for the same cookie anymore. v9.0.1 fixes the inevitable regression for a major release. v9.0.2 ships a few more bugfixes and performance improvements. v9.0.3 and v9.0.4 fix the CORS regression. - @fastify/type-provider-typebox adds support for typebox v0.30.0.
- fast-json-stringify v5.8.0 includes many performance improvements.
-
Release v5.23.0 · nodejs/undici · GitHub
What's Changed bump engines to node >= 16 by @ronag in #2119 Revert "bump engines to node >= 16 (#2119)" by @ronag in #2121 fetch: set referrer properly by @KhafraDev in #2125 fix: support truncat...
- pino-http v8.4.0 ships many bugfixes and prevent customProps to be set twice.
- autocannon v7.12.0 allows to specify body for PUT request via CLI.
- pino v8.15.0 adds default levels to opts in multistream.
- https://github.com/fastify/fastify-response-validation/releases/tag/v2.5.0 released on stream
- thread-stream v2.4.0 emits a warning if the worker gracefully exit before closing.
- retimer v4.0.0 updates the minimum supported Node.js version, add types and add support for workers-timers.
- @fastify/type-provider-typebox add typebox@0.31 to peer dependency range.
- @fastify/swagger v8.9.0 exports
formatParamUrlutility. - pino-elasticsearch v6.6.0 fixes support for elasticsearch v7 + add types.
Articles
-
Implementing OAuth2 Login with GitHub and Platformatic
OAuth 2 is a popular protocol for authentication and authorization, allowing users to authorize access to their data without the need to share sensitive passwords and other credentials with third-party applications. While this makes Oauth 2 a great o...
-
Platformatic Now Supports Arrays in PostgreSQL
When it comes to SQL, one of the most unnatural things to do is to map arrays to a table, since arrays cannot be mapped into SQL columns. Let’s consider a very common case of a “tag cloud” of a blog, where every “Page” would have multiple tags associ...
-
Distributed Tracing with Platformatic and Open Telemetry
In a microservices architecture, a request usually spans between multiple services deployed on different nodes. When this happens, it can be hard to trace issues precisely, particularly when errors happen, or when bottlenecks need to be identified. ...
-
Introducing a New Wizard for Platformatic DB
In this article, we will explore how to deploy an application built with Platformatic DB, a powerful tool that simplifies API development by automatically generating OpenAPI and GraphQL documentation from your database, using Platformatic Cloud. Over...
-
How to Deploy a TypeScript Application to Platformatic Cloud
Platformatic Cloud offers developers a robust cloud environment for deploying Node.js applications. With pull request previews, rapid deploys, and built-in observability, it ensures a smooth development-to-production workflow and provides a robust fo...
Fascinating articles in the wild
- Breaking: AWS Begins Charging For Public IPv4 Addresses
- HashiCorp adopts Business Source License
- HashiCorp’s New Licensing
- Vite
- Red Hat’s commitment to open source: A response to the git.centos.org changes
- Hijacking S3 Buckets: New Attack Technique
- Node.js Performance Analysis Without Changing Your Code
- The growing pains of database architecture
- Prevent Attacks and Redirect Users with OAuth 2.0 State Parameters
- Fundamentals of Fastify: The Best Node.js Framework - read this introductionary article about Fastify!