Solving the 'exit' problem of Pino transports and many more Adventures in Nodeland - Issue #11
Hi Everyone! Another week has passed and it came with so many important Open Source contributions in the communities I participate in: Pino, Fastify and Node.js Core. Check them out!
Pino
I already wrote extensively of the new transport system in Pino@7 that we are developing at https://github.com/pinojs/pino/pull/1003: it will allow you to create in-process transport similar to Winston or Bunyan, with no overhead on the main thread, as they will run inside a worker thread.
One of the biggest challenges of this approach is handling the shutdown of the process correctly. Two cases cause Node.js to exit: it can exit naturally if there are no more asynchronous things to do or manually by calling `process.exit(code)`. For the first case, we are calling unref() on the Worker thread to let the process exit even if the transport thread is running. In the second case, we thought of adding a handler to the `‘exit’` event to gracefully shut down the worker and ensure that all log messages are delivered independently of the status of the main thread. However, this approach had a significant obstacle to overcome: the `‘exit’` handler must be synchronous.
To achieve this, I updated the thread-stream `.end()` method to close the stream synchronously - this relies on work my friend mafintosh did a long time ago on Node.js stream: the autoDestroy mode (https://github.com/nodejs/node/pull/22795). Thanks to autoDestroy, we can guarantee that all streams emit a `‘close’` event when the underlining resource is fully closed. This setting became `true` by default in Node v12 onwards, while in readable-stream, it is still false by default.
Finally, I developed a demo for my upcoming talk at JSNation, where I showed how to use ElasticSearch with the new transport system. I had to make minimal changes to the transport: https://github.com/pinojs/pino-elasticsearch/pull/82.
This would not have been possible without the help of David Mark Clements, who helped me out when I was stuck!
Fastify
My colleague Simone Busoli fixed fastify-multipart on Node.js v16. While the change itself looks trivial - one line removed - it was actually really hard to understand why the plugin stopped working on the latest and greatest of Node.js, and it might be possibly relevant to you.
|
|
|
fastify-static got a significant update as well with the support of pre-compressed assets! Thanks to this change, you will now be able to avoid compressing assets on the fly, saving precious CPU cycles.
|
|
|
The long-time Fastify collaborator Manuel Spigolon has been working relentlessly to improve the data validation/serialization support inside Fastify to make it more flexible and maintainable at the same time. The introduction of the ajv-compiler dependency allows us to remove the direct dependency to Ajv, making it fully swappable. Check out the PR, the change is coming to a Fastify release this week.
Node.js
I’m so grateful for Stephen Belanger’s work to improve the performance of async_hooks in Node.js. Thanks to this change, the overhead in handling promises was significantly reduced from 82% to “just” 35%. This is part of the plan that we drafted at Diagnostics Summit back in https://docs.google.com/document/d/1g8OrG5lMIUhRn1zbkutgY83MiTSMx-0NHDs8Bf-nXxM/edit
|
|
|
Like all new things, we spotted a bug in this new optimization, as it breaks when used inside Jest: https://github.com/fastify/fastify-request-context/issues/49. In fact, this is problematic even outside of Jest: https://github.com/nodejs/node/issues/38781.
Thanks
This newsletter has reached 800 subscribers! I’m really happy you all find this interesting, keep the comments coming as they are deeply motivating.
I would also like to thank Glen Keane for proofreading my rumblings!
See you all next week!