Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
feat: add support for native brotli and iltorb #158
Conversation
updated the README to reflect the new flag to opt in to brotli restored v10.15 to the travis build list. This is the last LTS version that does not support brotli in zlib, so it is important to test is, as well as 10.16 (where brotli was added to zlib)
|
What became of this? It seemed like you were there, just a couple of conflicts but because it's conflicting it looks to have been ignored for over a year while others chase -- potentially -- less elegant solutions. I especially liked your use of iltorb which is currently deprecated in favor of zlib but it was a good idea when you originally did it. :( |
|
@waffledonkey unfortunately the conflicts were a small part of the problem. Even iltorb requires a higher version of node than the express compression library currently supports. I still would like a compression middleware which supports brotli, but I think the better solution at this point is to create a separate middleware project which operates in a similar way, but does not have to maintain older node compatibility. To that end I started "compress-ts" as a Typescript repo in my own account, but life has gotten in the way of progress on that front lately. I will try and find some time soon to continue that work. |
|
Cool thanks. I'll stick with shrink-ray-current and deal with what is being described as a memory leak but I suspect might be cache growth. shrink-ray-current is a 'maintained' (quotes intentional) fork of shrink-ray which itself was based on compression so there might be an easy path forward in there. The one thing that is clever about shrink-ray is that it returns the first request for a thing with light compression and then after the response is sent it maximally compresses and caches the resource for any subsequent requests. How well that works obviously depends on if you burn the first request seeding a CDN, etc. but kinda neat nonetheless. Stay safe |
I know there have been a couple pull requests so far to add brotli compression, but it appears they have all stalled due to a lack of support for brotli in zlib < v10.16
This change adds support for native brotli where available, and polyfills that support with
iltorbif needed. I added unit tests to ensure this is working and not disrupting any other flows, but let me know if I need to add any more.Brotli support is disabled by default, but you can opt-in with this option:
enableBrotli: truein the top level options object. Additionally,useIltorbwas added as a method of testing compatibility between the 2 methods, without requiring environment changes. That option likely will not be terribly useful to an end-user, but rather exists for test cases.