StreamBrk & StreamFuz

I've been working for some time (very, very part time) on a small project, a streaming file transfer service in Node.js. Among the things that I've needed have been a simple way to a) break a large incoming file into small parts and b) take those small parts and merge them down into a large outgoing file. I've just rewritten much of that code and externalized it as two small open source packages. StreamFuz and StreamBrk.

StreamFuz

StreamFuz is a simple stream merger. It's slightly more than just that, though - there are plenty of other tools that will simply take some streams and concat them. StreamFuz is useful because it can parallelize and buffer the fetching of the smaller streams. My file-transfer service stores all of its file parts in Amazon S3. When we go to merge those parts into a whole file, we fetch the parts from S3. If we were forced to do that in sequence, the performance would suffer - hence, parallelizability.

Here's how to install:

npm install streamfuz

Check GitHub - StreamFuz for more detailed usage instructions.

StreamBrk

StreamBrk does the reverse of what StreamFuz does. Given a large incoming stream (say, the data from a POST'ed form), StreamBrk will take that data and split it up into multiple smaller streams of a given max-size.

StreamBrk doesn't really have any built-in parallelization, but the way in which StreamBrk requests a new part stream allows you to build in parallelization easily enough (my service does that).

Here's how to install:

npm install streambrk

Check GitHub - StreamBrk for more detailed usage instructions.

Future

I'm going to maintain StreamFuz and StreamBrk, in the hopes that they're useful to someone else. I'll keep updating them in npm. If you decide to make use of them, and have any issues, don't hesitate to submit an issue on github.