Monthly Archives: مهر 1395

How Developers use Node.js – Survey Results

RisingStack, the provider of Trace - a next-gen Node.js debugging and performance monitoring solution and silver member of the Node Foundation conducted a survey during 2016 Summer to find out how developers use Node.js and what technologies they prefer with it. This article summarizes the results.

The results show that MongoDB, RabbitMQ, AWS, Jenkins, Docker and Amazon Container Services are the go-to choices for developing, containerizing and shipping Node.js applications.

The survey also let us find out various aspects of developing Node.js and choices for async control flow, debugging, continuous integration or finding packages. The results also tell Node developers major pain-point: debugging.

The survey was open for 35 days from 11 July until 15 August 2016. During this period, 1126 Node.js developers completed it. 55% of them have more than two years of Node.js experience, while 26% uses Node between one and two years. 20% works at a company that is publicly traded, 7% at a Fortune 500 enterprise.

Technologies used with Node.js

MongoDB became the go-to database

Node.js Survey - What databases are you using? MongoDB wins.

Node.js Survey - What databases are you using? MongoDB wins.

According to the results, MongoDB is clearly the go-to database for Node.js developers. Roughly ⅔ of our respondents claimed that they use MongoDB with their Node.js applications. It's also worth noticing that the popularity of Redis is massively increasing with the experience of Node engineers. This trend is also true in the case of PostgreSQL and ElasticSearch.

Node.js Survey - Database usage and developer experience

Node.js Survey - Database usage and developer experience

Node.js Survey - What do you use for caching? Redis wins.

Node.js Survey - What do you use for caching? Redis wins.

Half of our respondents said that they are using Redis for caching, but a staggering 45% stated that they don’t use any. Cross referencing the answers with developer experience allows us to see that the popularity of Redis is quite high amongst long-time Node users, compared to engineers with less than one year of Node.js experience.

Node.js Survey - Caching usage and developer experience

Node.js Survey - Caching usage and developer experience

The popularity of messaging systems is still low

According to our survey, 58% of Node.js developers don't use any messaging systems. This means that either developers are rarely using messaging in their microservices system, they use REST API-s or they don’t have a sophisticated system in place.

Node.js Survey - What messaging systems are you using? RabbitMQ wins.

Node.js Survey - What messaging systems are you using? RabbitMQ wins.

Those who use Messaging systems answered that they mostly use RabbitMQ (24% of all respondents). If we only investigate the responses of people who use messaging systems, RabbitMQ beats the rest of the existing solutions by far.

Node.js Survey - Messaging system usage

Node.js Survey - Messaging system usage

Node.js apps are most likely running on AWS

According to our survey, 43% of Node.js developers use AWS for running their applications, but running an own datacenter is popular as well (34%) especially amongst enterprises (nearly 50% of them has own datacenters) - but this is no surprise.

Node.js Survey - Where do you run your Node.js apps? AWS.

Node.js Survey - Where do you run your Node.js apps? AWS.

What’s interesting though is that Heroku and DigitalOcean are competing neck and neck to become the second biggest cloud platform for Node.js. According to our data, DigitalOcean is more popular with smaller companies (under 50) while Heroku stays strong as an enterprise solution as well.

Node.js Survey - Running apps and company size

Node.js Survey - Running apps and company size

Currently, Docker containers are the go-to solution for most of Node.js developers (47% of all respondents claimed to use it - which is 73% of all container tech users in the survey). Docker seems to be equally popular within all company sizes - but advanced developers appear to be using it much more (the ones with over one year’s of experience).

Node.js Survey - What container techs or VMs are you using? Docker.

Node.js Survey - What container techs or VMs are you using? Docker.

64% of the respondents said that they use some container technology - which means that the popularity of containers rose since the last major Node.js survey from 45% with a significant 20% increase since January 2016.

Node.js Survey - Container techs and developer experience.

Node.js Survey - Container techs and developer experience.

Amazon Container Service is the first choice for running containers

Node.js Survey - How do you run your containers? Amazon Container Service wins.

Node.js Survey - How do you run your containers? Amazon Container Service wins.

While Amazon Container Service leads as the choice of running containers with Node.js, it’s worth noting that Kubernetes is already on 25% according to our survey, and it seems to be popular especially with enterprise Node.js developers.

Node.js Survey - Running Amazon and Kubernetes with company size

Node.js Survey - Running Amazon and Kubernetes with company size

Node.js development

Configuration files are being used more often than environmental variables

The majority of Node developers (59% vs. 38%) prefer config files over credentials. Only 29 respondents (3%) stated that they use both.

Node.js Survey - Environment variables or config files? Config files wins.

Node.js Survey - Environment variables or config files? Config files wins.

Using only configuration files suggests a possible security problem since it implies that credentials are stored in the repositories. If you have the credentials to productions systems in GitHub, you can quickly run into trouble with rogue developers.

Using Env vars is highly recommended for secrets - while developers can still use config files in general.

Promises lead with async control flow

In Node.js - most of the core libraries are working with callbacks. The results show that Node.js users are leaning towards using promises right now.

Node.js Survey - What do you use for async control flow? Promises wins.

Node.js Survey - What do you use for async control flow? Promises wins.

Around half a year ago there was a pull-request in the core Node.js repository asking for async functions to return a native Promise. The answer for this was: “A Promises API doesn’t make sense for core right now because it's too early in the evolution of V8-based promises and their relationship to other ES* features. There is tiny interest within the TC in exploring this in the core in the short-term.”

Maybe it’s time to revisit the issue - since the demand is present.

Developers trust the console.log for debugging

Console.log is leading the race amongst other debugging solutions like the Node Inspector, the Built-in debugger and the debug module. Around ¾ of Node developers use it for finding errors in their applications - while much-sophisticated solutions are available as well.

Node.js Survey - How do you debug your applications? Using the console.log

Node.js Survey - How do you debug your applications? Using the console.log

A closer look at the data lets us know that more experienced developers are leaning towards the Node Inspector and the Debug Module as well.

Node.js Survey - Debugging applications and developer experience

Node.js Survey - Debugging applications and developer experience

According to the responses in our survey, only ¼ of Node.js developers use APMs - application performance monitoring tools - to identify issues in their applications. Although, emerging trends in the dataset suggest that APM usage grows with company size and developer experience.

Node.js Survey - How do you identify issues in your app? Using logs.

Node.js Survey - How do you identify issues in your app? Using logs.

According to the answers in our survey, using shell scripts is the most popular way of pushing code to staging or production environments - but Jenkins clearly wins among continuous delivery and integration platforms so far, and is becoming more popular as company size increases.

Node.js Survey - What do you use to push code or containers? Shell scripts win.

Node.js Survey - What do you use to push code or containers? Shell scripts win.

Node.js developers rarely update dependencies

Frequently updating dependencies is highly recommended with Node.js applications - since around 15% of npm packages carry a known vulnerability & 76% of Node shops use vulnerable dependencies according to a recent survey.

Node.js Survey - How often do you update dependencies? Less frequently than a month.

Node.js Survey - How often do you update dependencies? Less frequently than a month.

Updating dependencies less frequently than every week exposes applications to severe attacks all the time. According to our survey, 45% of Node.js developers update dependencies less frequently than a month, and 27% of them update dependencies month-by-month. Only 28% answered that they update dependencies at least every week.

These numbers correlate neither with company size nor with developer experience.

Node.js developers Google for their packages

According to our survey, the majority of developers use Google to find packages and decide which one of them they should use. Although the popularity of the npmjs.org/npms.io search platforms is 56% amongst our respondents, the data shows that it goes up to almost 70% for the demographic group of experienced (more than four years of Node development)! Preference increases with experience in this case.

Node.js Survey - How do you decide what package to pick? People mostly Google for them.

Node.js Survey - How do you decide what package to pick? People mostly Google for them.

Junior Node.js developers don’t know what semantic versioning is

Although 71% of our respondents uses semantic versioning when publishing/consuming modules, this number should be higher in our opinion. Everyone should use semantic versioning since npm works with semver! Updating packages without using it can easily break Node.js applications.

Node.js Survey - Do you use semantic versioning? Mostly yes.

Node.js Survey - Do you use semantic versioning? Mostly yes.

If we dig deeper in the dataset we can see that around half of the Node developers with less than a year experience don’t know what semver is or don’t use it - while advanced developers are embracing it on a much higher level.

Node.js teams introduce new tools and technologies very fast

According to our survey 35% of Node developers can introduce new tech/tools/product to their companies in a few days, and 29% in just a few weeks.

Node.js Survey - How much time is needed to introduce new technologies, tools or products to your company? A few weeks.

Node.js Survey - How much time is needed to introduce new technologies, tools or products to your company? A few weeks.

If we investigate the data more thoroughly, a not-so-surprising pattern emerges which lets us know that the time needed to introduce new tech/tools is gradually increasing with the size of a company.

Debugging is the most severe pain-point for developing with Node.js

We also asked Node developers about what's their biggest pain points regarding development. The top answers were:

  • Debugging / Profiling / Performance Monitoring
  • Callbacks and Callback hell
  • Understanding Async programming
  • Dependency management
  • Lack of conventions/best practices
  • Structuring
  • Bad documentation
  • Finding the right packages

Conclusion

Developing Node.js is still an interesting and ever-changing experience. We'd like to thank for the the engineers who took their time with answering to our questions, and we hope that the information presented in the article is valuable for the whole Node community.

The full dataset is going to be released and linked in this blogpost in a few days.

Node.js at Scale – npm Publishing Tutorial

With Node.js at Scale we are creating a collection of articles focusing on the needs of companies with bigger Node.js installations, and developers who already learned the basics of Node.

In this second chapter of Node.js at Scale you are going to learn how to expand the npm registry with your own modules. This tutorial is also going to explain how versioning works.

Upcoming chapters for the Node.js at Scale series:

  • Using npm
  • Node.js Internals Deep Dive
    • The Event Loop
    • Garbage Collection
    • Writing Native Modules
  • Building
    • Structuring Node.js Applications
    • Clean Code
    • Handling Async
    • Event sourcing
    • Command Query Responsibility Segregation
  • Testing
    • Unit testing
    • End-to-end testing
  • Node.js in Production
    • Monitoring Node.js Applications
    • Debugging Node.js Applications
    • Profiling Node.js Applications
  • Microservices
    • Request Signing
    • Distributed Tracing
    • API Gateways

When writing Node.js apps, there are so many things on npm that can help us being more productive. We don't have to deal with low-level things like padding a string from the left because there are already existing modules that are (eventually) available on the npm registry.

Where do these modules come from?

The modules are stored in a huge registry which is powered by a CouchDB instance.

The official public npm registry is at https://registry.npmjs.org/. It is powered by a CouchDB database, which has a public mirror at https://skimdb.npmjs.com/registry. The code for the couchapp is available at https://github.com/npm/npm-registry-couchapp.

How do modules make it to the registry?

People like you write them for themselves or for their co-workers and they share the code with their fellow JavaScript developers.

When should I consider publishing?

  • If you want to share code between projects,
  • if you think that others might run into the very same problem and you'd like to help them,
  • if you have a bit (or even more) code that you think you can make use of later.

Creating a module

First let's create a module: npm init -y should take care of it, as you've learned in the previous post.

{
  "name": "npm-publishing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/author/modulename"
  },
  "bugs": {
    "url": "https://github.com/caolan/async/issues"
  },
  "license": "ISC"
}

Let's break this down really quick. These fields in your package.json are mandatory when you're building a module for others to use.

First, you should give your module a distinct name because it has to be unique in the npm registry. Make sure it does not collide with any trademarks out there! main describes which file will be returned when your users do a require('modulename'). You can leave it as default or set it to any file in your project, but make sure you actually point it to a valid filename.

keywords should also be included because npm is going to index your package based on those fields and people will be able to find your module if they search those keywords in npm's search, or in any third party npm search site.

author, well obviously that's going to be you, but if anyone helps you develop your project be so kind to include them too! :) Also, it is very important to include where can people contact you if they'd like to.

In the repository field, you can see where the code is hosted and the bugs section tells you where can you file bugs if you find one in the package. To quickly jump to the bug report site you can use npm bug modulename.

#1 Licensing

Solid license and licenses adoption helps Node adoption by large companies. Code is a valuable resource, and sharing it has it's own costs.

Licensing is a really hard, but this site can help you pick one that fits your needs.

Generally when people publish modules to npm they use the MIT license.

The MIT License is a permissive free software license originating at the Massachusetts Institute of Technology (MIT). As a permissive license, it puts only very limited restriction on reuse and has therefore an excellent license compatibility.

#2 Semantic Versioning

Versioning is so important that it deserves its own section.

Most of the modules in the npm registry follow the specification called semantic versioning. Semantic versioning describes the version of a software as 3 numbers separated by "."-s. It describes how this version number has to change when changes are made to the software itself.

Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards-compatible manner, and
  • PATCH version when you make backwards-compatible bug fixes.

Additional labels for the pre-release and the build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

These numbers are for machines, not for humans! Don't assume that people will be discouraged from using your libraries when you often change the major version.

You have to start versioning at 1.0!

Most people think that doing changes while the software is still in "beta" phase should not respect the semantic versioning. They are wrong! It is really important to communicate breaking changes to your users even in beta phase. Always think about your users who want to experiment with your project.

#3 Documentation

Having a proper documentation is imperative if you’d like to share your code with others. Putting a README.md file in your project’s root folder is usually enough, and if you publish it to the registry npm will generate a site like this one. It's all done automatically and it helps other people when they try to use your code.

Before publishing, make sure you have all documentation in place and up to date.

#4 Keeping secret files out of your package

Using a specific file called .npmignore will keep your secret or private files from publishing. Use that to your advantage, add files to .npmignore that you wish to not upload.

If you use .gitignore npm will use that too by default. Like git, npm looks for .npmignore and .gitignore files in all subdirectories of your package, not only in the root directory.

#5 Encouraging contributions

When you open up your code to the public, you should consider adding some guidelines for them on how to contribute. Make sure they know how to help you dealing with software bugs and adding new features to your module.

There are a few of these available, but in general you should consider using github's issue and pull-request templates.

npm publish

Now you understand everything that's necessary to publish your first module. To do so, you can type: npm publish and the npm-cli will upload the code to the registry.

Congratulations, your module is now public on the npm registry! Visit
www.npmjs.com/package/yourpackagename for the public URL.

If you published something public to npm, it's going to stay there forever. There is little you can do to make it non-discoverable. Once it hits the public registry, every other replica that's connected to it will copy all the data. Be careful when publishing.

I published something that I didn't mean to.

We're human. We make mistakes, but what can be done now? Since the recent leftpad scandal, npm changed the unpublish policy. If there is no package on the registry that depends on your package, then you're fine to unpublish it, but remember all the replicas will copy all the data so someone somewhere will always be able to get it. If it contained any secrets, make sure you change them after the act, and remember to add them to the .npmignore file for the next publish.

Private Scoped Packages

If you don't want or you're not allowed to publish code to a public registry (for any corporate reasons), npm allows organizations to open an organization account so that they can push to the registry without being public. This way you can share private code between you and your co-workers.

Further read on how to set it up: https://docs.npmjs.com/misc/scope

npm enterprise

If you'd like to further tighten your security by running a registry by yourself, you can do that pretty easily. npm has an on-premise version that can be run behind corporate firewalls. Read more about setting up npm enterprise.

Build something!

Now that you know all these things, go and build something. If you’re up for a little bragging, make sure you tweet us (@risingstack) the name of the package this tutorial helped you to build! If you have any questions, you’ll find me in the comments.

Happy publishing!

How Developers use Node.js – Survey Results

RisingStack, the provider of Trace - a next-gen Node.js debugging and performance monitoring solution and silver member of the Node Foundation conducted a survey during 2016 Summer to find out how developers use Node.js and what technologies they prefer with it. This article summarizes the results.

The results show that MongoDB, RabbitMQ, AWS, Jenkins, Docker and Amazon Container Services are the go-to choices for developing, containerizing and shipping Node.js applications.

The survey also let us find out various aspects of developing Node.js and choices for async control flow, debugging, continuous integration or finding packages. The results also tell Node developers major pain-point: debugging.

The survey was open for 35 days from 11 July until 15 August 2016. During this period, 1126 Node.js developers completed it. 55% of them have more than two years of Node.js experience, while 26% uses Node between one and two years. 20% works at a company that is publicly traded, 7% at a Fortune 500 enterprise.

Technologies used with Node.js

MongoDB became the go-to database

Node.js Survey - What databases are you using? MongoDB wins.

Node.js Survey - What databases are you using? MongoDB wins.

According to the results, MongoDB is clearly the go-to database for Node.js developers. Roughly ⅔ of our respondents claimed that they use MongoDB with their Node.js applications. It's also worth noticing that the popularity of Redis is massively increasing with the experience of Node engineers. This trend is also true in the case of PostgreSQL and ElasticSearch.

Node.js Survey - Database usage and developer experience

Node.js Survey - Database usage and developer experience

Node.js Survey - What do you use for caching? Redis wins.

Node.js Survey - What do you use for caching? Redis wins.

Half of our respondents said that they are using Redis for caching, but a staggering 45% stated that they don’t use any. Cross referencing the answers with developer experience allows us to see that the popularity of Redis is quite high amongst long-time Node users, compared to engineers with less than one year of Node.js experience.

Node.js Survey - Caching usage and developer experience

Node.js Survey - Caching usage and developer experience

The popularity of messaging systems is still low

According to our survey, 58% of Node.js developers don't use any messaging systems. This means that either developers are rarely using messaging in their microservices system, they use REST API-s or they don’t have a sophisticated system in place.

Node.js Survey - What messaging systems are you using? RabbitMQ wins.

Node.js Survey - What messaging systems are you using? RabbitMQ wins.

Those who use Messaging systems answered that they mostly use RabbitMQ (24% of all respondents). If we only investigate the responses of people who use messaging systems, RabbitMQ beats the rest of the existing solutions by far.

Node.js Survey - Messaging system usage

Node.js Survey - Messaging system usage

Node.js apps are most likely running on AWS

According to our survey, 43% of Node.js developers use AWS for running their applications, but running an own datacenter is popular as well (34%) especially amongst enterprises (nearly 50% of them has own datacenters) - but this is no surprise.

Node.js Survey - Where do you run your Node.js apps? AWS.

Node.js Survey - Where do you run your Node.js apps? AWS.

What’s interesting though is that Heroku and DigitalOcean are competing neck and neck to become the second biggest cloud platform for Node.js. According to our data, DigitalOcean is more popular with smaller companies (under 50) while Heroku stays strong as an enterprise solution as well.

Node.js Survey - Running apps and company size

Node.js Survey - Running apps and company size

Currently, Docker containers are the go-to solution for most of Node.js developers (47% of all respondents claimed to use it - which is 73% of all container tech users in the survey). Docker seems to be equally popular within all company sizes - but advanced developers appear to be using it much more (the ones with over one year’s of experience).

Node.js Survey - What container techs or VMs are you using? Docker.

Node.js Survey - What container techs or VMs are you using? Docker.

64% of the respondents said that they use some container technology - which means that the popularity of containers rose since the last major Node.js survey from 45% with a significant 20% increase since January 2016.

Node.js Survey - Container techs and developer experience.

Node.js Survey - Container techs and developer experience.

Amazon Container Service is the first choice for running containers

Node.js Survey - How do you run your containers? Amazon Container Service wins.

Node.js Survey - How do you run your containers? Amazon Container Service wins.

While Amazon Container Service leads as the choice of running containers with Node.js, it’s worth noting that Kubernetes is already on 25% according to our survey, and it seems to be popular especially with enterprise Node.js developers.

Node.js Survey - Running Amazon and Kubernetes with company size

Node.js Survey - Running Amazon and Kubernetes with company size

Node.js development

Configuration files are being used more often than environmental variables

The majority of Node developers (59% vs. 38%) prefer config files over credentials. Only 29 respondents (3%) stated that they use both.

Node.js Survey - Environment variables or config files? Config files wins.

Node.js Survey - Environment variables or config files? Config files wins.

Using only configuration files suggests a possible security problem since it implies that credentials are stored in the repositories. If you have the credentials to productions systems in GitHub, you can quickly run into trouble with rogue developers.

Using Env vars is highly recommended for secrets - while developers can still use config files in general.

Promises lead with async control flow

In Node.js - most of the core libraries are working with callbacks. The results show that Node.js users are leaning towards using promises right now.

Node.js Survey - What do you use for async control flow? Promises wins.

Node.js Survey - What do you use for async control flow? Promises wins.

Around half a year ago there was a pull-request in the core Node.js repository asking for async functions to return a native Promise. The answer for this was: “A Promises API doesn’t make sense for core right now because it's too early in the evolution of V8-based promises and their relationship to other ES* features. There is tiny interest within the TC in exploring this in the core in the short-term.”

Maybe it’s time to revisit the issue - since the demand is present.

Developers trust the console.log for debugging

Console.log is leading the race amongst other debugging solutions like the Node Inspector, the Built-in debugger and the debug module. Around ¾ of Node developers use it for finding errors in their applications - while much-sophisticated solutions are available as well.

Node.js Survey - How do you debug your applications? Using the console.log

Node.js Survey - How do you debug your applications? Using the console.log

A closer look at the data lets us know that more experienced developers are leaning towards the Node Inspector and the Debug Module as well.

Node.js Survey - Debugging applications and developer experience

Node.js Survey - Debugging applications and developer experience

According to the responses in our survey, only ¼ of Node.js developers use APMs - application performance monitoring tools - to identify issues in their applications. Although, emerging trends in the dataset suggest that APM usage grows with company size and developer experience.

Node.js Survey - How do you identify issues in your app? Using logs.

Node.js Survey - How do you identify issues in your app? Using logs.

According to the answers in our survey, using shell scripts is the most popular way of pushing code to staging or production environments - but Jenkins clearly wins among continuous delivery and integration platforms so far, and is becoming more popular as company size increases.

Node.js Survey - What do you use to push code or containers? Shell scripts win.

Node.js Survey - What do you use to push code or containers? Shell scripts win.

Node.js developers rarely update dependencies

Frequently updating dependencies is highly recommended with Node.js applications - since around 15% of npm packages carry a known vulnerability & 76% of Node shops use vulnerable dependencies according to a recent survey.

Node.js Survey - How often do you update dependencies? Less frequently than a month.

Node.js Survey - How often do you update dependencies? Less frequently than a month.

Updating dependencies less frequently than every week exposes applications to severe attacks all the time. According to our survey, 45% of Node.js developers update dependencies less frequently than a month, and 27% of them update dependencies month-by-month. Only 28% answered that they update dependencies at least every week.

These numbers correlate neither with company size nor with developer experience.

Node.js developers Google for their packages

According to our survey, the majority of developers use Google to find packages and decide which one of them they should use. Although the popularity of the npmjs.org/npms.io search platforms is 56% amongst our respondents, the data shows that it goes up to almost 70% for the demographic group of experienced (more than four years of Node development)! Preference increases with experience in this case.

Node.js Survey - How do you decide what package to pick? People mostly Google for them.

Node.js Survey - How do you decide what package to pick? People mostly Google for them.

Junior Node.js developers don’t know what semantic versioning is

Although 71% of our respondents uses semantic versioning when publishing/consuming modules, this number should be higher in our opinion. Everyone should use semantic versioning since npm works with semver! Updating packages without using it can easily break Node.js applications.

Node.js Survey - Do you use semantic versioning? Mostly yes.

Node.js Survey - Do you use semantic versioning? Mostly yes.

If we dig deeper in the dataset we can see that around half of the Node developers with less than a year experience don’t know what semver is or don’t use it - while advanced developers are embracing it on a much higher level.

Node.js teams introduce new tools and technologies very fast

According to our survey 35% of Node developers can introduce new tech/tools/product to their companies in a few days, and 29% in just a few weeks.

Node.js Survey - How much time is needed to introduce new technologies, tools or products to your company? A few weeks.

Node.js Survey - How much time is needed to introduce new technologies, tools or products to your company? A few weeks.

If we investigate the data more thoroughly, a not-so-surprising pattern emerges which lets us know that the time needed to introduce new tech/tools is gradually increasing with the size of a company.

Debugging is the most severe pain-point for developing with Node.js

We also asked Node developers about what's their biggest pain points regarding development. The top answers were:

  • Debugging / Profiling / Performance Monitoring
  • Callbacks and Callback hell
  • Understanding Async programming
  • Dependency management
  • Lack of conventions/best practices
  • Structuring
  • Bad documentation
  • Finding the right packages

Conclusion

Developing Node.js is still an interesting and ever-changing experience. We'd like to thank for the the engineers who took their time with answering to our questions, and we hope that the information presented in the article is valuable for the whole Node community.

The full dataset is going to be released and linked in this blogpost in a few days.

Node.js at Scale – npm Publishing Tutorial

With Node.js at Scale we are creating a collection of articles focusing on the needs of companies with bigger Node.js installations, and developers who already learned the basics of Node.

In this second chapter of Node.js at Scale you are going to learn how to expand the npm registry with your own modules. This tutorial is also going to explain how versioning works.

Upcoming chapters for the Node.js at Scale series:

  • Using npm
  • Node.js Internals Deep Dive
    • The Event Loop
    • Garbage Collection
    • Writing Native Modules
  • Building
    • Structuring Node.js Applications
    • Clean Code
    • Handling Async
    • Event sourcing
    • Command Query Responsibility Segregation
  • Testing
    • Unit testing
    • End-to-end testing
  • Node.js in Production
    • Monitoring Node.js Applications
    • Debugging Node.js Applications
    • Profiling Node.js Applications
  • Microservices
    • Request Signing
    • Distributed Tracing
    • API Gateways

When writing Node.js apps, there are so many things on npm that can help us being more productive. We don't have to deal with low-level things like padding a string from the left because there are already existing modules that are (eventually) available on the npm registry.

Where do these modules come from?

The modules are stored in a huge registry which is powered by a CouchDB instance.

The official public npm registry is at https://registry.npmjs.org/. It is powered by a CouchDB database, which has a public mirror at https://skimdb.npmjs.com/registry. The code for the couchapp is available at https://github.com/npm/npm-registry-couchapp.

How do modules make it to the registry?

People like you write them for themselves or for their co-workers and they share the code with their fellow JavaScript developers.

When should I consider publishing?

  • If you want to share code between projects,
  • if you think that others might run into the very same problem and you'd like to help them,
  • if you have a bit (or even more) code that you think you can make use of later.

Creating a module

First let's create a module: npm init -y should take care of it, as you've learned in the previous post.

{
  "name": "npm-publishing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/author/modulename"
  },
  "bugs": {
    "url": "https://github.com/caolan/async/issues"
  },
  "license": "ISC"
}

Let's break this down really quick. These fields in your package.json are mandatory when you're building a module for others to use.

First, you should give your module a distinct name because it has to be unique in the npm registry. Make sure it does not collide with any trademarks out there! main describes which file will be returned when your users do a require('modulename'). You can leave it as default or set it to any file in your project, but make sure you actually point it to a valid filename.

keywords should also be included because npm is going to index your package based on those fields and people will be able to find your module if they search those keywords in npm's search, or in any third party npm search site.

author, well obviously that's going to be you, but if anyone helps you develop your project be so kind to include them too! :) Also, it is very important to include where can people contact you if they'd like to.

In the repository field, you can see where the code is hosted and the bugs section tells you where can you file bugs if you find one in the package. To quickly jump to the bug report site you can use npm bug modulename.

#1 Licensing

Solid license and licenses adoption helps Node adoption by large companies. Code is a valuable resource, and sharing it has it's own costs.

Licensing is a really hard, but this site can help you pick one that fits your needs.

Generally when people publish modules to npm they use the MIT license.

The MIT License is a permissive free software license originating at the Massachusetts Institute of Technology (MIT). As a permissive license, it puts only very limited restriction on reuse and has therefore an excellent license compatibility.

#2 Semantic Versioning

Versioning is so important that it deserves its own section.

Most of the modules in the npm registry follow the specification called semantic versioning. Semantic versioning describes the version of a software as 3 numbers separated by "."-s. It describes how this version number has to change when changes are made to the software itself.

Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards-compatible manner, and
  • PATCH version when you make backwards-compatible bug fixes.

Additional labels for the pre-release and the build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

These numbers are for machines, not for humans! Don't assume that people will be discouraged from using your libraries when you often change the major version.

You have to start versioning at 1.0!

Most people think that doing changes while the software is still in "beta" phase should not respect the semantic versioning. They are wrong! It is really important to communicate breaking changes to your users even in beta phase. Always think about your users who want to experiment with your project.

#3 Documentation

Having a proper documentation is imperative if you’d like to share your code with others. Putting a README.md file in your project’s root folder is usually enough, and if you publish it to the registry npm will generate a site like this one. It's all done automatically and it helps other people when they try to use your code.

Before publishing, make sure you have all documentation in place and up to date.

#4 Keeping secret files out of your package

Using a specific file called .npmignore will keep your secret or private files from publishing. Use that to your advantage, add files to .npmignore that you wish to not upload.

If you use .gitignore npm will use that too by default. Like git, npm looks for .npmignore and .gitignore files in all subdirectories of your package, not only in the root directory.

#5 Encouraging contributions

When you open up your code to the public, you should consider adding some guidelines for them on how to contribute. Make sure they know how to help you dealing with software bugs and adding new features to your module.

There are a few of these available, but in general you should consider using github's issue and pull-request templates.

npm publish

Now you understand everything that's necessary to publish your first module. To do so, you can type: npm publish and the npm-cli will upload the code to the registry.

Congratulations, your module is now public on the npm registry! Visit
www.npmjs.com/package/yourpackagename for the public URL.

If you published something public to npm, it's going to stay there forever. There is little you can do to make it non-discoverable. Once it hits the public registry, every other replica that's connected to it will copy all the data. Be careful when publishing.

I published something that I didn't mean to.

We're human. We make mistakes, but what can be done now? Since the recent leftpad scandal, npm changed the unpublish policy. If there is no package on the registry that depends on your package, then you're fine to unpublish it, but remember all the replicas will copy all the data so someone somewhere will always be able to get it. If it contained any secrets, make sure you change them after the act, and remember to add them to the .npmignore file for the next publish.

Private Scoped Packages

If you don't want or you're not allowed to publish code to a public registry (for any corporate reasons), npm allows organizations to open an organization account so that they can push to the registry without being public. This way you can share private code between you and your co-workers.

Further read on how to set it up: https://docs.npmjs.com/misc/scope

npm enterprise

If you'd like to further tighten your security by running a registry by yourself, you can do that pretty easily. npm has an on-premise version that can be run behind corporate firewalls. Read more about setting up npm enterprise.

Build something!

Now that you know all these things, go and build something. If you’re up for a little bragging, make sure you tweet us (@risingstack) the name of the package this tutorial helped you to build! If you have any questions, you’ll find me in the comments.

Happy publishing!