A few thoughts about composer and how people use it

A few thoughts about composer and how people use it

composer has changed the PHP ecosystem like no other tool introduced – almost everyone is using it today. Now, I have written about composer before, and have always been a big proponent of using it. However, as I have spend some time with looking more closely on a few things, there is a few problems (some with composer, some with how people (ab)use composer) that I would like to write about.

I’m pretty certain that my point of view is not the only valid one, and that some of you will disagree with what I say – use the comments if you want to add something or have questions, and if you blog a repsonse, please use a pingback so I’ll notice your post.

Please keep in mind: this is not a rant against composer. You should be using it.

Number 1 – composer gets slow and resource hungry

composer builds up quite huge dependency graphs to find out which packages to install, which means projects that depend on packages that have a lot of dependencies with history, and lax to no restraints on those 2nd level dependencies, will cause composer update to eat tons of ram, and take quite a while.

One option to “fix” that is to add your 2nd, 3rd etc. dependencies to your root composer.json, with tighter version contraints. While this actually limits the problem to some extend, its not a real fix though, as it means sooner or later you end up doing composers job manually yourself.

Another option is to turn of your memory limit, and keep supplying more memory to your machines, however I’ve been told that this breaks for windows users. Also it gets quite cumbersome when a composer update starts taking longer than a cigarette/coffee break.

Thats not the only thing that keeps it slow – composer is retrieving meta-data (and the contents) in a blocking manner, which means if one request is slow the whole process will be held up. Using something like reactphp could actually decrease the time composer needs for those tasks.

Number 2 – people are using composer as an installer

Something I did wrong myself when I started working with composer. Not only does composer make it easy to be abused in that manner, having the option to have composer check your php and your php extensions makes it seem like a logical step.

However, that is problematic for various reasons:

  • composer is a developer tool, as such it shouldn’t run on live machines
  • composer will fetch dependencies from a lot of sources, which means that your live systems would need to be able to fetch from there (ask your admin what he thinks about that), and to not get rate limited on github for example, you would need to put credentials on your live machine…
  • if you don’t backup your dependencies (and when using composer as an installer you don’t have an explicit step for that) you might get a bad surprise when someone decides to remove his project. Packagist only stores meta-data, if someone removes the git or the dist zip from github (or any other repository) you can only pray that you have cached it somewhere. I’ve seen a hand full of packages on packagist that are not available on github anymore.
  • your deployment becomes dependend on systems that are not under your control – if github gets ddosed again, or packagists provider fucks up routing again during deployment, you might end up with a much longer maintenance period that anticipated.

Let me from my perspective mention how I think composer should be used:

  1. developer initializes a new projekt using composer create-project
  2. developer adds requires as needed, he runs composer update to create the composer.lock file, which is added to the scm. this developer at this time is responsible to verify that the version(s) he updated to are working as required.
  3. when pulling a changed composer.lock from scm, each developer working on it will run composer install to have the correct setup.
  4. once a release is necessary the person who holds the release manager role will create a build of the project (it does not matter if he runs a build script or elevates a build in your build system, in the end someone is responsible to decide when something should be released). Most likely within this build there will be done something like composer install --prefer-dist --no-dev -o the result of this step should be a package (directory, zip, tarball, .deb, or whatever else your deployment system needs).
  5. the admin uses the result of 4. to install the release on a live machine

now obviously this does not cover all variants, for example if your release is a library that you want to publish on github, then step 4 is the process where you tag your version number and step 5 is not relevant. Or you have more processes inbetween (staging/QA etc) but this should illustrate how responsibilites should be separated, and it should prevent you from getting in a half-installed-live-system-situation.

Number 3 – people use their own paths

Now, I’m pretty sure this is one that quite a few people will disagree with me.

Composer offers the ability to use custom installers, and even provies the composer/installers package, which provides an easy way to install packages into other folders.

Quite a few Frameworks and CMS make use of this, usually for one or two of three reasons

  1. to install assets somewhere
  2. to keep an own directory structure (for legacy reasons)
  3. to distinguish various types of packages (regular depedencies, modules, themes, plugins…you name it)

where 3. seems to be the most common usecase.
Obviously its conveniant, by slapping those dependencies into different directories, you can have your module loaders etc, very easy detect a new module, or a new plugin.

So why am i listing this as a problem?
because:

  • own folders lower the interoperatbility of packages (ok, granted, this is bean counting, your blogs theme wouldn’t work with my forum anyways)
  • own folders lower the transparency (vendor code is in vendors, own code in the project),
  • and raise the learning curve (ok, if you only use one framework constantly, you probably can remember 5 paths)

as you see, the argument against it is actually a bit weak, however to me those are still valid arguments, and there are better solutions (puli, assetic (for assets), your installer managing a configuration file vs. using a dir to find all modules) available.

Number 4 – people don’t adhere semver

Semantic Versioning actually makes a lot of sense – the version contraints in composer follow the assumption that semantic versioning is used, still people choose not to. Why? i don’t know. But if you release bc breaking changes in a PATCH release, then there is a special place in hell for you.

Number 5 – people don’t tag their releases / don’t release

This goes hand in hand with Number 6 – I’ve seen packages that are on packagist, without ever having a version tagged. Thing is, the moment you put it on packagist, in theory, you should have a first version. If you don’t tag it, people can’t properly set their dependencies, meaning they shouldn’t be using it.

Don’t be scared of releasing, release often, and Majors if needed.

Also, as i’ve seen this lately, there are people who use branches instead of tags – well at least thats how it looks to me (maybe someone wants to enlighten me for the reason), which means you end up with versions like “2.3.x-dev, 2.2.x-dev, 2.1.x-dev”, which means you will always be on unstable versions.

Number 6 – people release packages with dependencies to unstable versions

Do I even need to say something about this? To my surprise it is not exactly uncommon, and quite often a result of number 5. Oh the joys on you requiring a package that has a stable version, but relies on a dev-package, that itself relies on a dev-package from a -develop branch.

Those who know composer a bit better might already have spotted the hidden evil in this combination. Your root packages composer.lock is the only composer.lock that is relevant. When you run composer update on your root package you might end up with a completely different version of dev-randompackage-develop than the developer who build the dev-otherrandompackage, and yet another one than the one who build stablepackage – which makes for nasty heisenbugs.

So, what now? / TL;DR

As you can see the things i listed are quite different aspects.

Number 1: to tackle this problem it probably requires a major rewrite of composer, and maybe even rethinking of how to use it

Number 2: one can get around a part of this problem using toran proxy or a local satis install but in the end the real solution might be if there was more tutorials on how to get the installing part right.

Number 3: i named some better options already

Number 4,5,6: Thats us, all of us who publish packages, work on opensource projects etc. We have to show more responsibility and discipline here – and clean up the chaos we are having right now.

Advertisements

13 thoughts on “A few thoughts about composer and how people use it

  1. Are you sure that step 4 in your solution to Number 2 problem is correct? I just tried it and I still get a ReflectionException when trying to load the application. What I did is to run composer install –prefer-dist –no-dev -o as you write, zip the file and transfer it to another server. Loading the application from the new server does not work. This is an issue from my point of view, having to run composer at the production server. What if I don’t any ssh access to it? There must be some solution to this problem.

    1. Not sure if trolling, or just a language issue.
      The command is running an install with dist preferred, and without development dependencies. If this causes any form of exception then something with your dependencies or your code is not right. (or your environment, might be a difference between your build machine and your Webserver) in any case, you shouldn’t need to run composer on your live machine.

      1. No trolling.

        What I actually did is to copy a Yii 2 application that uses composer for its packages from my production server to mine at home (Linux -> Linux). Before compressing the whole application dir, I run the composer command you described, then zipped the directory and unzipped it in the other system. The application did not work. However, what I did not try is to first delete the contents of the vendor directory before running the composer command and copying it to the other system. Do you think this would make a difference?

  2. Number 7: Composer does not enforce security. Symfony shows how it should be done: http://fabien.potencier.org/article/73/signing-project-releases Composer just needs to check the gpg tags against the local keyring. And of course we need to force developers to sign their releases.

    Number 8: I can’t even find the documentation of the packagist protocol anymore. I read it once and was shoked how inefficient and needlessly complicated it is.

    Number 9: It’s not decentralized. There’s actually no reason why we need a central website like packagist. Composer could just as well go directly to the git repositories and use the URIs of the git repositories as identifiers. It’s not even necessary to depend on github. It can work with any git (or SVN) repositories. The ‘u’ in URI is for unique.

    Number 10: It’s written in PHP. Don’t we have enough package managers already? When will we see a package manager that works for multiple eco-systems? https://wiki.debian.org/PackageManagers

    1. 7: excellent point
      8: if by protocol you mean the api, you can simply add .json to any package on packagist to see its description. I’m not sure why you see it as needless complicated, infact i’d say that you can’t limit the replies content to versions only relevant to you (from what i can see), is a good sign for being to simple.
      9: I strongly disagree, at least with what you are saying – it is right that packagist being completely centralized is not exactly a good thing – on the other hand composer has the ability to handle multiple repositories and all that would be needed would be a few people setting up a few mirrors, to solve my issue with it.
      As for doing it through git/svn repositories without having a meta data repository, i think thats actually what would make it a lot more complex, and it would remove the ability to easy search and browse packages, or to build tools that use the metadata for further information.
      Bower in the JS world is a much more simplistic approach, where the central metadata is only linking to the git repository – which is still more than you are suggesting – and it already has various disadvantages over composer.
      10: I prefer specialized tools over an trying to do everything at once. Each language has its own quirks, and having a tool that works over multiple languages most likely makes the tool overly complex with very little benefit. If you have the need for dependencies in multiple languages, then your build tool should use the right dependency manager to combine those dependencies, as only you will know how they have to work together.

  3. @ppetermann Running composer install at the other system makes the application load fine afterwards, so this is definitely a composer issue. Anyway, thank you for your help.

  4. I’ll talk about number 5 (mambo?).
    People use branches when they need to maintain multiple versions. Older ones for back porting, new ones for alphas, while maintinaing a LTS/mainline version. Also, they’re often used to diff between them (2.1 – 2.2). They’re not exactly for releasing, but they can be used that way, yes.

    1. The problem is that if you have a 2.1 branch, and you don’t tag the actual releaes somewhere, then 2.1 = 2.1.x-dev, which makes no difference between 2.1.0 and 2.1.10.

      also you can diff against tags ;)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s