Welcome, visitor, to my weblog, or blog as they call it nowadays. This is my space to reflect, ramble, rant, ridicule, rampage, and relay about whatever or whomever I feel like; this is the one space where I can happily self-proliferate and merily make a fool of myself without any bad feelings.
I am aware that my blog is currently quite horrible to look at and that it lacks all sort of navigation abilities. I apologise. I hope to be able to fix this soon. In the mean time, please report any problems you may encounter. Thanks!
I've been trying for three weeks to live-stream the picture from the camera onto the local network. I have tried crtmpserver and vlc, read several dozens of how-tos, but so far I have not been able to get a streaming setup working, no matter what I tried.
Hence my plea to the lazy web: does anyone have such a setup running on top of Debian? Would you please let me know how you did it?
Thanks a lot!
NP: Eels: End Times
Salt has some very good ideas, for instance:
Configuration is just data in YAML format, which makes it really easy to whack up stuff — and also to generate it using other means;
There's a lot of plugability: runners, renderers, states, modules, returners, outputters, all of which can be replaced with custom implementations;
It builds on remote execution. In fact, in the docs, they talk first about remote execution and then about configuration management. Not only does this makes a lot of sense, it also gives the admin the additional benefit of having quick, remote access to all hosts in the infrastructure. I like that as it means I don't also have to deploy a parallel-execution solution.
Salt supports both, a push and a pull model for configuration management. In fact, the push model is part of its remote execution base and it's implemented through persistent connections from the clients to the master. This neatly solves all problems relating to firewalling and NAT, and it also means that the master always has a pretty good understanding of who's around. It feels good to me. And there is no NIH going on with respect to pulling: there is no daemon, you have to use
But there are also a couple of downsides to Salt:
The default templating engine is Jinja. While that may be an excellent choice for applications in inherently insecure environments, such as the Web, they are quite painful in the context of Salt: no multi-line logic, whitespace control is a nightmare, and a completely unfamiliar set of conditionals and filters, rather than access to e.g. Python. This is only the default, and mako is an alternative, but it seems like a poor choice and that does not instill confidence.
I don't think that the method of targeting hosts that Salt uses is scalable. I don't want to intersperse my configuration with if-then-else statements referencing hostnames (what would that mean if a hostname changed or a new host was added?). Instead, I want the configuration to be fully parametrised and assemble the parameters from a node database, ideally a hierarchical one. I have a solution for that which works with Salt, called reclass.
I don't think much of "pillars", which are Salt way to provide random data (parameters) to nodes, again because data needs to be targeted at hosts, rather than defined for a host. This is also solved with reclass.
Exceptions are not properly handled, the file descriptors aren't closed when daemonising, children are not reaped as they should, and the exit code will almost always be zero, even on failure.
State requirements are not enforced where they are required, but only come into play when a higher order function is called.
The code reinvents module loading and does that in very complex ways, while putting tight limitations on what's possible. State functions and modules cannot be namespaced, for instance, nor can they easily make use of each other. There is also no way to quickly define macros without resorting to the templating language (and thus a completely different syntax and paradigm), and the concept of having modular, self-contained states (e.g. one for
sudo, one for
SSH, etc.) with all their dependencies in one hierarchy is not really part of the design. Also, it's not trivial for a state definition to export an interface for other states to use.
While we're on the topic of code and design, there's a lot of duplicate stuff in need of refactoring (e.g. saltutil._sync, and it seems that this even causes a lot of crypto work to be done over and over again. It also doesn't help that
salt-call, which is used to execute commands on the clients (e.g. pull configuration), works completely independently of the daemon process running on each client (although the same code base is being used).
And why aren't state and execution modules implemented as classes? Instead, the module is itself treated as an entity, and a home-cooked "virtual" system is used to determine which platforms support which modules. Object-oriented programming principles gives you all of that, and more.
Fortunately, the Salt authors didn't cave in and write the entire communication layer themselves. Instead, they employed ZeroMQ, but it's still quite common for a client to become unreachable (permanently), e.g. due to an IP address change, or networking problems. What's worse is that the master does not keep track of who's listening.
While we're on the subject of standing on the shoulders of giants, Python's
msgpackapparently cannot handle Unicode. Yay.
Those are the big issues. There are many small issues too, but those won't be around for too long as the project is moving along quickly and the community is vibrant. This is surely an important point that speaks for Salt.
However, the above issues seem to hint at design choices that might well turn out to stand in the way later.
Following a day of frustration, I now feel the overpowering urge to write my own configuration management system, because of course I feel that I could do it better than everyone else. Does this sound familiar to you?
Let's just say — hypothetically — that I would, then I'd want to reuse as much existing functionality as possible. For instance, I'd want the entire remote execution framework to be independent from any configuration management implemented on top.
So what does this mean? What would such a remote execution framework need? Here are some thoughts:
I'd want to keep it the way of Salt, i.e. the clients maintain persistent connections to the master, and the master regularly pings the clients, for housekeeping. And the clients know to expect such pings at regular intervals, so can reset themselves in case they don't hear from the master. Or scream.
Of course, authentication and encryption need to be part of this. Ideally, key and roster management are already available in the tool that's being reused. I don't want to have to have a new PKI for this…
The protocol would probably be something like XMLRPC, with an extensible list of modules on the client to do the work. Data would be standardised to JSON-format.
Asynchronous execution would be a plus. I am even tempted to say that it should be all built asynchronously, even though I don't really know a use case for this.
The clients should be able to feed back information to the master, where data can be accumulated, allowing for cross-node configuration. This could be implemented using master-side polls to keep the protocol easier.
Doesn't this sound like a Unix botnet to you? ;)
I could imagine whacking this up with a bit of Python, some
socat and SSH: the server would have an
authorized_keys file with forced commands connecting
the client to the server process via sockets.
Or I could imagine using twisted for that.
But I would prefer if something like this already existed. Anyone?
Comments are broken on my blog, and I cannot be bothered to work on them. If you have any input, please write to me. I will (eventually) condense all feedback into a new article.
NP: Mouse on Mars: Parastrophics
Dear lazyweb, we need a new printer-scanner combination for our new office, ideally a laser printer and a high-resolution scanner (1200dpi). It must integrate nicely with CUPS and SANE, without requiring any proprietary, closed-licence drivers or plugins. It also has to come with an automatic document feeder in addition to the plain flatbed scanner.
Additional, would-be-nice features in decreasing order of preference are
- Colour printing (separate cartridges a must)
- Duplex printing (real duplex preferred over soft solutions)
- Direct scanning to PDF (distribution by E-mail, Samba or FTP)
I have investigated the Samsung CLX-6220FX, but the driver situation was such a nightmare (and the printer so loud), that we returned it right away. I should have read this article by Peter beforehand…
Does anyone have experience with these HP models and can recommend them for use with Debian stable?
Does anyone have another recommendation?
Please send me a message. Thanks!
I am moving into an office. And as any proud husband and father, I want to have photos of my wife and daughter on my desk. The year being 2012, I did not hike to the photo shop to get a print framed, but instead thought to myself that I should put a digital photo frame onto my table.
This idea exposed me to the ridiculous world of consumer electronics. It led me to conclude that digital picture frame designers need to be whacked with cluebats.
Step by step…
Once accepted, the thought of a digital photo frame developed into a product definition along the following lines: my idea photo frame would connect to my Wifi-network, and obtain the photos on-the-go from a folder exposed via HTTP or CIFS, and then go on to display them in random order, incorporating new photos as it encounters them.
With this in mind, I went to the shops, and since I believe in specialised retailers and want to support them, my first stop was Foto Sauter at Sendlinger Tor. Unfortunately, none of the frames they had came with Wifi, so I decided to look further. I vehemently oppose to the business practices of the Metro group, thus skipped Saturn and MediaMarkt, and eventually ended up at Conrad.
They had a frame with Wifi! I jumped for joy, until I read the manual: pictures can be obtained from Flickr and Picasa. Period.
All other models on the Internet seem to be similarly limited, including the new Sony S-frame.
The night before, Penny had researched the field a bit and came to the conclusion that the S-frame would be the best product available. This led me to scratch Wifi off my requirements list and get a model that would read photos off a USB stick.
I went back to the photo store and bought a "Sony S-frame", only to discover that it cannot show photos in random order. It has three viewing modes (single photo, collage, single photo with clock), … and a random mode, but guess what: the random mode randomly switches the viewing modes, which then display the photos in lexicographical order.
How stupid is that???
I returned the product and left the store after discovering that none of their products could do random playback.
I went back to Conrad and found an "Intenso MediaCreator" (what media does it create???), which displayed the photos seemingly randomly.
But at home I found out that the "random" order is always the same, probably because the bright engineer that programmed this thought it was better to sort filenames by last letter and call it random, than to figure out a way to roll a dice on the device.
I wrote to the support team and asked them. The response was that the desired functionality (random selection) is not possible and won't be made available.
So I am returning the product.
Would someone please tell me about a digital picture frame (8 inch or so) that can display images in random order, ideally loading them off a CIFS share via Wifi?
Or is it really the case that consumer electronics are completely useless these days, by which I mean that "consumers" have dumbed down so far to buy this crap?
Update: a lot of people wrote in suggesting to invest in a cheap Android tablet. Some suggested Raspberry Pis in USB host mode (emulating the USB stick and hence the source of the images, provided that the frame doesn't cache). Other suggestions included the Samsung SPF-85V which can display images according to an RSS feed but needs Microsoft for that (or maybe not), and the community-developed, Linux-based Joggler.
Regarding the non-random order on the Intenso frame, Paul Hedderly postulated that the order comes from the filesystems (FAT order) and can be changed by writing the files differently.
I've really had it with Puppet. I used to be able to put up with all its downsides
- Non-Unix approach to everything (own transport, self-made PKI, non-intuitive configuration language, a faint attempt at versioning (bitbucket), and much much more…)
- Abysmal slowness
- Lack of basic functionality (e.g. replace a line of text)
- Host management and configuration programming intertwined, lack of a high-level approach to defining functionality
- Horrific error messages
- Catastrophic upgrade paths
- Did I mention Ruby and its speed?
- Lack of IPv6 support
- [I could keep going…]
but now that my fourth attempt to upgrade my complex configuration from version 0.25.5 to version 2.7 failed due to a myriad of completely incomprehensible errors ("err: Could not run Puppet configuration client: interning empty string") and many hours were lost in trying to hunt these down using binary searches, I am giving up. Bye bye Puppet.
But I need an alternative. I want a system that is capable of handling a large number of hosts, but not so complex that one wouldn't put it to use for half a dozen machines. The configuration management system I want looks about as follows: It
- makes use of existing infrastructure (e.g. SSH transport and public keys, Unix toolchain, Debian package management and debconf)
- interacts with the package management system (Debian only in my case)
- can provision files whose contents might depend on context, particular machine data and conditionals. There should be a unified templating approach for static and dynamic files, with the ability to override the source of data (e.g. a default template used unless a template exists for a class of machine, or a specific hostname)
- can edit files on the target machine in a flexible and robust manner
- can remove files
- can run commands when files change
- can reference data from other machines (e.g. obtain the certificate fingerprint of each hosts that define me as their SMTP smarthost)
- can control running services (i.e. enable init.d scripts, check that a process is running
- is written in a sensible language
- is modular and easily extensible, ideally using a well-known language (e.g. Python!)
- allows to specify infrastructure with tags ("all webservers", "all machines in Zurich", "machines that are in Munich and receive mail"), but with the ability to override every parameter for a specific host
- should just do configuration management, and not try to take away jobs from monitoring software
- logs changes per-machine and collects data about applied configurations in a central location
- is configured using flat files that are human-readable so that the configuration may be stored in Git (e.g. YAML, not XML)
- can be configured using scripts in a flexible way
Since for me, Ruby is a downside of Puppet, I won't look at Chef, but from this page, I gleaned a couple of links: Ansible, Quattor, Salt, and bcfg2 (which uses XML though). And of course, there remains the ephemeral cfengine.
I haven't used cfengine since 2002, but I am not convinced it's worth a new look because it seems to be an academic project with gigantic complexity and a whole vernacular to its own. There is no doubt that it is a powerful solution, and the most mature of all of them, but it's far away from the Unix-like simplicity that I've come to love in almost 20 years of Debian.
Do correct me if I am wrong.
Ansible looks interesting. It seems rather bottom-up, first introducing a way to remotely execute commands on hosts, which you can then later extend/automate to manage the host configurations. It uses SSH for transport, and its reason-to-be made me want to look at it.
My ventures into the Ansible domain are not over yet, but I've put them on hold. First of all, it's not yet packaged for Debian (Ubuntu-PPA packages work on Debian squeeze and wheezy).
Second, I was put off a bit by its gratuitous use of the shell to run commands, as well as other design decisions.
Check this out: there are modules for the remote execution of
commands, namely "shell", "command", and "raw". The shell modules
should be self-explanatory; the command module
provides some idempotency, such as not running the command if a
file exists (or not). To do this, it creates a Python script in
/tmp on the target… and then executes that like
$SHELL -c /tmp/ansible/ansible-1350291485.22-74945524909437/command
Correct me if I am wrong, but there is zero need for this shell indirection. My attempts at finding an answer on IRC were met by user "daniel_hozac" with a reason along the lines of "it's needed, believe me", and on the mailing list, I am told that only the shell can execute a script by parsing the interpreter line at the top of the module.
Finally, the raw execution module also executes using the shell…
And there a few other design decisions that I can't quite
explain, around the command-line switch
--sudo — see
the aforementioned message…
In short: running a command like
ansible -v arnold.madduck.net -a "/usr/bin/apt-get update" --sudo
does not invoke
one might like; it invokes the shell that runs the Python script
that runs the command. Effectively therefore, you need to allow
sudo shell execution, and for proper automation, this
has to be possible without a password. And then you might just as
well allow root logins again.
The author seems to think that "core behaviour" is that sudo allows all execution and that limiting the commands to run is not a use-case that Ansible will support. Apparently, I was the first to ever suggest this.
There are always ways around (e.g. skip
sudo … as the command, simply ignore the
useless shell invocation and trust that your machine can handle it,
but when such design decisions remain incomprehensible and get
defended by the project people, then I am hesitant to invest more
time on principle.
Finally, I've looked at Salt, which is what I've spent most time on so far. From the discussions I started on host targeting and data collection, it soon became apparent that Salt is very thin and flexible, and that the user community is accomodating.
Unfortunately, Salt does not use SSH, but at least it reuses existing functionality (ZeroMQ). As opposed to the push/pull model, Salt "minions" interestingly maintain a persistent connection to the server (which is not yet very stable), and while non-root usage is still not unproblematic, at least there has already been work done in this direction.
I think I will investigate Salt more as it does look like it can do what I want. The YAML-based syntax does seem a bit brittle, but it's the best I've found so far.
NP: The Pineapple Thief: Someone Here is Missing
In the light of the recent announcement by the European central bank to bail out states without limits — which is breaking the very law that the EU was built upon – Italy's stock market supervisors have removed the restriction on short sales.
In Italy, you may now again sell stuff on the financial market that you don't have. The only condition is that you have to be able to prove that you could currently buy it. But that, of course, is not a guarantee for you to be able to buy the good/stock/whatever when the person you sold it to actually wants it, given the volatility of the markets.
I expect other countries to follow suit.
Currencies — and especially the Euro — was made by bankers for bankers to earn money. Who actually believes that the ESM, to which Germany enslaved itself this week would fix anything is simply naïve. Temporarily, the markets were on hold, how convenient that this coincided with summer break. Now everything is back to normal and the next financial crisis is being built.
NP: Porcupine Tree: Stupid Dream
Today was a black day for democracy in Germany. The German constitutional court ruled in favour of the European Stability Mechanism. In combination with last week's announcement by the European Central Bank to purchase government bonds without limits (breaking the No-Bail-Out clause at the core of their mandate more obviously and irreversably than ever before), the German people have lost a good deal of democracy today.
Why? — you may ask… because from now on, fiscal and financial policy will be made in Brussels, by people enjoying full immunity, but who are not elected democratically by the European people, let alone the Germans, and they will freely decide over who has to pay and be liable for whom.
I am talking about people like Klaus Regling, who was already involved the very first time the Maastricht Criteria were violated. He is now at the front of the largest and most powerful financial weapon ever conceived. With immunity.
And people like Mario Draghi, whom I would possibly call the most corrupt person I know. His announcement to save the Euro at whatever cost accidentally came only a day before his motherland Italy had to go to the market for more money and was able to place a bond at such ridiculously low interest rates that anyone who's kept up to speed with Italy's development had to rightfully ask how that was possible.
While in the past, for whatever reason, the European people have let the ECB get by saying that they are not bailing out countries when they buy bonds on the secondary market (wtf!), they have finally dropped that "restriction" (the law).
And as of today, the ESM is ready to go, along with the fiscal pact. Germany is now liable for more than quarter of all of the Eurozone's past and future debts.
And no citizen will be able to have any more influence in this, or reverse it. Budget, fiscal policy and currency control are forever gone.
Not that parliamentarian democracies were ever direct. Yet, in the past, one could at least vote for those people whose promises one was inclined to believe the most. You can still do that in the future, but those people won't be able to influence fiscal or financial policy anymore.
There is no way back. The ESM and its employees enjoy full immunity, and the ESM is forever-binding. There is no exit clause.
Thanks to the ECB's law breaking and the ESM, which I consider highly unconstitutional, at least in Germany, Eurozone-countries may refinance their debts at interest rates that are in no way related to their ability to pay back loans. All other countries — foremost Germany — are henceforth liable for others' debts. The fundamental rule of the EU that no country would have to stand up for another country, is gone with the wind.
Within an hour, the "markets" reacted. Germany, which previously had to "pay negative interest" (a sign of stability) saw interests on its bond shoot up. And Spain, Portugal, Greece and others who couldn't previously refinance their old debts, are now getting fresh money cheaper than ever.
Spain's president Rajoy today didn't even bother beating around the bush anymore, he's now going to apply for fresh money but won't bother with any saving schemes or other restructurings. Monti in Italy has suggested the same.
Wouldn't you take money if you were offered it for free, without the need to pay it back?
This is more than inflation, in my opinion. What is currently happening in Europe is active depreciation of individual wealth. Our heads of state are actively working against the people. The Euro has lost all credibility and everyone knows it. It is only a question of time until it will tremble and fall. Meanwhile, "the market" celebrates and continues their gambles while they still can, on the backs of our currency and our wealth.
Most affected are the people who have savings in Euros, whose life insurances are decreasing in worth and who cannot afford to diversify into other asset classes or currencies.
On the other hand, those who "let their money do the work" are being saved. Whoever previously invested into bonds of struggling states, hoping to reap massive interest gains, is now proven right. Brussels has eliminated the risk factor. What kind of message does this send???
Hands up if you thought that our politicians are even interested in closing the rapidly widening gap between rich and poor. Really? That's naive. The Eurozone is corrupt, and our currency has never been as virtual as today.
Nobody can say whether saving the Euro at all cost is the right thing and noone knows whether what's currently happening is just bad. I would have wished that our politicians had taken the crisis as an incentive to fix the system in the interest of the people and with a long-term focus:
Europe should have returned to the law it gave itself, without exceptions: The No-Bailout-Clause, as well as the Masstricht Criteria. In general, you cannot spend more than you earn, and this principle seems to have been forgotten entirely, and those who had most actively forgotten it, are being rewarded.
A shared currency requires a shared fiscal policy. This policy has to be in place before the currency, the currency can only ever be the result of a common strategy. The EU has done this the wrong way around and it failed. Trying to put a fiscal policy in place now is patchwork that is guaranteed to fail. And putting fiscal policy into the hands of undemocratically chosen people in Brussels is not representing the people.
But on the contrary! Europe's policiticans are making it crystal clear that the foundation upon which it was built, the laws and rules, the promises and guarantees, no longer apply. The people were not asked. The promises once made were broken. Our politicians have ruled over our heads.
More debts are being made, and more debts to pay off debts, and so on. It's long gotten out of control, now the process is institutionalised. I feel sorry for our kids. I find it irresponsible what is being done to them (in addition to the way we rape the environment).
I also feel deeply with the people in the struggling countries who are being screwed by the crisis and are not at fault. What our politicians are doing is unfortunately not going to help long term. The problems are just postponed, and with every day, the inevitable crash will be more painful. I am sorry.
Today is a black day for democracy. We have lost souvereignity. We have lost control over our currency. We have lost our budget rights.
And I have lost my faith in the last instance of the German government that I trusted. As of today, I know that the German constitutional court is nothing more than a puppet in the hands of the politicians (who are themselves puppets of Brussels and the banks).
The limit they imposed (Germany's liability must not increase beyond 190 billion Euros without the federal parliament's consent) is worthless. Soon the politicians will explain to us why it's inevitable that we must raise this limit. Not that the people could prevent it, but still…
I had hoped for a fundamental ruling. They should not have touched numbers. The EU had a no-bailout-clause from day one. It was conditional from the start. If one of the fundamental principles of a contract is broken, the contract becomes invalid. Not only did I expect the court to rule against socialised debt, I would have wished them to go a step further. The German national bank gave up control over the currency to the ECB only because the ECB incorporated the principles of the German national bank. Once the ECB overturned those principles, Germany should have reclaimed their souvereignity.
But noone else in Europe would have wanted that. Merkel became a puppet herself.
I am grateful that our daughter has dual citizenship.
NP: Porcupine Tree: Live at Atlanta 2010
The atmosphere in Munich's Backstage Werk just before the opening act to the Mouse on Mars was very chilled. People sat on the stairs or scattered themselves over the dance floor while low-fi ambient tunes came from the speakers. It wasn't loud, you had to try hard to hear the people mumble.
I have no idea who the opening act was, and their first tune was very nice and groovy. Then ensued a noise explosion, one could only pity the electronic equipment that was being asked to perform in ways that may be described as "everything else than you expect", and of course, the base beat shook the building; I am quite sure they didn't use treble at all, but I may also simply have been unable to hear it. Plus, it seemed to us that the musicians catered for what may be a widespread decrease of attention span: it was noticable how they jumped from one thing to the next, not leaving them (or their listeners) any time to get in the groove.
My brother and I went outside for a bit and talked about today's music and its simplicity. We postulated the repetitiveness as the basis of a mass movement, considered "scene" clubs that played heavy techno to an audience that is so entirely different to who historically frequented such musical performances, and in general tried to avoid assuming a position between simplifying society and accepting that individual freedom is as eclectic as can be.
When MoM opened, they continued pretty much in line with their openers and half way through the first tune, I started to wonder how long I would last, or when it would be reasonable to step outside again. I had been a little afraid this would happen, having bought and listened to their latest album Parastrophics in preparation of the concert and not being able to get into it.
However, what then followed blew us away. Still heavy, still all over the place, but now they were developing sound scenes, ripping them apart, having fun playing with and teasing the audience, while putting on a groove that inevitably made your muscles twitch with the beat.
David Bowie called MoM "the next big thing" and I have to give it to them: MoM have always had a certain aura of "that's what your music is like? we can improve on that!" to them, and yesterday, they continued along those lines with astounding consistency, and it felt fresh.
It also felt real. They weren't just pushing buttons and computers making music, they were making music and the computers were their instruments. Between the two founding members of MoM sat Dodo Nkishi, drummer and microphone artist, and if you don't believe, fast, big breakbeat can be performed live, well, you're wrong.
Most everyone in the room was dancing. And while I was more swaying in awe, watching and wondering how the heck they are doing what they are doing, I couldn't contain the bouncing any longer. They came back for an encore and there was no more stopping the crowd, Thomas or me.
Three tracks later, they waved goodbye and left, but a bunch of us simply continued to dance. Thomas questioned who would last longer and I started yelling loudly for another encore. The lights turned on, I considered it a slap in the face, but I did not stop yelling. Others tuned in. And then the lights went off and the band came back.
Following their 2.5 hour show, gosh was I exhausted. It was a magnificent show. If you aren't afraid of big beat electronica and you take pleasure in nonstandard art, I heartily recommend you ensure that MoM aren't soon playing near you without you there.
PS: MoM will play at the (Düsseldorf Open-Source Festival)[http://www.open-source-festival.de/en/] on 30 June 2012!
PPS: Now I listen to Parastrophics and I am really enjoying it.
NP: Mouse on Mars: Parastrophics
Please help stop ACTA. Our freedom is at risk. Whether you tell
people about it, write about it, use services like Twitter to tell the world about
#StopACTA, or whether you take the time to march
against what corporate entities are lobbying politicians to do
against their people — please help protect the Internet as we know
NP: God is an Astronaut: Moment of Stillness