Puppet and 64bits packages

Since I use puppet to manage my machines (and the machines of customers), I noticed that I had more packages installed then before, I noticed also obviously the same behavior in packages to update and bandwidth consumption during updates.

I realize that on 64bits machines, most of the time, the 32bits version of the packages managed by puppet were also installed.

This is what I did in my recipes before:

    package { "corosync":
        ensure => "installed",
        require => Yumrepo["clusterlabs"];
    }

This kind of package declaration installed then the two version of the package, in this case corosync and the dependencies too.

To avoid this I added the fact hardwaremodel and used the alias to keep my recipes consistent:

    package { "corosync.$hardwaremodel":
        ensure => "installed",
        alias => "corosync",
        require => Yumrepo["clusterlabs"];
    }

Hope this could help people having noticed the same behavior... or not :-)

High Availability Open-Xchange Server

Since I tested it 4 years ago, I like Open-Xchange (even if I'm not a java app fan). I like the layout and also all the feature it provides. The calendar is very complete.
For a customer where I set it up 4 years ago, I've migrated this service to a cluster running the last version.
The machines are fully installed via kickstart from a pxeboot (using cobbler)

This post describes the solution. The setup is based on CentOS and use the pair corosync / pacemaker as cluster.

The solution consists in two nodes where only one machine provides the service.

The components are :

- one ip balancing between the two nodes
- apache running on the "active/master" server (the server providing the service)
- open-xchange running on one node at the time
- funambol running on one node at the time
- openldap running on both machines in mirroring
- cyrus running on both machines as master/slave
- mysql running on both machines as master/master replication.

This is an overview of the crm:

Most of the needed steps are put in some puppet recipes to help the provisioning (you can find them on my github account)

With the cyrus-imapd delivered by default on redhat/centos, when the cyrus master starts without the slave running, cyrus won't reply for a long time... the bug we are hitting here as been resolved in newer version. I use cyrus-imapd 2.4.6, package from Simon Matter. You can find the source of this package here

MySQL & Friends Meetup at Fosdem 2011

Like last year I'll be present at the MySQL & Friends Meetup on Saturday evening of the Fosdem.

If you wanna share some experience around MySQL, please join !

You can register here

devops Meetup at Fosdem 2011

I will be present at the devops meetup of Fosdem 2011.

It will take place the Friday 4th February.

I hope to see you there if your are in the devops mind and if you wanna share some experience with us !

Register here

tomcat6 with APR on RHEL5.5

The other day at $CLIENT I had to install a webapp that needed tomcat6 and I found it was quite a hassle to install a packaged version on a 32bits RedHat RHEL 5.5.

During that painful process, lefred helped finding the magic package that solved it all so I felt it was only fair for me to publish the complete solution as a guest post on his blog!

devops… to package or not to package… this is the question !

During the Devopsdays in Hambourg, one of the most recuring discussion was about "packaging vs non-packaging, when and what?"

I won't try to convince people on what do do when, neither will I say I have the absolute best solution, this post just illustrates the solution I implemented with @zipkid.

Some points aren't finished yet, not implemented... or we have not yet decided which direction to follow.

First, let's start we the description of the environment:

A web based application (J2EE) with a MySQL backend, this product is delireved to us as a tgz package.
There are many interconections between gateways, applications, databases, map servers, etc... all these defined in configuration files.
We are using SLES from 10 to 11sp1 and we maintain a bunch of servers: physical machines of different types (dell, IBM blades,..) and virtual machines.

What tools do we use ?

- GNU Linux
- redmine + kanban board plugin to define the tasks
- a pxe installation system (autoyast in sles and cobbler in redhat/centos/fedora) to (re)install the machines
- puppet to deploy the configurations
- git to save all our configurations of puppet
- svn to save other things like specs files (this should be migrated to git)
- puppet-dashboard to have an overview of the deployed machines, an overview of puppet and define some variables we use in our recipes
- rpmbuild to ... euh... build the rpms :)
- jmeter to perform load test
- nagios to monitor the systems

What is the process then ?

To define the processe, we must first divide it in several categories :
- OS installation and maintenance
- "our business product"

To install a machine, we install a basic image on a machine (virtual or physical) via pxeboot using kind of kickstart files for redhat base system or autoyast for SLES.

We create the node in the dashboard, we add some variables if needed like ip, environment, task. We add the server in the autosign file of puppet.
In the dashboard and puppet we have several different environments that are linked to some git branches. This allow us to test recipes or settings without modifying the production.

Then puppet is started and takes care of everything : vlan interfaces, bonding the interfaces, dns resolving, install the needed package and change the configuration files via puppet.
Nagios checks are also configured by puppet.

For our product, we first create the package (rpm) from the tgz provided by the developers, and put it in our own repository.
After having installed it on the test servers we start some load test scenario.

Back to the big question then: do we package ?

The answer is definetively YES ! To keep a control of what is installed on the system (package version, release and not having orphaned files).
BUT the default configuration files are overidden by the puppet run. conf files, xml, shell scripts, cron jobs are indeed provided by puppet and available in git (which provides us version control too)

Of course puppet runs constantly on every machine to constantly guarantee the desired state, both on production and on the test machines!
This is only dangerous if you don't test your puppet recepies enough during the development phase.
We don't start the puppet client in deamon mode but we start the process via cronjobs to avoid any memory usage issues which we encountered with puppetd in daemon mode.

How to improve ?

We would like to improve the load test and automate the build, installation and test on the test server of "our product".
We plan to use hudson for the CI with jmeter for unit tests and why not tsung for bigger load tests ?
Some open question we still have if we deploy a CI system is how to link a build version with a puppet configuration ? Using a new branch in git linked to a new environment in puppet (and puppetdashboard) doesn't seem to be an optimal solution. We opted then with a git tag corresponding to the build release and only the last one in testing is deployed on the test machines. If needed we can rollback to a previous tag and package.

It would be also great to automatically test our puppet recipes with a tool like cucumber-puppet.

I think we are going in the right direction, but the road is still long to a fully automated processes with an overview control of all aspects.

But we all agree that puppet already helped us a lot to maintain all our servers.

This is a schema illustrating the process :

1. developers provides a tgz with their application (a java compiled application, they also use Hudson to test their package)
2. the "DEVOPS" machine is started ! Devs and Ops collaborate to write the specs for the rpm package and the puppet recipe (dependencies, configuration settings)
3. test the package build and the puppet recipe (with cucumber-puppet)
4. add the package to the rpm repository and commit the puppet recipe to git (and the rpm spec to svn in our case)
5. puppetmaster gets updated with the new recipes
6. this is only in case of a new machine, the machine is automaticaly installed via pxe
7. puppet client installs the needed packages and configure the system as needed
8. puppet also configures nagios and nagios automaticaly startsmonitoring the machine and the services, hudson also starts unit tests and load tests if needed
9. same as point 6
10. puppet installs the needed packages and configuration to the production machine. it also configures nagios to monitor the machine and its services

Ignite talks with impress!ve

During last devopsdays in Hamburg, Gildas presented a session of ignite talks.
He was using impress!ve but it seems the software was not really designed for that purpose: you should have manually defined the duration of the session but also calculate the duration of each slides...

I decided then to patch this very nice product to fit the "ignite" needs :)

The proposed patch automatically calculate the duration of the slides and add the a countdown for the slide display + the slide number. (see screenshots)

New argument :

$ impressive --ignite 5m MySQL-spider.pdf


Windows 7 and Samba

Today I tried to put in production an update of Samba 3 (3.5.4) to allow Windope 7 clients to join the domain.

After having performed what's on the samba wiki page about this topic [here], I could join the machine to the domain but I was not able to login !? :(

In the log :

[2010/08/20 16:55:20.682477,  0] rpc_server/srv_netlog_nt.c:714(_netr_ServerAuthenticate3)
  _netr_ServerAuthenticate3: netlogon_creds_server_check failed. Rejecting auth request from client RO-BACKUP machine account RO-BACKUP$
[2010/08/20 16:55:30.993850,  0] lib/util_sock.c:474(read_fd_with_timeout)
[2010/08/20 16:55:30.993958,  0] lib/util_sock.c:1432(get_peer_addr_internal)
  getpeername failed. Error was Transport endpoint is not connected
  read_fd_with_timeout: client 0.0.0.0 read error = Connection reset by peer.

The problem was easy to solve but not easy to find : the two machines had not the same time (30 secs delay !)

Fixing the time sync fixed the problem (and I'm not using kerberos and AD)

the culprit is always… SELinux :)

After having setup squid and dansguardian (using clamd) on Centos 5, I wasn't able to use it :(

I had always the following error, even if the dansguardian user was the same as clamd (clamav) :

2010.7.9 12:22:41 - 10.0.200.6 http://www.eicar.org/anti_virus_test_file.htm
 *INFECTED* *DENIED* /tmp/tfIlR1j6: lstat() failed: Permission denied. 
ERROR GET 15590 0 Content scanning 1 403 text/html

I just realize after having searched too long that SELinux (I know life is too short for it) was the culprit.
It was my mistake as I completely forgot that this machine had selinux enabled :-S

So in /var/log/audit/audit.log I had :

type=AVC msg=audit(1278673113.470:3489): avc:  denied  { getattr } for
pid=32164 comm="clamd" path="/tmp/tfCSCirx" dev=dm-3 ino=17 
scontext=user_u:system_r:clamd_t:s0 
tcontext=user_u:object_r:initrc_tmp_t:s0 tclass=file
type=SYSCALL msg=audit(1278673113.470:3489): arch=c000003e 
syscall=6 success=no exit=-13 a0=8cce370 a1=421f2dc0 a2=421f2dc0 
a3=8 items=0 ppid=1 pid=32164 auid=1004 uid=102 gid=114 euid=102 
suid=102 fsuid=102 egid=114 sgid=114 fsgid=114 tty=(none) ses=437 
comm="clamd" exe="/usr/sbin/clamd" subj=user_u:system_r:clamd_t:s0 
key=(null)

Note to myself: Never forget to check in audit.log !

To create the selinux policies, I used the following commands, which are quiet easy:

audit2allow -a -m dansguardian > dansguardian.te
checkmodule -M -m dansguardian.te 
checkmodule -M -m dansguardian.te -o dansguardian.mod
semodule_package -o dansguardian.pp -m dansguardian.mod
semodule -i dansguardian.pp

Et voilà ! Dansguardian is running and I didn't disable selinux :-)

Get rid of the touchpad while using the mouse

I've been fighting several weeks (and making a huge number of typo's due to that) with the touchpad of my macbook pro on Fedora/Gnome.

I've tested several solutions :

- disable it in Gpointing Device Settings --> fail (it always comes back after a short moment)
- use synclient TouchpadOff=1 --> fail
- creating udev rules : --> fail

ACTION=="add", SUBSYSTEM=="input", ENV{ID_CLASS}="mouse", RUN+="/usr/bin/synclient TouchpadOff=1"
ACTION=="remove", SUBSYSTEM=="input", ENV{ID_CLASS}="mouse", RUN+="/usr/bin/synclient TouchpadOff=0

So the best solution I've found (one that works) is : rmmod bcm5974

Now I'll try to add it into the udev rules too.

Fedora 13 on mac book pro 13″

I used preupgrade to updgrade Fedora from 12 to 13.

After the process, I had to resync the partition in refit to be able to boot Linux.

I rebuilded the needed packages for nvidia and the broadcom wireless card.

I needed also to do some modifications to be able to use the integrated iSight webcam:

1. download the apple firmware :

wget http://www.i-nz.net/files/projects/linux-kernel/isight/against-revision-140/firmware/AppleUSBVideoSupport

then extract it (using the Fedora 12 package isight-firmware-tools) :

[root@delvaux ~]# su -c "ift-extract --apple-driver AppleUSBVideoSupport"

after this operation it should be working for most of the macbook pro, but not for this model, another change is needed.

first find the idProduct number:

[fred@delvaux Desktop]$ lsusb -v | grep  iSight -B 3 | grep idProduct
  idProduct          0x8507

and mofify the file /etc/udev/rules.d/isight.rules with the returned value :

[root@delvaux rules.d]# cat isight.rules 
ACTION=="add", SYSFS{idVendor}=="05ac", SYSFS{idProduct}=="8507", RUN+="/usr/lib64/udev/ift-load --firmware /lib/firmware/isight.fw"

Which Microblogging client to use ?

I have tested 3 Twitter clients to use on my gnome desktop:

- Gwibber (gwibber-1.2.0-3.349bzr.fc12.noarch)
- Pino (pino-0.2.6-1.fc12.x86_64)
- Turpial (turpial-1.0-b1.fc12.noarch)

I've found Gwibber very nice, doing what I need but it's very slow and sometimes freezes (becomes dark). It has also facebook support (nice to see the status updates of your friends).

Pino is my favorite one for the moment, it's fast, doesn't support facebook (yet?).
Something I don't like is that you have to switch between the accounts (twitter and identi.ca) which is annoying.

Turpial is not was I really wanted, it doesn't fit really in the desktop and it's only in Spanish.

I hope that Pino will keep improving and stay fast as now.

http://gwibber.com/
http://code.google.com/p/pino-twitter/
http://code.google.com/p/turpial/

Update !

I've just tested Gwibber 2 (gwibber-2.30.0.1-1.fc12.noarch) and it has very nice features: possibility to see all the messages or only by account, possibility also to see images (updated in facebook for example). I will for sure test it and discover it a bit longer, nice improvements !

As MySQL Community Manager, I am an employee of Oracle and the views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

You can find articles I wrote on Oracle’s blog.

Continue