Openldap 3-way multi-master (delta-syncrepl)

Servers, Versions

This is a test setup with 3 VMs running on the same host:

• ServerID 000: phd-aa1 (initial master)
• ServerID 001: phd-aa2
• ServerID 002: phd-aa3

Versions:

• distribution: Debian GNU/Linux 9.0 (stretch)
• kernel: Linux phd-aa1 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
• slapd: 2.4.44+dfsg-5 amd64 OpenLDAP server (slapd)

Issues and questions:

1. Endless replication loop after fast subsequent modifications

Is this observation normal behaviour?

While testing subsequent ldapmodifys of the same attribute of the same DN object on one host, we noticed that the replication will get in an endless loop when subsequent modifications occur too fast after each other, for example:

• Modify attribute A of cn=00000,ou=tt,dc=phys,dc=ethz,dc=ch on phd-aa1
• Wait for a short period of time (<= ~10-100ms), or not at all.
• At this point the change was most probably not yet synced to the other nodes, before that sync happens, repeat the modify:
• Modify attribute A of cn=00000,ou=tt,dc=phys,dc=ethz,dc=ch on phd-aa1
• Now the 3 slapds are somehow in a loop, sending cookies forever
• Sometimes a restart of all slapds helps to kill the loop
• Sometimes a stop of at least 2 slapds was nessecary to kill the loop
• Subsequent modifications had to occur in a variable time frame lower than 10-100 ms in my tests to cause the loop.

2. Occasionally slapd hangs on shutdown

Occasionally when restarting/stopping the slapd service (using systemctl restart/stop slapd), the process just hangs with 100% CPU load.
Only a kill -9 <PID> helps. In the logs it states that slapd is waiting for some tasks: slapd shutdown: waiting for 2 operations/tasks to finish.

3. Occasionally slapd dies when another master is restarted

Occasionally when making an update using ldapmodify and after that restarting slapd on the same host,
it can happen that a slapd process on one of the other masters just dies away (with nothing in the error log).

4. Unwilling (53) slapd after service restart

Can the following symptoms safely be ignored?

This seems to be a temporary hickup under some circumstances, and fixes itself after another ldapmodify on the unwillig slapd.
We did the following:

• Setup phd-aa1 (ID 000): initial master
• Setup phd-aa2 (ID 001): replicates all data
• Setup phd-aa3 (ID 002): replicates all data
• Test replication: ldapmodify an attribute on each master separately: all syncing correctly
• Restart slapd on phd-aa1: all ok (all syncing)
• Restart slapd on phd-aa2: now this one starts reporting in the logs that phd-aa1 (rid=004) is unwilling:

do_syncrep2: rid=004 (53) Server is unwilling to perform

• Restart slapd on phd-aa3: now this one starts reporting the same.
• Syncrepl tries to reconnect according to the retry parameter interval, logging the error above on each cycle.
• All slapds are still syncing correctly though.
• After another ldapmodify on phd-aa1 the error message above is gone and all logs look ok.

5. Is n-way multi-master using delta-syncrepl supported?

• I could not find any concrete hints in the documentation that it was not supported.
• I found some mailing list hints, where people reported it working.
• It seems to be working in our setup (most of the time).
• On this page it is mentioned that it is not supported as of 2.4.35:

"Delta Replication (Accesslog) using OLC: This example assumes a Master (provider) of ldap1.example.com and a Slave (consumer) of ldap2.example.com. It could, of course, be equally applicable in a N-way multi-master configuration at the expense of some added complexity (though it is not - as of 2.4.35 - currently supported for N-Way multi-mastering)."

Setup procedure

Our setup procedure is as follows:

Set up the initial master slapd

• Set up the initial master slapd (installed using apt) with this config.
• Slapd started using systemctl, output of ps -ef when running:

openldap 16316 1 0 16:23 ? 00:00:00 /usr/sbin/slapd -h ldap:/// ldaps:/// ldapi:/// -g openldap -u openldap -F /etc/ldap/slapd.d

• The config DIT (cn=config) configured for replication: using syncrepl (no delta-syncrepl)
• The normal DIT (dc=phys,dc=ethz,dc=ch) configure for replication using delta-syncrepl (accesslog: cn=deltalog)
• Set up kerberos KDC on the same host with slapd as backend
• Add our custom dphys schema:

/usr/bin/ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/dphys.ldif

• Save a dump of our data in the existing old server using slapcat. Remove the following lines in the dump:

^createTimestamp
^creatorsName
^structuralObjectClass
^entryUUID
^entryCSN
^modifiersName
^modifyTimestamp

• Load the modified dump of our old data into the new slapd initial master:

/usr/bin/ldapadd -Y EXTERNAL -H ldapi:/// -f /root/dump.ldif

Set up the additional slapd masters (one after each other):

• Generate required kerberos principals and keytabs on the initial master and transfer the files.
• Stop slapd on the initial master (maybe not nessecary), dump the slapd config using:

/usr/sbin/slapcat -n 0 -l /etc/ldap/config/config.ldif

• Start slapd on the initial master and transfer the config.ldif to the other masters.
• Install slapd using apt and stop slapd.
• Remove the the config and data, recreate directories:

rm -r /etc/ldap/slapd.d
mkdir /etc/ldap/slapd.d
rm /var/lib/ldap/data.mdb
rm /var/lib/ldap/lock.mdb
mkdir /var/lib/ldap/deltalog
chown -R openldap:openldap /var/lib/ldap

• Add the configuration from the initial master:

/usr/sbin/slapadd -F /etc/ldap/slapd.d/ -n 0 -l /etc/ldap/config/config.ldif
chown -R openldap:openldap /etc/ldap/slapd.d

• Start slapd
• It will then start syncing the data from the initial master.