[HaBo]

derhesse

Wie www.pro-linux.de gestern bekannt gab, sind Server vom Debian Projekt gecrackt worden. Gecrackt wurden die Server mit der Fehlerdatenbank (master), den Mailing-Listen (murphy), Web- und CVS-Server (gluck) sowie der Server mit Security, Non-US, Web Search, WWW-Master und Q/A (klecker).
Aus diesem Grund sind einige Dienste zur Zeit nicht erreichbar. Das Debian Projekt will diese Dienste erst nach eingehender Prüfung wieder online nehmen.
Angeblich sollen keine wichtigen Dateien verändert worden sein. Es empfiehlt sich aber, frisch eingespielte Security Updates später zu wiederholen, da zeitgleich Debian 3.0/r2 veröffentlicht wurde.
Gerüchte sagen, es war kein Exploit oder anderes Sicherheitsloch, sondern ein schlechtes Passwort. Später dazu mehr.

Der Hesse

poiin2000

Auf www.debian.org und www.security.debian.org konnte ich keine Bestätigung dazu finden, auch wenn mir diese Meldung schon länger bekannt ist.

mfg p2k

heraklit

ich auch nicht, aber lists.debian.org ist nicht erreichbar und das spricht ja wohl für sich selbst, oder?

sieben

Es ging über debian-announce.

Some Debian Project machines have been compromised
From: Martin Schulze <joey@infodrom.org>
Date: Friday 21 November 2003 12:40:12

------------------------------------------------------------------------
The Debian Project http://www.debian.org/
Some Debian Project machines compromised press@debian.org
November 21st, 2003
------------------------------------------------------------------------

Some Debian Project machines have been compromised

This is a very unfortunate incident to report about. Some Debian
servers were found to have been compromised in the last 24 hours.

The archive is not affected by this compromise!

In particular the following machines have been affected:

. master (Bug Tracking System)
. murphy (mailing lists)
. gluck (web, cvs)
. klecker (security, non-us, web search, www-master)

Some of these services are currently not available as the machines
undergo close inspection. Some services have been moved to other
machines (www.debian.org for example).

The security archive will be verified from trusted sources before it
will become available again.

Please note that we have recently prepared a new point release for
Debian GNU/Linux 3.0 (woody), release 3.0r2. While it has not been
announced yet, it has been pushed to our mirrors already. The
announcement was scheduled for this morning but had to be postponed.
This update has now been checked and it is not affected by the
compromise.

We apologise for the disruptions of some services over the next few
days. We are working on restoring the services and verifying the
content of our archives.


Contact Information
-------------------

For further information, please visit the Debian web pages at
<http://www.debian.org/> or contact <press@debian.org>.

Rushjo

Hier ist noch mal der Bericht von Martin Schulz (Debian Maintaincer)
an die bugtraq@securityfocus.com-Mailingliste.

MfG Rushjo

sieben

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

*NB* bear in mind that:
a) the information on the break-in in comes from compromised machines
and thus has to be taken with appropriate skepticism.
b) the investigation is still ongoing - as I was writing this draft
further information came to light which may invalidate a lot of
it. [Or not - as it turns out].

Detection
---------

On November 20 it was noticed that master was kernel oops-ing
lots. While investigating this it was discovered that murphy was
showing the exact same oops, which was an overly suspicious
coincidence. Also klecker, murphy and gluck have aide installed to
monitor filesystem changes and at around the same time it started
warning that /sbin/init had been replaced and that the mtime and ctime
timestamps for /usr/lib/locale/en_US had changed.

Investigation revealed the cause for both these things to be the
suckit root kit (see the "Suckit" appendix for more info).


What happened?
--------------

On Wednesday 19th November (2003), at approximately 5pm GMT, a sniffed
password was used to access an (unprivileged) account on
klecker.debian.org. Somehow they got root on klecker and installed
suckit. The same account was then used to log into master and gain
root (and install suckit) there too. They then tried to get to murphy
with the same account. This failed because murphy is a restricted box
that only a small subset of developers can log into. They then used
their root access on master to access to an administrative account
used for backup purposes and used that to gain access to Murphy. They
got root on murphy and installed Suckit there too. The next day they
used a password sniffed on master to login into gluck, got root there
and installed suckit.

See the "Time-line" appendix for more details on times.

Response
--------

Gluck was powered down and an image has been made of it's disks for
forensic analysis.

Since we didn't have direct physical access to klecker it's Internet
connection was shut down and disk images were made via serial console
to a local machine on a firewalled net connection.

master and murphy were kept running for a short while in order to make
an announcement of the compromise, after which they were also taken
off-line and imaged.

Cleanup
-------

After a thorough cleanup and reinstall of modified files the non-US and
security archives were verified by looking at mirror logs for changes and
comparing MD5 checksums of the files on Klecker and those on three
different trusted mirrors.

Gluck, Master and Murphy were wiped and reinstalled from CD. data and
services are in the process of being restored.

All machines and data were checked for devices outside of /dev, suid
executables, writable files, etc. and all suspicious files were
removed. Services (and their scripts/programs) are being compared to
known-good sources and sanity checked before being re-enabled.

Since we now knew we had compromised accounts and sniffers on our
hands we had to assume that that an unknown number of accounts were
now compromised, so all accounts were locked, passwords invalidated
and ssh authorised keys removed.

How could this happen?
----------------------

All the compromised machines were running recent kernels[1] and were
up-to-date with almost all security updates[2].

However there was two problems.

(1) The kernels running on the machines in question didn't all get a
ptrace fixed kernel as fast one might have liked. Master, Klecker
and Murphy got new kernels in May but Gluck for various reasons
didn't get upgraded till August (although I believe it had
/proc/sys/kernel/modprobe fixed to at least block the most common
exploit before that).

(2) Master had a copy it's old harddrive still lying around by
accident. Unfortunately it had a lot of old, unpatched suid
binaries on it.

Although these could have been the attack vector, I don't believe they
were. (2) seems unlikely simply because master wasn't, AFAWK, the
first host compromised. Although it's possible an attacker with local
access to gluck got root through (1), it seems unlikely they'd sit on
that for <n> months and then use it on several machines only to
comeback and rootkit several debian.org machines and at least one
(that we know of) other unrelated system at the same time (and which
didn't have an extended ptrace vulnerability exposure.)

Based on that and the forensics on the unrelated system mentioned
above, I believe that there was an as of yet unknown local root
exploit used to go from having local unprivileged access to having
root.

Where do we go from here?
-------------------------

Unfortunately due to the fact there is (I believe) an unknown local
root exploit in the wild, we can't yet unlock the Debian accounts.
Obviously we can't continue without LDAP accounts for very long
either. At the moment I'd ask for a little more patience both a)
while the painful and painstaking task of restoring machines one by
one is completed and b) while we try and exhaust all reasonable
avenues of investigation to determine how the attacker went from
unprivileged to root.

Obviously we're looking at hardening our boxes and tightening up our
procedures to try and stop this from happening again. I'll send more
details on that later.

Finally
-------

Developers worried about their own machines might like to have a look
at:

http://www.wiggy.net/debian/developer-securing/

================================================== ==============================

Appendices
~~~~~~~~~~

Thanks
- ------

o Adam Heath and Brian Wolfe for their work on master & murphy.
o Wichert Akkerman for his work on klecker.
o Dann Frazier and Matt Taggart for their work on gluck.
o Michael Stone and Robert van der Meulen for their forensics work.
o Jaakko Niemi for his work on checking and re-enabling lists.debian.org.
o Colin Watson for his work on checking and re-enabling bugs.debian.org.
o Josip Rodin for his work on checking and re-enabling the lists web archives.

[This text is based on a draft by Wichert Akkerman.]

========================================

Time-line
- ---------

All times in GMT.

o Klecker init timestamp: Nov 19 17:08
o Master sk timestamp: Nov 19 17:47
o Murphy sk timestamp: Nov 19 18:35
o Oopses on Murphy start: Nov 19 19:25
o Oopses on Master start: Nov 20 05:38
o Gluck init timestamp: Nov 20 20:54

========================================

Suckit
- ------

Suckit is a rootkit which installs a sniffer, a process hider, a file
hider and a backdoor login in a running kernel. Apparently there was a
flaw in its kernel code which caused the kernel to oops on master and
murphy. This also explained why /sbin/init was replaced: the new init
loads suckit into the kernel and then proceeds to start the real init,
making sure that it is still active after a reboot.

========================================

Footnotes:

[1] Klecker: 2.4.22, Master & Murphy: 2.4.21-rc2, Gluck: 2.4.22rc2

[2] klecker was missing the latest postgresql updated. ssh on all
machines was a DSA-customized version which was missing only the
3rd and final round (i.e. Solar Designer's patches) of ssh
updates.

========================================

P.S. As always, I speak only for myself.

- --
James
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8 <http://mailcrypt.sourceforge.net/>

iEYEARECAAYFAj/GnvoACgkQgD/uEicUG7BG8wCgriGB843o5TwmZZszRGlVPosL
b/0An2cpuoQEAaq4iMls0KlWrLcBlkEo
=i8am
-----END PGP SIGNATURE-----