If lulzsec (or anyone else) find a critical security vulnerability in the official bitcoin client, that could really cause a lot of damage. More than double spending or "hacking the block chain", I guess.
The coins in my wallet are save, as long as I don't sign a transaction using one of my private keys, which are stored in my wallet.dat.
If someone finds a bug in the bitcoin client that can be used remotely to send a transaction (or steal the private keys), then the list of listening nodes on IRC can be used to send a lot of money to their (the hackers) wallet.
Even worse, if they happen to be a listening node on IRC themselves, they can attack all bitcoin clients that use IRC to find peers to connect to.
So if you have a lot of BTC stored in your wallet, maybe you should conduct a security review of the bitcoin client.

(or store them in a savings wallet that is not used most of the time)