Having spent some time researching what actually goes behind rooting an Android phone, the main reason that leads to the ability root is an inherent vulnerability on the Linux kernel that leads to the exploit and getting the 'su' binary to be installed.
My question is: How come the OS is not hardened yet? Who is responsible? This is a Linux OS we're talking about, which is considered "secure". Yet, people always find a way to get in and root devices.
it's not really the "linux OS", it is the linux kernel. The vulnerability is not necessarily in the OS, or the kernel. There are different exploits that are used. I remember some device being able to load any zip file in the recovery if it was "pre-pended" by a signed zip file. So they used a signed zip and added an unsigned zip to the end of the file. I think this exploit was for the Original Droid.
No software is 100% bug free. if it was, then even a mature product like Windows would not have to do security patch releases every month.
No matter how smart the developer is, or thinks he is, more likely than not, there is probably someone smarter (or at least they may know more about exploiting code).
When exploits are discovered, Google (or the manufacturer) has (usually) patched them. But since a lot of the devices don't receive updates very often, the exploit remains available.
Rooting is always possible because rooting, a.k.a. user switching, a.k.a. setuid, is one of the most fundamental feature of Unix and Linux.
For many devices, rooting does not actually involve any security exploits; rooting methods that requires flashing a file through the bootloader and/or ROM updater utility (e.g. Samsung's Odin, HTC's RUU, etc) are not a security breach, as they are legitimate features specifically designed for flashing ROMs. A large number of devices are rooted in this way.
On those devices, rooting are possible because the manufacturers actually provides the facility for it. They certainly do not make rooting an easy one-click process (for good reasons; most people who wanted to root do not actually need root), but they do provide an officially sanctioned method to root under the condition that your warranty voids (official rooting method often leaves a permanent mark to let authorized technicians know if a phone had been rooted).
There are not that many quite a few rooting methods that actually uses a security exploits (e.g. rageagainstthecage, zergRush, gingerbreak, etc), and these exploits are often repackaged into a form that are much easier to use for the masses (e.g. SuperOneClick). Rooting through exploits are often hit and miss since they get patched fairly quickly for devices that are still within update period; but sometimes they provide advantages such as avoiding triggering the warranty marking.
The issue becomes a little complicated when the tech news and blogs reported when a developer wrote tutorials or tools to ease up legitimate rooting process. They often do not understand the nature of the rooting method, and certainly I have never seen them distinguish between legitimate rooting methods and rooting through security exploits, worse they also often report the repackaging of existing exploit and the porting of an existing exploit to a new device as if they're a totally different exploits. Thus the confusion that makes it appear that Android seems to have more exploits than it really does.