Zen and the Art of Systems Administration

Presentation to the Columbus chapter of the League of Professional Systems Administrators (LOPSA) on August 27th 2020.

In the early 70s, Robert Maynard Pirsig submitted a manuscript to over a hundred publishers. His manuscript, Zen and the Art of Motorcycle Maintenance: An Inquiry Into Values, was finally published in 1974 by William Morrow, earning him $3000. Pirsig was told by his publisher that they would be lucky to sell the initial run of just a few thousand copies. Within the first year of publication, Zen had sold over a million copies, his book was a best-seller for a decade, and has sold many millions of copies since. Zen and the Art of Motorcycle Maintenance has become a classic text, especially among those college students studying philosophy for the first time, as it takes western philosophy and tests its basic assumptions – contrasting the classical western European view of metaphysics against the assumptions and philosophies of Southern and Eastern Asia.

Pirsig has been quoted as saying that despite its title, “Zen and the Art of Motorcycle Maintenance” should “in no way be associated with that great body of factual information relating to orthodox Zen Buddhist practice.”, he has also said, “It’s not very factual on motorcycles either.”

What Robert Pirsig accomplished in his “Zen and the Art of Motorcycle Maintenance”, was to create a metaphysical, even scientific, explanation of why people engage in religious/philosophical ritual and practice. For thousands of years groups of human beings have created and practiced all sorts of rituals from the Norse Blot Sacrifice of mead and grain, to the Catholic Mass – body and blood of Christ, to American Indian sweat lodge and pain rituals, the prayers of Brahmans over their sacred fires in India, or the contemplation of our ancestors and many, many others that we will never know. Why? Why do we do these things? Because these rites, rituals, and practices work. These activities are useful to us – we are not machines – we are men and women and we need to believe in something to live and work together. These beliefs and practices are the basic operating systems of our minds. Our rituals help us to remember who we are, that we are not alone, and connect us to our community.

Like Zen and the Art of Motorcycle Maintenance, my talk today, Zen and the Art of Systems Administration is not the definitive work about Buddhist Zen philosophy, it is not going to be the end all, go to work about DevOps or UNIX administrative and engineering principles. What I am going to attempt to convey are a few things I have learned in maintaining and deploying networked UNIX systems during my career in the context of the essential core beliefs of Zen Buddhism. I believe that by following in the footsteps of the Buddha we can become better people, and more productive systems administrators. People who are making a meaningful contribution to our organizations in the good work of administering networked computing environments.

My name is Doug Mitchell, and I am currently calling myself a Linux Systems Engineer. I have been designing, deploying, and maintaining networked UNIX systems for longer than I want to admit. I have established a limited-liability company Auspicious Clouds Computing, but am currently with Alliance Data in Columbus. For more about my particular career, you may go to the web site at auspiciousclouds.us.

I am also a lay chaplain, having completed a course of study at Bishop Anderson House at the University of Chicago and my practicum at Alexian Brothers Hospital in Elk Grove, IL. I give sermons at my local Episcopal church from time to time, I write poetry, fiction, and send letters to people in positions of authority – reminding them of our common humanity. Many of these writings are posted online on the web site whatisgood.us

It would be remiss of me to give a talk about Zen Buddhism without talking about the life of Lord Siddhartha, the founder of Buddhism. One of the legends about his birth and childhood is that his father raised the young Siddhartha such that he never saw anyone sick, or injured, or suffering in any way. When Siddhartha was a young man, he was shocked to see the old, the infirm, the sick in his city. He sought out spiritual teachers who could explain to him why mankind seemed to be doomed to suffer, and after many years of spiritual discipleship in several different ascetic traditions finally decided to meditate under a bodhi tree for inspiration. For seven days and nights he mediated and when he rose from the bodhi tree he had achieved a mental and spiritual awakening – the title ‘Buddha’ means ‘Awakened One’. Siddhartha, the Buddha now had a new enlightened understanding of the nature of the mankind’s relationship with the universe.

Siddhartha then traveled in search of his fellow spiritual students, and these fellow travelers became his first disciples, in time, spreading Buddhism to every corner of our world.

According to Buddhism, all suffering is because of our desires. Our desires and attachments to things, ideas, and people and also our aversions or fears – which are our negative desires; our fears of losing what we have, our fears of losing our loved ones.

All we have to do to escape the suffering of the world is to give up our desires and attachments. Once we are released from the chains of our desires and the need for attachments to things and other people – we are truly free to be ourselves. We will see the world clearly, without the distortions of our desires. We will stop projecting how we want things to be on our view of the world – we will be able to evaluate the evidence of our senses objectively and accurately.

Getting rid of our desires and attachments is not so easy, it completely goes against how most of us conduct our lives, accumulating stuff until the day we die. Buddhism is a practical philosophy, providing a path to this redemption – the eight-fold path – eight principles to help any of us to attain enlightenment – so that we too may be like the Lord Buddha – truly awakened to the nature of reality.

Getting back to Systems Administration….

Here we see an example of the source of our pain and suffering as systems administrators – from the point of view of Buddhist philosophy. This is a very simple example of perception verses reality, from the point of view of the Business, the Systems Admins, and the actual reality of a deployed system. We all simplify, make assumptions, use mental shortcuts in interpreting the world around us. To the Business, this system is a very small, minor application. They are barely aware of it at all, they know nothing about how it works, what depends on it. Perhaps it has never failed so it has never really has been brought to their attention.

To the Systems Admins it has a specific shape, it is more important, they are aware of the dependencies on it, how it is installed, how to restart it. But this system is a ‘black box’ in that they have no visibility inside of it – about how it actually performs its function.

The reality is that there are all sorts of internal structure to this system. But no one in my example is aware of this.

Let’s say that the systems team makes some predictions about porting this system to another type of operating system, or that it will work with a different version of Java, or different kernel libraries. And they are wrong – they made these predictions without the knowledge of the reality of this system. They perform a deployment, which crashes and burns because this vital system no longer works in the way it did before.

How can we avoid situations like this?

How can the Eight-fold path help us to see our systems clearly?

These initial Buddhist practices come directly from the fault and root cause analysis of our simple perception verses reality example. To make wise decisions, we must be aware of what is right in front of us. We must have a true understanding of how our systems work, what the dependencies are, and only then can we reliably maintain them.

We must continually, test our assumptions in the course of our daily work. After many years of logging into UNIX systems – I am very aware of how long commands take to complete, the general performance of a system simply from the command response.

I have often ‘discovered’ error states on systems while performing maintenance on something completely unrelated. For example, just a couple of months ago I was performing a refactoring of my Ansible scripting – completely changing how I support virtual server inventories. After the refactor, I did some testing against groups of servers with the Ansible ping command. A production server did not respond to the ping – so I checked it out. It turned out that the server was non-responsive, and our monitoring tools did not pick up on it because it was still executing, still seemed to be booted, but it could not respond to the Ansible ping. It was part of a cluster of database servers – so nothing was off-line, but I had discovered a new failure state – from testing some scripting changes.

We should always be mindful, and present even when doing the most routine of maintenance; having a mind open to the possibility of learning something new.

Our words and our actions, define who we are. Have you ever been in a situation which you made infinitely worse – by something you said. (pause) I have. (pause) Wisdom has more to do with what we do not say. If you are in doubt about making a comment – just keep it to yourself. This is not to say that we should not speak the truth – if the situation demands it. It may cost us our job – but if you are ‘in trouble’ for telling the truth to your management – you really need to be looking for another position.

Systems Administrators by the virtual of the access we have to the computing environments we maintain have to exercise a great deal of caution in the commands we enter, the widgets we click on, the systems we decide to access. In the course of my career I have rebooted the wrong servers, deployed the wrong software to whole groups of servers, run scripts and programs that have brought UNIX systems to their knees. Most of the time if we are mindful, and present we can recover from these mistakes.

Many years ago, I was supporting Solaris servers which were part of a tape backup solution. The company was a subcontractor for AT&T performing tape backups at all their data centers. The home office was in North Carolina. One day the central office sent us a data collection script, we were to execute on our servers. A data file was to be created which we were to email back to the home office. I took a quick look at the script, it did not seem to be too invasive. I started it. The server immediately was loaded down, command response on other terminal sessions was poor, the disk activity light was going crazy. I hit Cnt-C to kill the process. The script had a recursive loop error in it that I had missed, which would keep writing the same data to an output file over and over again. Eventually filling the root file-system of the system it was running on as the output file was hard-coded to write to /. I quickly contacted by brother and sister admins across the company at our other sites – but we had service outages in about 40% of our data centers while recovering from this self-induced issue.

Being aware, and present to the here and now – saved me from some extra work that day.

I’ve mentioned mindfulness and being present several times in this talk already and there it is on this slide as a spiritual discipline for a Buddhist.

A more disciplined approach to our deployments can help us complete them a great deal more easily. Often times we are unaware of systems dependencies until that one system that is usually so reliable goes down.

Another example of this is how Databases are accessed by applications. I once made a functional systems diagram of an e-commerce site showing all the web application interactions with our database servers. It looked like a spider web, because every server in the deployment had to access the database in putting items in the shopping cart, placing the order, or order fulfillment. The e-commerce application had been written by several developers over several years and when they needed data from the database, they just issued a call for it. As functions were distributed to different servers, the scripting moved to a new server and now that server also needed direct access the database as well. Every server now needed to access the Database server for the implementation to work. I was tasked with implementing better network security – but with this e-commerce system – any network security measures broke some part of the ordering process.

Applications do not have to be implemented this way.

I worked for a startup in Chicago which created and operated an online Commodities trading platform. We completely reworked our systems architecture, clustering solution and software applications three times in four years, while supporting active users for 3 of those years. It was a very small team – 5 developers including myself. After all those implementations, we settled on an architecture that separated the Client facing RESTful API layer from our market data layer, and order processing layer. Our final version was a marvel – the best work I’ve ever done. The final systems executing in AWS were completely modular. All the EC2 Ubuntu instances were interchangeable with each other – depending only on parameters set in a GitHub release which differentiated each server from each other. I did no patching, replacing these generic virtual servers every 2-3 months. Every boot of every virtual server redeployed both configuration and application version.

One of the habits I developed in that Chicago position was my morning commute by train into the city. I started reading from the Christian book of Psalms on my morning and evening commutes. In reading and meditating on these psalms (which means songs), I was unintentionally entering a meditative state, encouraged a presence of mind, a centering of the soul, which carried though into my work day. One of the great misunderstandings of our modern age is that we have forgotten how the state of our minds affect our health, our work, the relationship between our mind and body and our interactions with other people.

We would never operate our cars or our motorcycles without regular oil changes, or tuneups, yet we do no regular maintenance on the most important thing we have, the essence of who and what we are, our minds.

Regular meditation, intentional introspection and commitment to this type of a disciplined routine mental exercise is essential in the proper functioning of our minds just as regular physical exercise is essential to the body. It might also help us in the performance of our best work.

This talk has turned out to be very different than the one I had imagined. I was going to base it more on Robert Pirsig’s book Zen and the Art of Motorcycle maintenance. But I am not the same person I was when I first read Pirsig’s book in the early ’80s. I have changed. Since then, I have found a spiritual home in the Christian traditions of both the Episcopal Church and the Society of Friends – the Quakers. Pirig’s whole thesis of proving that a mental, spiritual discipline is necessary for us to be our best selves is second nature to me now.

What I have discovered in the formulation of my very brief introduction to Buddhist belief and practice here is how this type of discipline has helped me in my work. I never really put it together before – that the one time in my life I following a regular practice, just because I was commuting by train to downtown Chicago – I also performed my best work as a systems administrator. I had always thought that it was the particularly smart people I worked with. I have thought that the startup nature of the company might have been the reason. I think these contributed to a positive work environment. But I cannot now discount that I was also a little different, perhaps more ‘centered’ than I had been before. Perhaps I was more ‘mindful’, and ‘present’.

It seems I need to get out my psalter and start a regular practice of again reading from the Psalms every day, (pause) or perhaps, (pause) from the sutras of the Buddha.

Thank you.