Recientes del blog

Six steps to the Accidental DBA Nerdvana

marzo 15th, 2018 | Trucos y Tutoriales

Spoilers ahead. You have been warned.

In small companies there is the occasional role of the Accidental DBA (type that into an Internet search portal if necessary… from 10.000 feet it’s just about a developer taking sysadmin/devops responsibilities), these are typically less-than-part time occupying tasks one of your developers is taking care of, with more or less success and conformity.

The Accidental DBA is in great trouble, so let’s get straight to the helping tips for her/him.

Step #1 – Understand your mission as the cork of the bottle.

Imagine this wine is the evil shenanigans you have to confront, you are the cork preventing evil from spreading out of the bottle. That, my friend, is your job.

#1 – Be aware of your responsibilities.

In other words, the first thing you are going to learn, no matter if one way or the other, is you have to make sure the first time an incident happens is the last one. That’s the reason they are called incidents. If not, they are another different thing. If you are just a little meh about this, resign now while it’s not too late.

Step #2 – Get in charge and show them who is in charge now.

You belong to me.

#2 – Get those folk the f$%^ out of there.

Quite a part of your problems come from your fellow developer colleagues, so never too soon to begin retaliating back. Mercy is for the weak.

Find some kind of way to establish a private or semiprivate repository. Ensure the persistence of your knowledge after your death by bringing read write credentials to a selected part of your team, never more than half of your team and in no case more than two or three trustworthiest fellow developers. Ask them not to change your stuff unless in moments of great need, or if absolutely necessary, like your holidays or so.

You need to do this as is pre-charging a bill on your team in payment for all the future pain you are yet to get (#5 later on this document). Also your stuff is not their thing, things being strict. And many of them most probably won’t even really care, even though they possibly are making like it seems they do. And they really have important business-related development stuff to do anyway. It works the same as with management. You want them for they free you from having to deal with their stuff. They are your Accidental DBA abstraction layer. So you are going to be OK on your own… Most likely.

Step #3 – Know everything all the time.

Theo knows everything about you all, and Penny is eager to find you and kick your ass.

#3 – Get a big enough information delivery pipeline.

Following the article about automated service and server information delivery up with this mini raid into the particular facet of the Accidental DBA needs. Adding up to the scarce information your cloud provider is giving you about your data bases, each one of them (or the alternative dedicated agent/s) should report your platform’s vitals residing in your data bases. What? You say you don’t have any of those? Think again. Believe us, your platform has vitals residing in your data bases. Either as information or as meta-information. You are going to know.

If you can’t really run code on all your data bases because you don’t know, or you suck at it, or it seems you can not do in all or any of them can not; just code something written in your favorite programming language, abstract it in a way you can host it in different forms, but don’t try to code every one of them at once and please please please do a favor to your future self and make it not only simple but do the actual main deployment of it in a server-less style.

Chances are you have to manage a multi vendor data bases ecosystem, so look carefully what do you need to abstract data base operations regardless the exact type, vendor, family, and even paradigm if you can. Even though mileage may vary, we encourage you to write a connection (or cursor, ugh… don’t get dirty with the technical insights right now…) wrapper using a well known (if possible also well known to your team) programming language for which data base libraries are already written for all the vendors and families of all your data bases; it’s not hundred percent sure, but chances are you are going to need either Java or Python here. Too bad those two are not at all the better languages to code in 2018, fortunately soon those will be just a remembrance.

Step #4 – Release an endless storm of fire.

Long you have waited until this cathartic day when you finally come to enforce the rules you already warned of.

#4 – Automatic issue management.

An issue is the same as an incident, it just was detected early on its buildup and got taken care of before becoming an actual incident.

Establish a process (meaning,.. a business one, you know) to limit the data base connections of all the data bases at the company, not just the SPOF master/s, or your read write data bases, but also the read only replicas; or your abstract cluster or your multi master replicas, if you / your company could afford any of those operation models.

E.g. make sure data base connections are closed when reaching a particular number over a given level, measured on a per-user and per-host and per-application basis. At first it will be hard because your fellow developer colleagues may have not coded their applications in a way so those report their application name, if so, you are going to need to fix that first… and remain vigilant of possible rogue new applications, as long as you are serving as the Accidental DBA of the company.

The last couple paragraphs of the previous step apply verbatim. But do not try to achieve this step together with the previous one, or it is you who will burn in flames.

Step #5 – Face the unknown.

It’s like I’m suddenly in another world. And there I feel… cold.

#5 – Master trans vendor migration.

At some point, if not because management (or, hopefully not, mis-~), eventually stuff and sh will arise and you are going to need trans vendor migration. Arguably the most typical type now is from opensourcedb to proprietarydb, on an unexpected turn backwards of history; but probably as a society we have somehow achieved a maturity were the two directions are happening at the same time in similar rates. If numbers don’t seem to be fitting to your point of view, bear in mind that a large piece of that cake is really about deploying proprietary provisioning of opensourcedb, so assign numbers considering that freedom and surely you can understand the rationale then. The middle in there is actually a quite large grey area you can consider either proprietary or open as you might need.

Moving forward from the digression, at some point you will end up having recently migrated (as in different data base vendor) a data base and turning out that the application tier is not really hundred percent compatible, although in pre-production seemed to. This is a rare case because the data base migrations are not taken lightly, but even so, still stuff and sh can slip and permeate through the intensive screening development folk did. It can relate to a malfunction in a service of your own platform which has worked OK… all the time during and before the migration from its very inception, revealing itself in all its toughness only after the migration seems to be concluded.

This is frightening. If not, you were a bad pick to serve as the Accidental DBA, so shame on your boss/employer! This has necessarily to be frightening because it likely can be the source of the hardest incident in your career as Accidental DBA.

Here at the company, it so far seems we were lucky enough to have detected this one before it caused an impact, and believe us it could have been quite a bad strike. But that has also a lot to do with the fact we collectively decided to go on with the steps towards Accidental DBA Nerdvana way before being hit by an incident that would absolutely have justified it post mortem.

Yeah! We successfully performed trans vendor data base migration, it seems so far! We have a draft trilogy of blog entries about that, hopefully we will release that at some point.

Have this small appetizer: software people use to do this… really really suck.

Step #6 – Remain vigilant.

You need to be always in control, else nasty stuff is going to happen sooner or later.

#6 – Keep the reach towards constant Nerdvana. Don’t let it go down.

If you went that far, congratulations, so your data bases are no longer SPOF (but chances are you are now… yet, in any case, to way a lesser extent than those data bases were when you started); chill out, just a little bit, and keep up the hard work because that’s the true source of Nerdvana. That’s how you keep the evil held back in the bottle.

Sorry for the spoilers.

Throughout this article we picked IP bits from the series LOST, The Walking Dead, The Following, Game of ThronesStranger Things and Black Mirror. Madiva doesn’t own any rights on the six images shown in this post. We care about your intellectual property. So you see we played nice with your stuff, please do the same with ours.


Continúa leyendo este artículo

Operating system updates for pet servers

junio 26th, 2017 | Opinión, Trucos y Tutoriales

In a small company with a small team handling few but solid products, a number
of spawned projects for customers, and fleets of cloud machines provisioned for
serving the former; automation is key, even though not always necessarily
silver bullet. True complete automation reached in all technology operations is
utopia, or fake. Do not ever trust such claims.

Any operating system update can break the software, or the operating system,
even the underlying server in extreme cases. If you still have not found the
case, you better start applying operating system updates.

Completely automated information delivery is a different and healthy thing, for
the most part. Everything reducing fog of war in the modern company is key to
success, and maybe not as much recommended as it should be, when compared to
the continuous integration and continuous delivery groupthink of the second
half of the 2010s; not writing about the justified advocacy of the first half.
Current team and collaboration web application madness is good. Proper
resources discovery policies are gold.

Of course the modern platform thinking and current business constraints forces
us to treat servers as cattle, as everybody else, and some the information am
giving will not apply to servers such as web, application, data base read
replica servers or under certain safe k failure conditions elements in a read
write data base cluster. All of these am referring jointly as the pack.

Think more of a server that is ultimately responsible for the complete build of
absolutely everything. Or maybe a temporary architectural single point of
failure for convenience. Or instead something not critical at all, maybe not
even critical enough to be in the build and server orchestration
configurations, like to be expired proceedings for expiring customers. Think
about the phased death of biicode. Few companies disappear from dusk till dawn,
fewer technology platforms, even fewer in the case of technology platforms
intending becoming a build toolchain component. Bear in mind every single
company is doomed to disappear or else being intervened or merged. I just long
not to be there already when it gets to happen.

The pack action by default is just bring up a fresh machine and wipe out the
previous. For each case meriting distinction from the pack, a proper balance
must be reached when deciding update patching policy, between reaction speed to
the dreadier menaces, platform stability guarantees, low traffic hours, if any,
and operator time and involvement in the assessed stages of the proceedings, if

It does not really matter the degree of operator involvement with patch
applying you could want to use, like it or not, either way you are going to
know what your mileage is needing, and you are going to do what is cheaper in
your mileage.

Proper information delivery procedures are, instead, a must go. It is actually
quite simple, have here a snippet for servers info delivery:

You get to have a proper server discovery procedure first to populate your
server portfolio. That gets up to you, out of the scope of this article.

$srv_portfolio GETS RESULT OF CALL TO $populate_srv_portfolio

                                        FOR $server IN $srv_portfolio DO

                                        (*   '$type' is probably kind of an enumerated type, pun intended. *)
                                        $type GETS RESULT OF CALL TO $assess_type PASSING $server

                                        CALL TO $stack_msg PASSING RESULT OF
                                        CALL TO $check_for_updates PASSING $server AND $type

                                        (*   Here is where you do IRC instead of Slack,
                                        * for it is free so you won't get sold by. *)
                                        CALL TO $deliver_msg

That is all. Keep it simple. Nice pride ’17, stay safe, and do not take
anything for granted. Please do the challenge if you already have not.


Continúa leyendo este artículo

Continúa leyendo artículos en nuestro blog