The Aristocrats

A team of engineers walk into a venture capitalist’s office with a pitch for a business idea.  Their hope is to get funding to launch their business.

VENTURE CAPITALIST: So, tell us about your business idea

LEAD ENGINEER:

We start out by creating an open development environment where every team member can commit code and make changes to the product.  There are a few guidelines but for the most part we use a ticketing system and everyone is their own boss when it comes to individual tasks.

There are no passwords and everyone gets access to the private key for our AWS Cloud Account.  No one has left yet but we don’t have a plan for locking down the system when someone leaves.  One time we were offline for a day because our developers didn’t know how to deal with a root drive that filled up on one of our Linux hosts.

Also we have lots of different code that comes from different places.  Some of them are third party developers in far-flung parts of the world that hate commercialism and aren’t big fans of Corporate America.  A few times one of our developers rooted us for fun just to show his friends.

Some of our stuff is really poorly written.  It’s using scripts we had someone write a few years ago and then we forgot how they work, but they seem to work for all of our continuous integration.  As long as we never reboot the system and leave the firewall wide open, it will just work with the third party tie-ins through HipChat and GitHub.

Also, our Sysadmin is a convicted felon.  He has to wear an ankle bracelet and isn’t allowed within a bank because he once hacked into the Automated Clearing House system and canceled a bunch of transactions.  He hates business people and has extreme political views and RON PAUL tattooed underneath his neckbeard.

Finally, we have no idea what we are doing.

 

VENTURE CAPITALIST (*shuddering*): What do you call your business?

 

LEAD ENGINEER: DevOps.

 

The Future of IT Staffing

For years every technologist has been talking about how cloud-based virtualization is the way to do business.  Instead of building a data center, customers can pay a few pennies per hour per virtual machine or even physical hardware and have exactly what resources they need for exactly the right amount of time.

With additional features such as auto-scaling and automatic shutoff, customers don’t need to worry about running up against resource limitations that could hamstring their application performance.  The old days of building a high fixed-cost data center and then waiting for the business to arrive are over.  Now, you sign up with one of many cloud providers and be up and running by the afternoon, doing investor demos by tomorrow morning.

The future of IT staffing will be no different.  For a lot of small companies, it makes very little sense to interview, hire, and train someone, only to have them leave to go work somewhere else.  There is no way to obligate someone contractually to work for you, no matter how much you pay them.

The solution to this conundrum will look something like the cloud-based virtualization model, but with people.  Instead of hiring a team of people and having them sit idle, the future of IT will involve project management software that automatically interfaces to outsourcing sites such as oDesk or Guru through some kind of API and automatically places a job description, takes bids based on some kind of evaluation heuristic, and automatically extends contracts to people who fit the acceptable parameters.

Though a feedback loop much like cloud-based monitoring, individuals will join and depart as they are needed.  The end result will be better usage of talent, and fewer labor market inefficiencies.  Best of all, small one and two person firms will be able to lay claim to a whole virtual board room of executive talent even if they only pay them a fraction of their salary.

What Does This Green Button Do?

Recently, I was explaining continuous integration to my parents. My mother is a retired OB Nurse who helped deliver hundreds of babies and my father is a retired broadcast engineer who helped deliver tens of thousands of hours of television. Sometimes I like to explain things to industry outsiders because it helps me solidify my own understanding of what exactly it is that I do.

A lot of my clients use Amazon Web Services such as EC2, RDS, and Route53 to launch their application, and a surprising number of them simply manage their instances from the Amazon Web Site or even a mobile app.  While these are the easiest ways do to it, it can be highly error prone trying to remember which size instances need to be launched.  In addition, there is a great degree of tedium that is involved in building security groups, and determining what ports to block.

Amazon has a great stack orchestration service called Cloudformation, that allows customers to upload a properly formatted JSON (JavaScript Object Notation) file to the site, and AWS will provision everything as described in the JSON.  It’s a great way to build a VPC (Virtual Private Cloud) quickly, and correctly every time.

Since the JSON file is an ASCII text file that is typically human edited, or perhaps generated from some other application, it lends itself to fit nicely into source control systems such as GitHub.  Configuration artifacts can be checked out using git, modified with regular expressions and deployed to Amazon using the command line API.

Initially I developed this as a Makefile, borrowing from decades of experience at the UNIX command line, then I realized I had access to a powerful CI (Continuous Integration) system known as Jenkins.  Jenkins allows you to check out, build, and package releases and then add any other desired post install tasks through a configurable interface.

After perfecting my JSON file for this particular client, I created a Jenkins project that checks out the JSON file, manipulates the entries in it with sed (Stream editor) and launches a complete stack, along with RDS instances, Route53 DNS configuration.  This became the de facto way for developers to launch practice environments.  I even added a timeout that would automatically terminate certain stacks after the end of the work day, like replicants from Blade Runner, in essence saving money by turning off lights, or in this case, expensive AWS resources.

This initial effort took about two weeks of my time to develop, and saved hundreds of hours allowing developers to build and deploy their own environments, like Vagrant but on a much larger scale and in about 17 minutes.  All they had to do was click the green button.

That’s how continuous integration should be.  It should be a moderate fixed cost effort, automating as much as possible, and then a low marginal cost effort.  If it takes 15 minutes of a developer’s time versus 2 weeks of a DevOps Engineer’s time, it’s a net savings of thousands of dollars.

Automate. Celebrate. Vacate

What’s the single biggest thing you can do to make yourself completely indispensable?  Automate yourself out of a job.  It seems counter-intuitive, but the truth is, unless you are a famous cardio-thoracic surgeon, a Soyuz Pilot, or Marilyn vos Savant, there are at least 250 other people in your city who can probably do what you do, better, faster, cheaper, and without complaining as much.  In case you were wondering, anyone can write these blog posts, and probably make them more interesting than I can.

To a large extent, employers keep knowledge workers around to extract marginal utility from them, whether it is in marketing communication, C++ programming, or quality assurance, with the understanding that rank and file employees will get their job done, meet or slightly exceed expectations, and stay out of trouble.  Most people at the center of the bell curve will not excel beyond their job duties, and that’s ok, because they don’t feel as passionately about the business as its founders or equity stakeholders.

Why is it good to automate your job away?  For the same reason that a good sales person can sell anything.  A good efficiency expert can optimize any job process or work flow, not just their own, and those are good people to have around.

When I was a kid, growing up in the Upper Midwest, there were still cars being manufactured in Detroit on a large scale, and things were assembled by hand.  The jobs paid very well, until robotics became commonplace and more and more dangerous occupations got replaced by servos, motors, and embedded systems.

Not very many people got out ahead of the automation, and sadly a lot of them lost their jobs and never re-adjusted.  The simple fact is, if it can be automated, it will be, and if it can improve the bottom line, and it’s good for the business, it will happen.

In the 1970s, the US went from being a labor intensive to capital intensive economy.  Because labor is relatively scarce compared to capital, anything that a knowledge worker can do to reduce labor costs will be recognized by good management, and if they are working for a company that is unable to recognize that, then their job and the job of the CEO will come to an end eventually, and that’s usually for the better.

Automate yourself out of a job, and if you are working for a company that is worth working for, they will give you more responsibilities and reward you with a bigger paycheck.

Virtual Office of the CIO

One of the greatest things about telework is the ability to hire a diverse team of people from all over the country, and even all over the world.

No longer are you stuck interviewing people, having to provide office space, a key card, parking, or even a laptop for them, as mobile computing technology has become so ubiquitous.

However, with telework comes the belief that if your contractors are out of site, they are out of mind, and likewise the concern that if you don’t see them on a daily basis they aren’t going to get the job done.

ThoughtWave Technologies is pleased to announce the Virtual OCIO service. Starting at $100/hour for as little as 8 hours a week, we will provide an on-site project coordinator supported by a back-end team comprising dedicated Linux or UNIX administrators, QA testers, technical writers, and even an administrative assistant, ready to deliver your project complete, on time, and within budget. The staff supporting your project will be dedicated to your effort — you won’t be subsidizing the high maintenance customers.

In addition, you’ll get our technical architecture expertise to make sure that not only is your project complete, on time, and within budget, but that it is done right.

Contact Jonathan Kalbfeld at 310-317-7934 for more information.

The Valuation of the Internet

Have you ever asked how big the Universe is? It’s about 93billion light years across. So, now that we are talking about big numbers, let’s talk about how big the Internet is.

The original 32-bit IPv4 address space is broken up into 5 “classes”, denoted by the letters A, B, C, D, and E. Over time, ISPs have adopted classless routing systems, so everything in the A, B, and C space is now a mishmash of address blocks varying in size from 256 on up to 16,777,216 IP addresses in a single block.

In the usable space, there are 222 routable “Class A” networks, starting at 1.0.0.0 and going up to 223, not including 127, which has a special purpose. Each “Class A”, or more modernly referred to as a /8 (stylized as ‘slash eight’), can hold up to 65536 routable networks of the smallest size.

The smallest block that is allowed to route independently on the Internet has approximately 256 IP addresses, of which 253 are usable, and 3 are reserved with a special meaning.

If you assign each usable IP address with some kind of monthly cash flow number, perhaps $12 per month, which is reasonable considering the cost of ISP services, data plans, hosting charges, cloud instances, web sites, you get an annual revenue of $144 per IP address. Doing a discounted cash flow analysis with a 5% growth rate, and the S&P growth rate at 11%, we arrive at a valuation of $1,602.84 per IP address. That means these tiny blocks of 253 IP addresses are worth about $405,518.52 for a single block.

A medium sized company might have 65,536 IP addresses, and they can be valued at $103,812,741.12. One hundred and three million dollars. There are a lot of companies out there with that kind of address space and nowhere near the valuation of $103M.

A large sized company like Apple, the Post Office, or Level 3 might have one or many Class A networks. If you extrapolate these out, you get a valuation of over $26.5billion per class A network. Know of a bankrupt company with a Class A?

In case you were wondering, the 222 usable Class A networks are worth $5,899,885,703,331.84 using this model.

How to Save Money In The Cloud Without Really Trying

Most customers using Amazon Web Services are way over-provisioned.  They have come from a background where their applications are running on static hardware, because that’s The Company Way, as Mr. Twimble would say.

The truth is, optimizing your resource usage should be a weekly exercise.  If your applications are properly redundant, there is no reason you can’t scale up and down your instances on demand and save a great deal.

For instance, Amazon’s top end RDS (Relational Database Service) instance costs about $6,500 a month running in redundant mode in Northern California.  That gives you 244GB of usable memory.  If you are only using 2% of your CPU and you are doing 500 queries a minute, you don’t need that.  You can scale it back to a slightly less powerful machine, and even without a reserved instance, save $5000 a month.  Or you can hire ThoughtWave Technologies to do this on a weekly basis, and the ongoing fee would be tied directly to the cost savings baselined from the start of engagement.  If we can’t save you any money in a particular month, you don’t have to pay us for that month.

Contact Jonathan Kalbfeld at +1 310 317 7933 for more information.

Don’t quit your day job

I once had an opportunity in 1999 to share an elevator in a sprawling Menlo Park office complex with a luminary of the venture capital world. At the time, my employer, Webvan, was preparing to list on the Nasdaq as WBVN.

For one reason or another, I was in these offices on Sand Hill Rd., and got 30 seconds to ask him his best advice to a budding entrepreneur. I was 22 at the time, and had an idea for a company that would allow people to send postal messages directly from their computer to people who had not yet connected to the Internet. At the time, access was very costly, and most mobile phone plans did not include data. If they did, it was not reliable.

His advice to me was “Don’t quit your day job, kid!”

I thought it was derisive at first, but he had not yet had enough of an opportunity to really gauge me as a person, and I decided not to take it personally. It wasn’t until February of this year, at the ripe old age of 37, that I discovered what he really meant.

He could have just said “your best seed capital is your paycheck.”

All of the best entrepreneurs were either in school full time or working for someone else. It makes sense. You get paid a salary which keeps the lights on and the water running, and the rent or mortgage paid, and then with your free time, you can think of something amazing. In fact, having only 2-3 hours a day to work on something forces you to learn time management, and gives you definite time when you have to be productive. If you are doing it full time, you will actually take your free time for granted.

If you are working on something else, like a creative endeavor, it’s the same story. Keep your day job. A number of my friends are producers, directors, and actors, and they make a healthy living JavaScript-ing, XML-ing, and python-ing while they work on their submission to Sundance.

If someone is going to pay you to think, take the money, do what they ask of you, and with your free time, work on your dream until you don’t have to work for them any more. Then, you will have developed the time management skills and you’ll be even more productive and successful. And if it doesn’t work out, you still have your day job.

I Haz A Sad

The geek community lost someone truly special today.  Someone who touched a lot of lives and inspired a lot of people.  Someone who may not be fortunate enough to have a movie or book made about her life, but who inspired so many of us to do amazing things.

I am speaking of my old friend Melody Bliss.  Back in the 1990s, I knew her as Melody Yoon, and she lived across the street from me when I rented a small one-bedroom apartment in Mountain View in 1997.  Mel would regale all of us with her expert wisdom on all things UNIX, Texas, gun safety, and kittens.

I was really a beginner, a noob as they say, and Mel was one of those people who always made time to explain things.  I used to joke that she taught me how to spell ‘ls’, the first UNIX command I ever used.  It’s probably actually true.

Melody taught me how to set up dialup networking on my Sun Sparc IPX, running the Solaris operating system.  She made things look easy, because they were easy for her.  I remember being completely dumbfounded and baffled when she set up my computer to use PPP over the Metricom Ricochet wireless service.

We all knew her as _Melody_ on IRC (Internet Relay Chat), and I believe she probably added the underscores in her name because she was so popular that it made an extra few steps so that she would get fewer chat messages and things would be manageable.

Mel had persistent health issues and lost her battle today.  We all lost her battle.  She will be truly missed.  Whatever you are doing, and whatever your field of expertise is, take a moment to think about someone who helped you when you were a beginner, and thank them.

The Cloud is Older than Your Mom

Not really, but it sounds cool.  People like to talk about the cloud like it is something new that was newly created in recent history.  In fact, the concept of virtualization as a service has been around since the 1960s, with mainframe technology.

In the 1990s I worked at a company called Sun Microsystems.  We used this fantastic software from Platform Computing called LSF, or Load Sharing Facility.  In a sense, it was a “cloud” of servers that were used for important things like simulating microprocessor regression testing, and doing off-hours massively-parallel analytics on a contract basis for other companies.

Later on, the world discovered server farms — specialized collections of load-balanced equipment designed to run important internet luminaries like Yahoo and Alta Vista and MySpace.

In 2007, when I was  at NASA’s Jet Propulsion Laboratory, I started exploring a piece of software called QEMU.  QEMU was a virtual machine emulator, that was a software-based system for running x86 operating systems on non-x86 platforms like SPARC, PowerPC, or ARM.

Over the years, QEMU adapted into something called KVM, which is Kernel Virtual Machine, a hypervisor that allows the simultaneous running of multiple operating systems on Intel-based hardware.

Now, instead of painstakingly installing each server and running the OS on bare metal, we can instead virtualize everything into a disposable container that is created and destroyed as it is needed.

Clouds are just made up of nodes, like undifferentiated stem cells.  They are still server farms, and load sharing facilities, and massively parallel systems, but we call them something else now.  Welcome to the 1960s!