I’ve been working with computers since July 5th, 1976. On that day, I walked into a computer store, the first in the state, and asked for a chance to work with computers.

The owner sat me down at a PolySci z80 computer, loaded BASIC on to it from cassette tape, then told me to have fun with it. He and his partner were having a discussion about me over at the side. I wondered over and asked them how to find the integer value of a number. They gave me the function.

A few minutes later, I had a stream of numbers floating up the screen. I had written my first computer program, a prime number sieve.

This was the first time I had sat down at a computer. It was the first time I had heard of BASIC. It was the first time I had written a computer program.

It worked.

I spent the next few years working with them. Biking, then when I got my license, driving the 5 miles to work every day. I headed off to University to study Computer Science.

The “microcomputers” I learned on were the 8080/z80 and the 6502. When I got to my first Computer Science course, I had to learn to use the punch card machines. Then feed the cards through the reader, wait for the “main frame” to process my job, after which the big printers would spit out my program and results, to be filed for me to pick up.

Whereas the microcomputers could run one program at a time for exactly one person, the owner of the hardware (person controlling the hardware), this main frame could run many programs at the same time. Each program belonged to a different person.

This computer was so powerful, it had multiple CPUs and even had auxiliary processors to help things move faster.

We had one big computer doing the work of many smaller computers. There were no personal computers in the computer science program. When they were finally introduced, they were used as “smart terminals”.

My point in this, is that we made one big computer to do the work of many smaller computers. We gave that one big computer extra as many extras as it could handle.

It had washing machine sized disk drives that held a stack of platters you could take out in a special container. I don’t remember how much they held, but it was likely measured in Megabytes.

When I left to work for the Army, we again had one big computer that many people logged into and did work. The first time I wrote a program on that Cray X/MP I was sure the computer was broken.

After you finish writing your program, you need to compile it. The command to do that is cc test.c Where test.c is the name of the file holding your code.

After the computer compiles your program, it then links your object with system libraries to create a file holding an executable program.

The Cray was so fast that it had finished the compile/link process between the time I pressed “enter” and the time the cursor had scrolled to the next line. I thought it had failed. It hadn’t. It was just that fast.

For the next 15 years, the team I was working with was attempting to create better and better “super computers” by clustering smaller computers together to form distributed parallel processors.

At the same time, we were working with new disk technology, RAID. With RAID, you could combine multiple disk drives to create a larger logical disk drive.

There were three primary technologies involved.

1. Mirrors. Exactly two drives hold exact copies of each other. These increase read performance and redundancy. A single disk failure does not cause the drive to fail, as the copy is good to go.
2. Striping. This takes two or more physical drives and treats them as if each drive contained a “stripe” of the logical drive. This could be used to make logical drives that were larger than any physical drive available.
3. Striping with parity. This takes three or more drives. When a block of data is written, it is written across all but one of the physical drives. The remaining drive holds the parity of each block.

We normally refer to “Striping with Parity” as RAID. In this mode, you can lose any one drive of the array and the remaining drives can be used to reconstruct the missing data.

This means that if you are using a RAID with 7 drives, you are get 6/7 of your disk space available to use.

All of this is very good.

Now we come to the problem, this large drive is attached to exactly one machine. That machine has to share it before other machines can use it, over the network.

You can find these for sale as Network Attached Storage, or NAS.

Mirroring and RAID have a limitation of the drives have to be nearly the same size. If you have 6 1T drives, the last drive has to be a 1T drive as well.

This means you are stuck. If you want to “upgrade” a RAID, you have to create a new RAID with the new drives, transfer all the data to the new drives, then decommission the old drive. Simply put, this is a painful process and not easy. I might have room for 6 drives in the case, I don’t have room for 6 more drives.

The file system which, in my opinion, did the best at this is ZFS. ZFS combined the low level mirroring, striping, and RAID with an actual file system. It was easy to use and had many great characteristics.

Friday, I had 4 ZFS pools of devices on two different computers. Total disk space exceeding 25 TB. The issue was that half of that space wasn’t usable by me. I wanted something that met my requirements of good redundancy and good file sharing and great stability.

I decided to use ceph.

With Ceph, you set up a replication policy. This policy tells ceph how many copies are needed and how distant they need to be from each other.

Consider a ceph cluster with 5 nodes, each node holds 2 Block Storage Devices (osd). If the policy simply has a replication policy of “have 2 copies”. Both copies might be on the same node. If that node fails, you lose your data, until the node is back up.

If you set your policy so that you require 2 copies and those copies are required to be on different nodes, your cluster doesn’t lose access to the data, even if one node goes down.

Ceph is even more powerful than that. It has a hierarchy of groups. Root, datacenter, room, row, pod, PDU, rack, chassis, and host.

A PDU is a power distribution unit. With this level of grouping, you could easily say: I want three copies. I should be able to survive any room in the data center going dark and not lose access to my data.

So where does this put me?

In a stupid place. I had 2 primary home nodes. Those nodes were configured to have one virtual node each, for a total of 4 nodes. This allowed a replication policy of 3 copies. No two on the same OSD. There must be a copy on a different host.

The problem with this, is that I am now burning three times as much space. There are ways with ceph to reduce this, I’ve not done it yet.

Using 3 times as much space meant that I couldn’t move the data I needed to free up the ZFS drives to use else where.

I was recently gifted a computer. That computer now has 4 drives in it. 3 TB, 12 TB, and 12 TB. In 30 minutes, I added 24 TB of space to the cluster, and a new node. I will add two more nodes to make this cluster safe and redundant.

I now have data flowing from ZFS to the new drives. Once that is completed, I will be able to take 12 TB that was part of a ZFS pool and turn them into OSDs.

Life is good, learning is good. Today is rearranging the shop day. I am going to take the CNC router apart, stash the important parts, ditch the things that aren’t worth keeping or turn them into metal for other uses.

This will give me back 1/3 of my shop space, and access to the shelves behind. The new workbench will have a place to go, and I will make progress on organizing my shop. To make it easier to work in.