Small shepherd's sundial

Some time ago, I made a simple wooden shepherd's sundial.  It's crude and doesn't work well, but I set it on a lanyard and occasionally use it.  When I decided to use it recently, my wife asked for a small one as a necklace.  Commissioned artwork time!

Since I wanted the sundial to be stylish as well as functional, I thought a bit about materials.  I ended up with a simple design that uses 3/8" copper tubing.  It's the same stuff that you can get at the hardware store for various plumbing jobs, but it's small enough for the task.  I traced out the hour lines using a short python script, and printed them out for use as a guide.

The gnomon is held in a steel cylinder with a slight lip to retain it inside the copper tubing, which has a matching (reverse) lip bored in the end.  You slide the cylinder in from the bottom of the sundial.  Boring the copper was a bit of a problem until I got the boring bit set properly.  I also ended up bursting through the side of the first cylinder because I didn't correctly account for the thickness of the tubing. 

I cut the tubing with a jeweler's saw and filed it to final shape for the noon curve.   The cylinder has two drillings: an axial drilling for the hanging loop and a radial drilling for the gnomon.  The hanging loop is just a piece of brass wire bent into a loop at the top.  I thought of soldering it in place, but decided not to.  I hammered the bottom end of the wire into a small rivet, so the loop is free to turn. 

Here is a trial assembly.

I spent considerable time polishing the copper, the pin, and the cylinder.  The machining was not too arduous, though it did take a few tries, but the engraving was much more tricky.

Here's the final product.

I did not use a watchmaker's square graver (too unwieldy on the round surface) nor the more usual round graver (difficult to get it to bite consistently) to do the engraving.  After a bit of trial on a scrap piece of copper tubing, I found that a very small screwdriver that I had previously sharpened to a narrow cutting blade worked much better for engraving.

To keep everything aligned, I taped the printout to the surface of the copper, and cut through the paper.  This got the copper marked in roughly the right places.  Then I removed the paper and finished the engraving under the microscope.  This took about an hour, and was a bit nerve-wracking because every slip-up is visible.  I'm not too proud of how the engraving came out -- many slips, wiggles, and other awful mishaps are visible if you look closely.  However, it doesn't look embarrassingly bad if you don't use a microscope. 

The sundial certainly looks nice enough on its own, but I am not sure how practical the shiny surface will be in use.  It's cloudy now, which might be for the best!

Fixing a violin bow ferrule

The ferrule on my son's violin bow snapped open.  This is the little metal piece near the end of the frog...  You can see the split along the edge.  I'm holding the two pieces together in this picture.

This not being a particularly expensive bow, I decided to try my hand at repairing it.  I am not a luthier -- any more than I am a watchmaker or clockmaker; that doesn't stop me -- and this is not a usual repair...  That said, it's not particularly difficult either.  I was inspired by the instructions on this page.

These are the parts that need to be removed to access the ferrule.

 

The idea is that ferrules are typically soldered together from two pieces: a flat piece and a round piece.  This video explains the process, which reminds me of many jewelry soldering tasks.

But since this bow was not that expensive, I think the ferrule was assembled as one piece.  From looking inside the break, it looked like the ferrule was brass plated with something shiny.  It is slightly magnetic, so perhaps it's stainless steel? 

In any case, I can solder brass with a little flux!  So apply flux wherever you want solder... and don't put it where you don't!

The task is basically a soldering job and a jewelry job.  For that, I like to use fairly soft, lead-free silver solder.  It's easy enough to slice off a small chip to use with a chisel.

Bind the works up with soft iron wire.  This holds the ferrule together as you solder, and gives you a handle that is far from the flame!

If you're careful, you can plant the solder chip in the blob of flux.  Then, when you heat the joint, the flux melts, and then the solder follows the flux into the joint.  Be very careful to apply heat slowly, so that the flux doesn't splatter and knock the solder away!  Also, if you overheat the joint, the flux will burn away, leaving oxidization everywhere, and then the solder won't stick to the joint.  Worse, it will probably stick somewhere else...

I didn't photograph the soldering process itself because I didn't have enough hands free.  Once the soldering is done, there is some scaling, oxidization, and flux.

I like to dissolve most of the flux in isopropyl alcohol.

Then you can file away the excess scaling and solder (gently!)...

Try the fit of the ferrule on the frog. 

File until it slides on with some friction, but not too much.  It's difficult to describe what you're after, but it's very easy to feel what "too tight" feels like.  I also interleave the job of polishing with the job of fitting, since polishing the inside of the ferrule adjusts the fit to the frog.  Because the solder made a bit of a fillet, which I wanted to retain, I also filed the frog a little bit.  You wouldn't want to do this to an expensive bow, but this bow had a plastic frog, and didn't seem to mind the offense.

To start the polishing, I start with sandpaper.  Things were clean enough that I could start to sand with 600 grit

... and then move to 1500 grit

... and finally polish all surfaces.

I like to use premade polishing paste in a stick.  Even though it's intended to be used on a powered buffing wheel, I just scrape the stick onto a piece of fabric and polish by hand.  You have better control of the polishing of small parts that way.

Keep trying the fit of the ferrule on the frog as you polish, until it fits perfectly and is polished.

Slide the ferrule onto the bow hair.  Make sure it's oriented in the right direction!

This bow has the hair anchored by a screw...  Slide the cover on...

Slide the ferrule on..

And press the spreader in place.  The spreader may be a bit tighter than before, since the solder joint takes up some space.  Don't force it!  I did force, and was rewarded with a split solder joint.  I had to start back at the beginning!  If it doesn't fit, carefully file the spreader so that it does....

And finish the job by reassembling the frog.

Antiques and heirlooms

Among amateur watch repairers, it is good advice refuse to work on antiques or valuable heirlooms.  Doing so without sufficient experience can lead to frustration!  Old watches also have non-standard parts and other oddities that can trip up the unsuspecting would-be repairer.  I have followed this advice until recently... and ended up in a bit of an adventure.

What tempted me down this path was an antique heirloom watch that was in very poor shape.

It was apparently my great grandmother's watch.  It is quite pretty, but when I received it, it had not been well kept by its owner.  Furthermore, it had been to some very poor watchmakers!  Slowly, over the course of three years, I made a number of repairs, none too difficult:

1. I replaced the missing glass with an acrylic face.
2. The case wouldn't close because it was slightly out of round... I put it back into the round, so the case opens and closes correctly.
3. I remade a case screw that someone had snapped the head in half (!). There are marks on the plate where the screwdriver skidded off after snapping the screw, so clearly way too much force was involved.  Ouch!
4. The winding stem that was acting up, because a brake was tensioned incorrectly.
5. I put the hairspring back in order because it was twisted badly.
6. Finally, recently, I took the movement apart and cleaned it.  This was completely uneventful.

It now runs reliably, but gains about 5 minutes per day.  That's terrible, of course.  But given that I don't have a replacement hairspring and given the age of the watch, it's well within what I'm willing to tolerate.

I had a very different experience -- and it's not over -- with a watch from my wife's family.

This watch wouldn't run, but didn't have anything visibly wrong with it.  Upon disassembly, I found that the lever fork cock was ever so slightly bent and was pinning the fork down.  Since it couldn't move very well, that kept the train stuck.  That wasn't the only problem...

As I was reassembling the balance, I was a bit less careful than I should have been.  I noticed the balance was not able to turn freely in both directions.  The balance was simply on the wrong side of the fork, and needed to be carefully lifted over the fork.  Unfortunately,  I couldn't visualize what was wrong.  Instead of the correct move, I gave the balance a sound push the other way.  That was not a good plan, and I really did know better!  Disaster!  The impulse pallet jewel was shattered!

After carefully sizing the missing pallet (seems to be about 19/1000"), I realized that it would be useful to make the pallet out of blued steel.  It should have nearly the same characteristics, and it should "just work."  (I also ordered a ruby impulse pallet, but I'm not confident of my measurements...)

It took five tries to make and install the impulse pallet, because I lost the first three pallets I made, and although the fourth worked, it was very poorly made.  The fifth looks nice enough and seems to be working. 

Of course, the first step was to disassemble the balance.  I removed the old pallet by heating the roller table with a soldering iron with a fine tip at roughly 250 degrees F.  Using a needle, I cracked off all the old shellac.

First, I trued up the end of a 0.5mm blued steel rod, using an arkansas slip.

After truing, I rounded and burnished the end.  Next, under the microscope, and on a soft piece of wood, I ground a flat surface using the arkansas slip.  In the picture below, the flat is visible as the slight smear at the end of the rod.  (The flat is about 1 mm long.)

This made the rod into a D shape.  I kept grinding the flat until the rod barely entered the hole in the roller table.  Then, I burnished the flat surface.

Unfortunately, the rod was too large to fit the slot in the lever fork.

So, I ground a fine taper to the rod by hand, by eye, under the microscope until the rod would just barely bind in the lever fork as I rotated it with the tip in the fork.

Then I switched to a steel burnisher to smooth the surface of the rod.  After burnishing, the rod freely fit the lever fork, with the base of the taper still fitting the roller table.

To cut the pallet off the rod, it's important to cut it to the precise length.  Since I ended up making five pallets, I was able to zero in on the correct length by comparing with an earlier attempt.  But in any case, I reversed the rod in the pin vise, and then used a narrow file as a saw to slice off the pallet.

Once again, I switched back to the arkansas slip to true the end, and then polished it with the burnisher.  Even though this cut end is completely invisible, this was an important step because the cutting raised burrs.  Burnishing removed those burrs.

OK, now the pallet was small!  It's the apparent "grain of sand" on the left of the frame in the balance cock in the picture below.

Getting the impulse pallet into the hole in the roller table was... irritating.  The pallet is slippery by design, and rounded... which means that holding it too tightly (or gripping the wrong surface) with tweezers will launch it across the room.  I lost three attempts irretrievably and lost another one for a very long time.  Eventually I got it pressed in place.

To anchor the pallet, I gripped the balance in surgical clamps and applied shellac.

Someone before me was way too liberal with the shellac and glued together several turns of the hairspring.  I was able to knock them apart, but if it impacts timing, I may have to immerse the spring in alcohol.  I'm loathe to do that because the spring has a nice overcoil and is very, very delicate!

Here is the balance reassembled with the new impulse pin installed in the roller table.

And a zoom-in:

 
But the balance reassembly did not entirely go well the last time...  I bumped the attachment point with the tweezers, and snapped the spring at the balance attachment point.  Previously, I had thought it nice that there was a removable clamp that made removing the balance easy.  However, that clamp was installed very permanently onto the hairspring.  It took a good two hours to remove the old pin (brass, extremely tightly fit; I think it was staked in place at the factory!), fashion a new pin (I used soft copper, and much longer, since I wanted be able to get it out in the future if needed), and get everything back to working order.  As wooden clockmaker Clayton Boyer says, "Mistakes take a lot of time. I spend some of my best woodworking time making them."  Yeah, I agree.

Still, after all that, the watch still would not run.  What!?!  No visible trouble... actually, not.  I had never worked with a watch with an overcoiled hairspring.  It can happen that the overcoil turn can foul on both the balance cock and the main turns of the spring.  That was the problem.  A very slight touch on the spring attachment point fixed the overcoil's path, and the watch sprung to life!

The movement is now running, which is a relief!  It sounds different from the other movements I've repaired, because it's quite a bit lower frequency (around 2 Hz) than other movements of the same size (typically 4-6 Hz).  Assuming it all goes back together, I wonder if it will keep reasonable time?

One final thing, though.  The winding works for this watch is not the usual Swiss mechanism.  It's a bit simplified, which seems elegant at first.  Here it is when correctly assembled.

However, should the stem get loose and you simply try to reinsert it, you will likely knock the cylinder gear assembly off its retaining spring.  This is rather subtle.  In the picture below, the lower retaining spring is supposed to be running in the cylindrical groove right below it.  All you need to do to repair it is slip the stem back in, and push the spring down gently.  It reseats easily, and then you can carefully withdraw the stem, leaving everything in place.

This process is extremely annoying, because although fixing it takes mere seconds, to access this mechanism you have to uncase the movement, dismantle the hands, and dismantle the dial.  Reassembling the movement into the case involves reinserting the stem, which has a tendency of knocking the winding mechanism out of order... so you need to back up, disassemble everything and try again...

Fluidyne experiments

A fluidyne is a piston-driving heat engine whose only moving part consists of a column of fluid (usually water).  It is a kind of Stirling engine, and it seemed like a neat idea to try to make.  The initial design was invented by C. West, and is described in a nicely written report.

Rather than starting with the "easy" standard design, I tried making a 3-piston version described later in West's report, after describing his initial design.  Here's my rendition.

I did some analysis of this design.  My model sets a displacement for each piston (water column) and temperature for each cylinder (air column).  Half of each cylinder is hot, and half is cold.  The engine is driven into oscillation because the cylinder isn't centered between these two sides, so sometimes it's heating and sometimes it's cooling.  The weight of the pistons acts as restoring force, so assuming they're all of the same length, they have the same resonant frequency.  Surprisingly, the cross sectional area of the cylinders and the length of the cylinders doesn't matter in the analysis.

Here is a typical plot of running frequency as a function of the length of the pistons and temperature difference. 

The engine seems to stall below a certain temperature difference between hot and cold sides, and the stall temperature depends strongly on the coefficient of temperature flow into/out of the cylinders.

The way the engine is supposed to run is that you point a heat gun at the three copper tubes on one side.  These are the hot sides of the cylinders, while the others are cold.

The sad part is that while this engine runs beautifully in theory, I could not get mine to start.  Keeping the temperature of all cylinders the same is critical or the pistons get out of order, and even with that managed, it never started. 

Just to verify that I wasn't completely crazy, I decided to modify the plumbing to make a more standard single piston machine.

I didn't do the analysis of this engine, but it also seemed very unwilling to run unless I flooded the hot cylinders with the pistons. 

Under this condition, the tops of the pistons boiled, which raised the pressure substantially.  Since the water vapor then condensed on the other (cold) side, the piston didn't boil away completely! 

Just to make sure that the fluidyne plumbing wasn't superfluous, I disconnected the plumbing and flooded the hot cylinder on its own. 

Here is a video of this test. 

You can see the top surface of the cold side of the piston oscillating slightly, but much less than when plumbed.  So the fluidyne was "running", even though the boiling seemed very wrong, at least from the original report's description.

However, at this point I recalled reading a rather different design by Morris Dovey.  He described a simple prototype that seemed worth trying, at least by modifying the materials I had at hand.  Although he claims that his design does not depend on resonant frequencies to run, that is dubious at best.  Forced or loaded resonances can vary a bit from their natural frequencies anyway.

I disconnected most of the pipes, added the cold cylinder just hanging off to the side of the hot cylinder, and plugged the cold cylinder with a small screwdriver.

This machine really runs!

It even (sometimes) starts nicely...

But I ran into temperature problems all around, as the clear vinyl tubing is not really able to take the heat from the heat gun.  After a minute or so of running, the tubing invariably springs a leak and the machine stops.

I tried flipping the positions of the hot and cold cylinders -- basically just plugging the end of the hot cylinder and leaving a big air bubble below it -- but this did not work.  I have not done the analysis of Dovey's design to get a more analytical handle on its operating regimes, but it is still definitely a resonant device.  You can feel it try to start running once the heat is sufficient, which is a sure sign that there's a forced resonance.

Misadventures with an ESP-01 board (ESP8266 wifi module)

To add wifi capability to an electronics project, the ESP8266 chip is apparently a popular choice.  It is a microcontroller with a wireless ethernet stack built in.  We got three ESP-01 boards that feature this chip, an antenna, and some flash memory (1 MB).  They are very small!

The tutorials on this board are generally quite confusing, as you can incorporate it easily into a project of your choosing in many ways.  Many makers are familiar with the Arduino platform and are comfortable with its development tools, even though this isn't entirely necessary.  Also, there appears to be a tradition of "hello world = flashing LED" tutorials for Arduino.  This tradition extends to ESP-01 boards in a somewhat unfortunate way. 

The ESP-01 board ships with an old-school Hayes-modem "AT" command interface.  It talks to a host via serial at 115200 baud out of the box -- though with 3.3 V signaling instead of 5 V or RS-232 levels.  (Nearly everyone warns you that 5 V will kill the device, but that actually seems a bit suspect.  I imagine RS-232 levels would really toast it, though!)  In any case, if you hook up a serial terminal, you can talk AT commands and pretty easily connect the board to a nearby access point.  The commands are

AT+CWMODE_CUR=3
AT+CWJAP_CUR="<ssid>","<pwd>"

If you want to see what access points are visible, use

AT+CWMODE_CUR=3
AT+CWLAP

If you want to see what IP you've been assigned once connected, use

AT+CIFSR

The documentation for all the commands is located here. a nice short listing of commands is here.  It's essentially like a cross between old-school modem commands and sockets programming.

Since modern computers don't have a hardware serial port (and most old hardware serial ports don't support 115200 baud), you typically use a USB-to-serial converter.  There are many of these available -- I have several -- but for a quick hack you can coerce an Arduino into helping.  Actually, "coerce" is the right word, because what you do is hold the Arduino in a reset state (it's effectively disabled) while you hijack its USB-to-serial chip.  Here's the circuit that I used.  The resistors form a voltage divider that brings the Arduino's 5 V signaling down to 3.3 ish volts.  (It didn't make much of a difference when I hooked the RX lines directly together, though that is apparently ill-advised...) 

Since I am using an Arduino Mega held in reset, it seems that plenty of 3.3 V is available for the ESP-01, contrary to the many tutorials warning that a separate power supply is required.  I imagine you might have problems if you allow the Arduino to boot up.

OK, back to the unfortunate part of the story.  Since the tradition is to make an LED flash as your first project, there are many tutorials explaining how to make an ESP8266 flash an LED.  This involves -- naturally enough -- overwriting the flash memory with a program that simply toggles one of the GPIO pins.  However, this means that the modem firmware gets overwritten!  If you're a newbie (or even if you're not), this means that the ESP-01 board is now useless as a wifi dongle!  Putting the original firmware back is not exactly a trivial task.

I was faced with the task of reflashing 2 of the 3 ESP-01 boards we have.  (The remaining one was fortunately unharmed, so I was able to figure out how it was supposed to work!)

First of all, you need special software that understands how to flash memory on the board with the ESP8266 chip.  Fortunately, the right tool for the job is an open source python script esptool which has minimal dependencies.  Starting this up was no problem, and I started by re-using my existing circuit.  I figured that I would start by clearing the flash memory, since I already knew it to be useless...

$ python esptool.py --port /dev/ttyACM0 erase_flash

Sadly, this just hung while "connecting".  I hooked up my logic analyzer to the TX and RX lines, and was able to verify that data was getting transferred, so it looked like the chip was merely ignoring the commands.  After digging some more, I found that the right course of action was to ground the GPIO0 pin first.  This puts the ESP8266 chip into a "flash mode", which is required for programming it.  To start programming, you briefly pulse the RST pin to ground to reset the chip.  Then it happily started flashing the memory, or so I thought. 

However, as it turns out my board has a flash chip that doesn't like the default SPI settings.  Unfortunately, they appear to work just fine, and the files I flashed even verify correctly through esptool.  But they are wrong!  There are some dark words in Amazon reviews for the ESP-01 about defective boards, but that didn't seem to add up.  The boards were indeed talking -- I could watch the data transfer both directions on my logic analyzer -- so I didn't think they were defective. 

After a bit more poking, I found the solution.  The problem is that the default setting in epstool optimizes runtime, using the fastest protocol to write the flash.  Apparently this isn't supported by all flash chips, and it silently causes problems with some of them.  There's a detailed explanation of what's going on in the esptool documentation and on the Wikipedia page for the Serial Peripheral Interface. 

OK.

Let's flash the board correctly!

You can use the stock firmware, by following detailed instructions about the flash map in their documentation.  But there is also an enhanced version of the firmware that adds a few additional AT commands.  It's a little easier to flash the enhanced version since it's all in one contiguous block starting at address 0x00000 rather than several separate files flashed into different addresses, which is how the stock firmware is distributed.  In any case, the command to do the job is

$ python esptool.py --port /dev/ttyACM0 write_flash 0x00000 AiThinker_ESP8266_DIO_8M_8M_20160615_V1.5.4.bin --flash_mode dout

This set things right, and the boards once again respond to AT commands!

Clock 4 now runs with intermittent impulsing

"Intermittent" can be a problem, but not in this case!  Based on the numerous power budget calculations I've done, impulsing the pendulum in Clock 4 every minute is too much to ask.  After having gotten the escapement to impulse every period (2 seconds) reliably, with run times around 8 hours, it seemed like the right time to go back to trying to get the intermittent part working again.  Especially, the run times without intermittent escaping were limited by drive cord length -- I had a four fall pulley in place for the clock to run that long. 

Therefore, I added more deep cuts to the count wheel, now five in total.

This means that the escapement should be triggered every 30/5 = 6 pendulum periods, or every 12 seconds.  The pin wheel has 30 pins, so will then have a period of 12 seconds * 30 = 360 seconds = 6 minutes.  The pin wheel is driven through a 1:10 mesh for the drive wheel, so it should make a rotation every hour.  I can therefore drive the minute hand from the drive wheel, although it will run counter clockwise.

With some tuning, the Clock 4 runs with 8 lb of drive weight, directly driving a barrel of 1.2 inches.  The clock's run isn't perfect, as (1) the count wheel double counts immediately following an impulse and (2) sometimes this double-counting skips over an impulse.

But given these issues, Theodore measures the following periods in current configuration:

• 53 seconds for the count wheel

• 4 minutes 24 seconds for pin wheel

Given these measurements the drive barrel will make one rotation about every 44 minutes.  In that time, the weight will have dropped 3.7 inches. 

Thus the power consumed is:

3.7 inches / (12 in/ft)  * 8 lb / (44 min * (60 s/min)) = 9.4 * 10^(-4) ft lb / s = 1.28 mW

More power calculations with Woodward's intermittent grasshopper

By joining the count wheel pusher lever of the the Woodward escapement to the escapement trigger, you can make the escapement trigger once per period.  This is the most frequent that the intermittent grasshopper can be triggered.  Triggering every period already happened by accident, but I decided to force it to occur by linking the mechanisms together without using the count wheel.  This way, I could debug the escapement mechanism... and there were indeed problems there.  I think I've resolved them, and this modified mechanism reliably runs until the weight hits the floor.

Currently, the mechanism runs on 1 lb 14 oz, falling 3.9 inches every 10 minutes.  Converting to standard units, this means that the weight falls

3.9 inches * 25.4 mm/inch / (10 min * 60 s/min) = 0.17 mm/s
1 lb 14 oz = 0.85 kg = 8.3 N

Thus the power consumption is 0.17 mm/s * 8.3 N = 1.37 mW.

This is substantially more pessimistic than my previous figure of 0.325 mW averaged over one minute for the count wheel assembly.  This is even with an improvement resulting from a few changes I made.  The pendulum is now hung from two sharp brass points resting in brass cups.

This new hanger ensures a positive positional lock and a definite axis of rotation for the pendulum with substantially less friction than before.

I also made a number of small improvements including reshaping one of the pin wheel pinion teeth, aligning the impulse hook, and stopping the detent's fall a bit earlier.  Finally, I removed every other pin in the pin wheel, which means that the period of the pin wheel is one minute.

Update: 5/13/2019.
By clipping off the tail of the locking detent to make it somewhat more delicately balanced, I can reduce the drive weight by 6.5 oz.  Thus, the power consumption is

3.9 inches * 25.4 mm/inch / (10 min * 60 s/min) * (1.47 lb * 4.43 N/lb) = 1.08 mW.

Restoring a pdp-11/45

When I was a student at RPI, I found a pdp-11/45 on the JEC loading dock.  The machine had no power supply and had a destroyed memory management board (M8108).  I cobbled together a power supply and replaced the board, and it was running well enough to flash its lights. 

That was about eleven years ago -- the little boy in the video above is turning 13 tomorrow -- and the machine stopped working shortly after the video was taken.

This month I started investigating the cause of the trouble.  I found that the +5V power was sagging severely when the machine was powered, which isn't terribly surprising because the CPU alone draws something like 40A!  Evidently the industrial power supply I had purchased wasn't doing its job.  Unfortunately, what used to be a $50 power supply was now upwards of $400, since the manufacturer had closed.  A little looking around, I decided to replace the +5V only at 50A. 

The resulting power supply got the machine to start up, but it was very erratic. It would crash at various weird places, and for no particular good reason.  This had me going for a long while, but fortunately the machine was made to be debugged... you can select from two internal clocks: a crystal (the default) and an RC oscillator.  You can adjust the frequency of the RC oscillator, but it's not very stable.  You can also wire in a single-step switch, which I had done in the past.  None of this seemed to help, but I learned that my frequency counter is pretty stable in the process.

Accidentally, I found that the machine would crash when I bumped the power supply wiring harness.... and found that the screws that connected the harness to the power supply were loose. 

Tightening the screws fixed that problem!

But there were more problems.  Several of them were easily resolved by polishing all of the card edge contacts.  This is not hard, but takes some doing.

This got the machine to the point where it had been before it stopped working.  It could load my custom, simple OS, EMON, and run programs entered from the console. 

But I had previously wanted to load other software, could I do that now?  Since I do not have any peripherals aside from serial lines, using a emulated TU58 tape drive seemed the best option.  Since I last checked about a decade ago, tu58fs has been written, and seems to be the most convenient.  So I tried to get XXDP and RT11 running, but both failed.

XXDP in particular was stopping at address 000550, this seemed to be similar to what was described here.  Using the disassembly provided there, I agreed that this was a checksum error.  But since the tape file booted on SIMH, I knew I had a good image. 

By manually digging through what the boot loader had loaded into the pdp-11's memory, I verified that everything was perfectly loaded.  The checksum was being computed incorrectly!  I wrote a simple program to compute the checksum of that good block... and the checksum came back wrong.  After simulating the code carefully in python, I determined that the cause was that the carry bit was getting set way too often. The carry was set whenever it should be, but also whenever the highest bit of the destination register was set.

I traced this back to the output of one multiplexer chip.  It was being held low when it shouldn't be, but only about 0.9V... an uncertain level.  Fearing the worst, I replaced the chip.

I socketed the replacement just in case I had to replace it again.  This is perhaps unnecessary, and it might cause issues if I ever get a floating point unit.  But since this board is the frontmost large board, it's not a problem.

Sadly, this didn't do the job... the carry still was getting set incorrectly.  Tracing forward, it turned out that the destination register's highest bit has a special active LOW line that runs on a trace parallel to the carry bit.  A tiny bit of solder had bridged those two lines at the following chip.

Evidently, this chip had been replaced previously -- not by me -- and the job was not done carefully.  Cleaning up the solder job and removing the bridge fixed the carry problem.

The machine now boots XXDP!

But, as the screenshot shows, there are still problems...  I originally had 32 kW of memory loaded, for a total of 28 kW available for programs.  (The top 4kW is always reserved for memory mapped I/O.)  I have verified that all of this memory can be accessed and used without issue.  Although the memory management unit is not needed to access 28 kW of memory, apparently XXDP prefers to have it available.  The memory management unit (embodied in the M8107 and M8108 boards) is actually installed, but it is evidently not working correctly.

In order to get XXDP to boot, I had to pull the top 16 kW out, which is the four cards above that are obviously displaced.  (The memory lives in a separate cabinet above the CPU because I had trouble finding a sufficiently powerful power supply for it.  Since it takes unusual voltages, I didn't fancy building an appropriate power supply from scratch.)

Here's a video of the entire boot process, in which I key in the boot loader using the front panel.