• Member Since 2nd Nov, 2012
  • offline last seen 5 hours ago

Admiral Biscuit


Virtually invisible to PaulAsaran

More Blog Posts896

  • Tuesday
    Story Notes: Unity 2 (part 2)

    If you got here without reading the previous blog post or Unity 2 you're gonna be confused. Just scroll through for the pony pics, or maybe skim it in the hopes of finding a useful horse fact.


    Source

    Read More

    3 comments · 149 views
  • Monday
    March Music Monday 7 (bonus 3!)

    I promised you Silver Apples and you're gonna get Silver Apples. No, that's not a pony, but it sounds like it could be.


    Source

    Betcha can't name 'em all

    Read More

    10 comments · 170 views
  • 1 week
    Story Notes: Unity 2, part 1

    Here we goooooo! As I try and remember all the different obscure references I put in this thing. If I miss one, anthro Sparkler is gonna come after me.


    Source

    Read More

    11 comments · 266 views
  • 1 week
    March Music Monday 6 (bonus 2!)

    As one of my friends in high school once said, "Blow ye winds like the trumpets blow, but without all that :yay: noise."


    Source

    Read More

    15 comments · 177 views
  • 2 weeks
    Missing: Hobo Shoestring

    I don't have the reach that a lot of YouTubers do, but I've got some railfans in my readership and probably some people who live in Tennessee . . .

    Hobo Shoestring was an inspiration for Destination Unknown, and he's gone missing. Southern RailFan is leading a search effort at a lake he liked near his house; here's a video if you want details or think you might want to help:

    Read More

    17 comments · 513 views
Apr
9th
2019

Mechanic: Fuel Pump Control Module failure on a Silverado; Low Speed Network Failure on a Denali · 1:46am Apr 9th, 2019

Welcome back!

Today we’re going to lead off with an easy one, but don’t be caught off-guard. What follows was a descent into madness. Have your favorite beverage on standby.


Source


We’ll start off gentle. Our first patient is a 2009 Chevy Silverado. Nothing fancy; your basic red truck. It got towed in ‘cause it would crank but not start, and it only took me a moment to determine that it was lacking fuel pressure.

Y’all may remember me blogging about the fuel pump driver modules on Fords (FPDM), and how they usually fail instead of the fuel pump. Well, Chevy decided that they might jump on that bandwagon, too, and so some Chevy trucks have a module that tells the fuel pump what to do. Of course, they couldn’t call it a FPDM, so they called it the Fuel Pump Control Module (FPCM)*

I did a bit of looking on Identifix, and had a diagnostic strategy all ready to go, when my manager interrupted me and sort of changed the route I was going to take a bit. I was going to follow Identifix’s test, but he thought that I should instead pull codes first and then follow that diagnostic path. Whatever.

So I did, and not surprisingly the FPCM had a code: P025A Fuel Pump Module Control Circuit/Open. So I print out the diagnostic instructions for that, and set to testing.

Here are the Identifix instructions, reproduced in full. Read carefully, because you want to understand them completely.

1: Connect a test lamp between terminal 13--control circuit--and ground at the FPCM.
2: Using scan tool, command fuel pump on and off.

--If the test lamp is always ON, test the control circuit [13] for a short to voltage. If the circuit tests normal, replace the PCM.
--If the test lamp is always OFF, test the control circuit for a short to ground/open. If the circuit tests normal, replace the PCM.
--If test lamp responds to commands, replace FPCM.
__________________________________
*He also got bent out of shape with me calling it a FPCM, because “GM calls it the ‘Chassis Control Module (CCM).’” Later, he had to eat a little bit of crow, because GM only calls it the CCM if it also has trailer brake functionality built in; otherwise, it’s the FPCM, as attested to by the Snap-On scan too, Identifix, and also the GM scan tool with which I programmed the new module.

Here’s the kicker. After I did that test, which identified the FPCM as being at fault (the test light turned on and off at command), I asked him if there were any other tests I should run.

“Nah, I ordered one from GM 15 minutes ago.” That was before I’d even tested the thing.

“Okay.”

“I did print out the testing instructions from GM.”

“Was something wrong with the ones I got from Identifix?”

“Well, the GM ones were easier to understand.”

. . . .

Let’s have a show of hands from my readers for who didn’t understand the diagnostic instructions.


And now, kids, it’s time to crack open that beverage. We’re about to dive into the depths of a 2016 GMC Acadia with an intermittent problem, which is always a bit of a horrorshow.

Making it even worse, it’s an intermittent network problem.

So here’s the setup. It gets driven in. The customer said that it had not cranked over, and they thought the battery was dead. They called triple-A, got a jump start, and it fired right up and they drove it in.

It did not crank when I went to start it, so I put the jump box on it . . . and it still didn’t start. My first thought was that it was the starter failing, and that’s unfortunate, because you have to pull the catalytic converter in order to put a starter on it. Also, we’d have to push it in.

After a few more desperate attempts to start it, I told the manager that I thought it might be the starter, and he went out to get a look-see as well

Wouldn’t you know, it fired right up.

The first thing to do in a situation like that is pull codes. Maybe there’s a clue as to what went wrong in there.


Sometimes I list out all the codes for y’all. This time I’m not going to, and it will be apparent why this is just a small sample in a moment.

Body Control Module (BCM):
U0155 Lost communication with IPC
U0164 Lost communication with HVAC
U0198 Lost communication with Telematic Control Module
U0200 Lost communication with Passenger Door Module
U0214 Lost communication with RFA (I can’t remember off the top of my head which module that is)
U0330 Lost communication with Liftgate Control Module

Instrument Panel Cluster (IPC):
U0140 Lost communication with BCM
U0164 Lost communication with HVAC
U0151 Lost communication with SDM (airbag)
U0159 Lost communication with Park Aid Module
U0168 Lost communication with Immobilizer
U0214 Lost communication with RFA
U0132 Lost communication with Side Object Detection
U023A Lost communication with Camera Control Module

And so it went.

Now, you might have noticed a pattern here. The repeated phrase “Lost communication with” implies a network failure.

I think I covered this before, but just in case, here’s a quick refresher. If you’ve got a bunch of modules on a network, and all of them say--for example--Lost communication with Passenger Door Module . . . except for the Passenger Door Module, who thinks everything is fine, you’ve identified which module is at fault.

If all the things go down, the network is at fault.


There are two ways a network fails. One of them is a broken or damaged wire. The other, harder-to-diagnose problem is that some module went rogue and is shouting nonsense onto the network and none of the other modules know how to deal with that. Especially since car networks (most networks, I’d assume) work by driving a voltage up or pulling it down, if you’ve got some module putting too much noise on the circuit, nobody can talk over that.

Of course, as you can imagine, the network wires run all over the automobile. And, as you can surely also imagine, the modules tend to be hidden, since customers don’t want their car to have lots of little boxes with wires that they can see.

So it was time to narrow this puppy down. What part of the network failed, what did the failed modules have in common, and so forth. One thing that’s often useful in this task is figuring out if all modules have failed past a certain geographical point (like, everything behind the front seats doesn’t work), or if it’s an entire network down (remember, most vehicles have several different networks).

I’ll put the network diagram right here:

I’ll also tell you that as I started to narrow the possibilities down, the only modules that would not communicate consistently were the Passenger Side Object Detection Module, the Digital Radio Receiver, and the Personal Audio Link Control Module. So if you like a good mystery, have a crack at the wiring diagram and feel free to let me know what you come up with down in the comments.

If you’ve got no idea and would rather read about me struggling, continue on.


The first thing I came across was a technical service bulletin (TSB) indicating that the wiring run above the headliner could have been pinched between the rear HVAC duct and the roof during assembly, which could cause the symptoms and codes I had. There was no other useful information I could find in Identifix.

So, I took the rear trim panels off, didn’t disarm the side curtain airbags because sometimes I like to live dangerously, and peeled the headliner back. This is the point where I should mention that the woman who owns this thing is a gentleman farmer, and she uses it as a farm truck. A 2016 Acadia Denali! So I had to remove two dog crates and of course the whole cargo area was thick with dog hair. And at this point, things hadn’t started to take their downhill slide.

It took a little bit of doing, but I found the wire run, and it was indeed tucked above the duct, and I was expecting this to turn into an easy fix after all, so I pulled it as free as I could and inspected it all over for any sign of chafing.

Nothing.

It was in perfect condition.


I believe the module you’re seeing front-and-center is the RCDLR [remote control door lock receiver], and you probably can’t see the duct in the picture since it’s dark above the headliner. Point is, there are modules in the attic of late-model cars!


It was at this point that I needed to bring out the big guns. Specifically, the GM scan tool.

You see, one of the features that GM recently introduced was a network diagnosis function. You could monitor the voltages on a particular network, which was nice (it would also tell you if a particular network was online or not [or not equipped on that particular vehicle, as the case may be]). And on top of that, they had the Holy Grail: an all-modules scan that displayed in nearly real time which modules had codes, which ones didn’t, and which ones were offline.

I also puzzled over the wiring diagram. If you scroll back up to it, you’ll notice that the modules I mentioned before mostly seemed to be behind splice pack JX405. That splice pack was conveniently located right behind the jack, and not terribly difficult to access.

This led me to believe that the issue might lie somewhere between the D-pillar and the roof, or at least in that general area.


Of course, the problem with it being intermittent is that . . . well, it’s intermittent. The Acadia started more often than not, and I quickly grew complacent. I reasoned that the modules that were constantly offline--the Passenger Side Object Detection Module, the Digital Radio Receiver (DRR), and the Personal Audio Link Control Module . . . well, they might be a problem, but it started when they were offline. This was my first mistake (but an understandable one).

I also had already determined--and some of you might have, too--that the Personal Audio Link Control Module and the Passenger Side Object Detection Module are not listed on the computer data lines circuit diagram (scroll up to verify if you don’t believe me). Obviously, they are on the low-speed network, but they aren’t shown.

It was time for more research.


It was also about this time that I got a bit more information. The side object detection had stopped working over the weekend. The customer was understandably more concerned about why her Acadia didn’t always start, but that was a bit of helpful information. So I unplugged both those modules. This was not as easy as it sounds. They’re located under the bumper cover, partially blocked by the mufflers on a normal vehicle, and in the case of this one, buried under what I hope was just mud, but she does have horses. I had to literally dig the passenger side one out so I could get to the connector. And when I finally did, I was hoping that it would be full of corrosion, because that would be the solution to my problem, but it wasn’t. It was just fine.

The vehicle continued to start reliably, as it had every time I’d checked since moving it into another bay. And the network stayed up.

During the course of my research, I discovered that the Passenger Side Object Detection Module is connected directly to the Driver Side Object Detection Module, who is connected to the network. This meant that a failure in the Driver Side Object Detection Module could potentially take down the Passenger Side Object Detection Module, and also explained why they hadn’t drawn it on the diagram. It wasn’t on the network, per se; it had a dedicated link to the other module, who then communicated with the network as a whole.

I was unable to discover what the Personal Audio Link Control Module is, or how it’s tied to the network. There was legit no documentation on it whatsoever.


The next idea I had was to start to disconnect things from the splice pack. If one module got removed, and all the problems went away, than that module was the culprit. Of course, since it was an intermittent problem, I wouldn’t know right away.

This is where things started to go south.

The proper way to do this is to literally unpin the terminals from the connector. That’s kind of a pain, so an alternate method is to make jumper wires in lieu of the comb, and remove them one at a time.

I didn’t have enough jumper wires to cover everything, so I thought I’d just do the circuits that were most interesting to me. Chief among them were the RCDLR, the DRR, the PAM and a few other things. I can’t remember exactly what, and it doesn’t matter. It seemed logical at the time.

I thought that maybe one of the modules that was offline--the DDR, and possibly the Personal Audio Link Control Module--would come back online when I disconnected the right thing.

It was a good thought.


Even with only four components still networked, the DDR did not come back online.

And before we move much further on, I want you to imagine this. I was in the cargo area of this thing, lying on my side, reaching through the hole where the jack goes, making sure that a bunch of jumper wires all stayed in place. I had my wiring diagrams around me, but the scan tool was all the way in the front, and I couldn’t really read the screen from where I was.

I should also point out, in case the picture that I ultimately upload doesn’t show it clearly, all the low-speed bus wires are green. They are all exactly identical in appearance.

So, I’d make a change and go look at the scan tool. Everything was operating normally, until I disconnected the RCDLR.

As soon as I saw the scan tool, I noted that it had stayed up.

I went back and looked. Its jumper wire was unplugged, and not touching anything where it might make a connection.

I verified with the wiring diagram. Pin 6 was the RCDLR, pin 7 was the PAM. I hadn’t accidentally unplugged the wrong thing, so this was a result, but what did it mean?

Then when I unplugged the PAM, the RCDLR did go down.

So I got my manager to look at the scan tool. I unplugged the PAM, and asked him if both modules had gone down. He said yes.


It was back to the diagram. How could this happen? It didn’t make any sense. Somehow, the network wire from the RCDLR had to have chafed through or broken completely, and miraculously managed to make contact with the network wire from the PAM, so that when the PAM went, they both went.

Or else the harness was somehow made wrong at the factory.

If the vehicle had been in an accident, it was possible, but it hadn’t. Just the same, I started wiggling all the wires I could get to, and I heard a faint clicking like a relay turning on.

It was reliable, repeatable, and I couldn’t see the scan tool from where I was, so I didn’t know if it was a module I cared about, but still, it was a result! Maybe there were more wires chafed through; maybe there was also a power wire involved and when things moved about just right, it shorted on the data line. It could have been for a function that the customer hadn’t tried to use, so she didn’t know it had failed.

I wanted to get a better sense of where it was coming from, so I put my ear right up against the lower trim panel and wiggled that wire. Sometimes I had to wait, because there was other noise in the shop and this was very faint.

After about five minutes, I determined that it was the signal wire from the antenna wiggling where it clipped into the DRR.


Speaking of the DDR. I’d been ignoring it since it was perpetually offline, but what if it was the culprit?

What if, while it was in benign failure mode now (the vehicle started every time, still), sometimes it just started shouting garbage out into the network?

My manager said that he thought it would be a good idea to unplug it and check the continuity of the data line, and I thought so to. I could get to it; it was also behind the jack, and right where I was working. So I got out my trusty Fluke, plugged it into the data line from the DRR, and then into the splice pack.

And I got nothing. Open circuit.


Clearly, we had a wiring problem.


Source
(I figured that we needed a pony picture to break up the Wall-O-Text)

We had a PAM and a RCDLR that were joined at the hip even though they weren’t supposed to be, and we had a DRR that had an open wire between it and the splice pack, and by golly, that was a pretty short run of wire and it didn’t take very long to inspect it with absolute confidence that I would find the problem and I would fix the problem and soon enough this would just a bad memory.

I even had a good place to look; also tucked back in that corner is the motor for the power liftgate, and that sucker’s got big sharp pointy teeth on the gears, just the thing to chew up some wires.

And of course that wasn’t the problem.

There was no damage to the wiring harness at all. Sure, I could have unwrapped all the hockey tape GM puts around it for protection (it’s not HPHT, so it’s not full protection), but there was no visible damage and it was all routed in its looms and clips.


This is when it hit me like a ton of bricks.

Every single wire in that connector is green, and since they’re all on the same network (that’s the point of a splice pack), it doesn’t really matter which cavity which wire is in. If somepony wasn’t paying attention when they put it together, if they switched a wire, it would still operate normally.

And in fact, that was the case with the DRR. Its wire was switched with the Auxiliary Radio/HVAC controls.

I also found out that the PAM and RCDLR’s wires were switched, too. Or else the wiring diagram was wrong, which is also a possibility.

That didn’t explain why they’d both gone down when the PAM was unplugged, though. So I tried that experiment again, armed with my newfound knowledge of which wire went where, and I learned that it behaved exactly as expected.

I don’t want to point hooves here, but my manager claims that I didn’t ask him if both modules went down when I unplugged the PAM. I’m pretty sure I did.


So I had nothing. I’d spent maybe four or five hours on this thing, never discovered what the Personal Audio Link Control Module does or where it is, and I was done with it. It was close to the end of the day anyway, so the best thing to do was plug everything back together and start over with a fresh approach in the morning.

When I plugged the DRR back in, it came online. Apparently, it had just wanted to be unplugged and then plugged back in again. This does occasionally happen to modules; they go stupid and all you’ve got to do is power cycle them and they’re happy again. Maybe it was downloading Windows updates and crashed in the process, I don’t know.

And I got back under the rear bumper and plugged the Side Object Detection Modules back in, too. Once again bravely sticking my hands up into what was hopefully mud, but she does have horses. . . .


The entire low speed network crashed.


Earlier, I told you how the GM scan tool could monitor module health. And I’ve got a picture of what that looks like from the earlier stages of diagnosis--before the low speed network crashed.

The green check marks mean all is okay, no codes. Yellow warning triangles mean that that module has codes, and is talking. Red circles with slashes mean no communication. I’m not entirely sure what the blue information circles mean (two modules have those). It wasn’t something that I was concerned about during the present stage of diagnosis.

At the moment this picture was taken, we just had the DRR offline, the Personal Audio Link Control Module, and the Right Side Object Detection Module.


Imagine in your mind that all of a sudden, most of those warning triangles were replaced with red circles, because that’s what happened. If you scroll up to the wiring diagram, everything that’s only got one green wire on the network diagram just dropped out.

Every single module on the low-speed network dropped out in one fell swoop.

It also didn’t start.*

It also presumably set about a billion new codes.

So once more, I got my manager, and I laid under the rear bumper of that thing and I reached up into what was hopefully just dirt and I unplugged the Passenger Side Object Detection Module and the network came back. Well, minus the Passenger Side Object Detection Module (obviously) and the Personal Audio Link Control Module which had never been online as long as I’d been diagnosing this thing.

Plugging and unplugging it produced consistent results. Well, sort of.

Most of the time, the whole network didn’t go down. To humanize the network a little bit, what would happen is the scan tool would politely ask, “Hey, BCM, you there?” And the BCM would say yes. And so on down the list.

And sometimes when the scan tool asked, the Passenger Side Object Detection Module was screaming its death agonies over the network, and the scan tool would say, “I don’t know what that is,” and flag the module it had just tried to ping as offline.

It was actually cool to watch.

______________________________________________
*I never entirely explained the reason why the network failure caused a no-start (at least, I don’t think I did). One of the affected modules was the theft module. When you turn the key, the PCM asks the theft module if it’s okay to start, and when the network was down, all the PCM got as a reply was the Passenger Side Object Detection Module screaming gibberish, so it just shrugged, said, “Well, that wasn’t what I was expecting,” and did nothing.


I think there are a few lessons to be learned here. I don’t know how applicable they’ll be to your particular situation, but I wouldn’t feel right not including them.

1. Trust, but verify. I assumed that the wiring diagram was correct, and/or the vehicle was wired correctly. This turned out to be a false assumption. Luckily, it didn’t take up too much time, and I didn’t spend overly long running down the wrong path. And I should have known better; this isn’t the first time I’ve diagnosed a vehicle with a network harness incorrectly wired.

2. Think about what you’re actually diagnosing. I spent more time obsessing over the DRR than I should have, although as I’m in the process of writing this blog, I don’t think I was wrong to consider it as being a potential problem.

2a. Sometimes there is more than one problem which--while they seem related--aren’t. In general, the simplest solution is the most likely, but sometimes more than one thing fails at or about the same time.

3. Isolating a network problem by unplugging modules is a valid strategy, but always remember what’s unplugged. I never had a failure to start after unplugging the Passenger Side Object Detection Module, but as I got deeper into the woods with no idea what was the actual problem was, I got so used to seeing that it was offline that I took it out of consideration fairly early in the diagnostic process--about once I started to obsess over the DRR.

4. Something that lives in a harsh environment is more likely to fail, no matter how well it was designed. Given that over the last couple of weeks, Michigan has ranged from double-digit negatives to 50, and has had 6 inches of snow, about a quarter inch of freezing rain, and a decent thaw cycle, and the Passenger Side Object Detection Module was buried under mud inside the rear bumper (and it’s also exposed to however much heat the muffler gives off), of all the components I considered to be potential failure points, that one should have been the most obvious. It’s just not living a happy life there under the bumper.


I never did find the Personal Audio Link Control Module, but I know where it went. But that’s a blog post for later!

Comments ( 50 )

Also to all of y’all in the comments--I have the wiring diagrams in PDF form, which will surely be clearer than they are on-screen, but I have no idea how to turn them into a useful image. I used MSPaint, then posted on Discord, which is not ideal for several reasons, but it’s what I know to do and it works.

Sweet mother of mercy. That's more network troubleshooting than I do in a month. I'm up to two different general answers for things like this on the desk:
1) Let's reimage it and see if it's still mucking up.
2) I'll call HP.

At least when I'm troubleshooting my widgets, I don't have to worry about hearing an earsplitting 'BANG!' and having little bits of airbag packing drift down all around me. I did however find out today where the HP Zbook 15 battery is located. Up until today, I have never needed a screwdriver to swap batteries. Now I need a teeny star wrench and a lot of patience. If I'm ever going through TSA and they ask me to take the battery out of my lappie, I'm going to laugh.

Well that was a right old mess, clearly someone was skimping on the appeasement of the machine spirit. Try burning more incense and giving the car a thorough cleaning, that should help.

If not, you could call tech support.

derpicdn.net/img/2017/5/26/1446693/large.png

The limits of my car mechanic skills are:
I can change the air filter, the oil & the oil filter
I can replace spark plugs if I do them 1 at a time so that I don't mix up the wires

I understood those directions & believe that I could follow them,
if I had the list handy as a reference guide.

I used to do professional mechanic work for Toyota and Uncle Sam. Those diagnostic instructions looked 5th grade simple to me.

Sounds like you had fun.

Also, is the Personal Audio Link bs the device in the back of the center console (usually) that allows the rear passengers to control the radio, and sometimes has a headphone jack?

Can I say I think they are making these danged things toooooooo complex and someone living working on a farm is going to get into more gunk and garbage than someone using a truck as a daily driver. There things to be said for pickups from the 30-70s

Especially since car networks (most networks, I’d assume) work by driving a voltage up or pulling it down

Hi, former computer network tech here. On the physical level, yeah, everything is eventually electrical signals. What you're describing in this case is called a 'network storm' or 'broadcast flood' and different technologies have ways of handling it. Modern computer networks use control devices that will not spam a whole network if one device goes rogue. That's the principle behind 'switched' networks and it bemuses me that cars don't have that same architecture. That technology has existed since the 70s!

My only thought is that switched networks require all the attached devices to go through one device (the switch or router) and perhaps that presents wiring challenges in a car environment.

Also that bat pony is so adorable it *hurts*.

I can see my area and the server storage it connects to, and have some visibility into. The network stuff is all magic pixie dust to me. Glad to see that translates into everything else as well.

My understanding of logic exceeds my understanding of my native English language with those three instructions. To me they simplyfy to, doesnt matter what you test, if you are looking at the device, replace it. :twilightoops:

You trusted your Manager. He is a network problem.

I thought an Audio Access Module or whatever, especially if theres no documentation, is that thing thats one of the options for Hackers to walk past the network security by adding command codes as an audio file, SMS, phone call etc.

Dont forget the basic tenant of complex technology.

Everything is Hostile.
Everythings out, to do what it can. :pinkiecrazy:

Oh man. This reminds me of the absurd amounts of BS we had to go through doing avionics maintenance. After four years, I was convinced that the only way those aircraft managed to fly was because even the ground wanted nothing to do with them.

FTL

5041261
Is it telling that after hearing so much about your manager that I was not surprised that he found those instructions confusing and that I was even less surprised that he had ordered parts before doing any tests? It does make me wonder just what GM's instructions were that he could apparently understand them easier... maybe more pictures? :pinkiehappy:

For the Acadia I was keen to have a shot and I was initially disappointed that I could not read the diagrams but then I thought to myself, "This is going to go to hell just like any other old 70's 'hub' or 'ring' based network" and I suddenly knew I wanted nothing to do with it! :twilightoops: (I started out in the late 80's maintaining and redesigning a system that was developed and built in '74) . Those old networks suffered bus 'storms' and such just like your modern cars... it boggles my mind that what the computer industry worked out was a really bad idea around 30 years ago is the communications topology that automotive engineers use as 'cutting edge' today. :twilightangry2: As you said, your reasoning was sound but the 'intermittent' fault part led you to distraction and heartache.

It was actually cool to watch.

In that "Train Wreck" sort of way, eh? :twilightsheepish:

I never did find the Personal Audio Link Control Module, but I know where it went. But that’s a blog post for later!

We wait with patience... this sounds like another fun story from the annals of "The Memoirs of the Long Suffering Mechanic"

5041409
That was what we used to say "Helicopters only stay up because the earth repels them and that is because it also thinks they are repulsive."

5041455
Well yeah because they shake so violently.

If I had to guess, the PALCM is like Apple CarPlay & others of that nature

Thank you for making your suffering so entertaining for us. These are way better than mystery novels.

But the troubleshooting never revealed the important question... Was it mud? (She did have horses afterall)

These posts are usually over my head but I always have to make time to read them. I find them so fascinating.

In my father's day, mechanics were just guys who liked cars, started messing with them &
learned to fix them by doing. When I was young (70s), they had mechanic schools, auto shop & stuff
but a lot of mechanics started as guys who liked cars, started messing with them, & learned to fix them.
Now? It sounds as complicated as repairing computers with the added advantage
that you get filthy doing it.

These days, does anyone get started just fooling around with cars, or are they too complex?

5041270

Sweet mother of mercy. That's more network troubleshooting than I do in a month.

And they say that mechanics are all high-school dropouts. :derpytongue2:

At least when I'm troubleshooting my widgets, I don't have to worry about hearing an earsplitting 'BANG!' and having little bits of airbag packing drift down all around me.

That hasn’t happened to me yet, but I’m not going to say it never will.

I did however find out today where the HP Zbook 15 battery is located. Up until today, I have never needed a screwdriver to swap batteries. Now I need a teeny star wrench and a lot of patience. If I'm ever going through TSA and they ask me to take the battery out of my lappie, I'm going to laugh.

Consider yourself lucky. Your laptop might have a cute little screw, but you don’t have to deal with automakers inventing new fasteners. You got a Craftsman socket set? How do you like this Torx screw? Oh, you bought a set of Torx drivers? How about a triple-square fastener? And you got that covered? How do you feel about a three-lobe?

5041881 "And we took five bucks off the bill because we pulled out all of the quad-lobed, three-angled special Ford-only connectors and replaced them with hex bolts. You're welcome."

5041272

Well that was a right old mess, clearly someone was skimping on the appeasement of the machine spirit. Try burning more incense and giving the car a thorough cleaning, that should help.

I’m paraphrasing, but Tony Koester once suggested appeasing the gods by stuffing the component in a barrel and banging it with a stick was a worthy avenue to pursue, and there are days where I think he’s not entirely wrong.

Or, to shift to the generally terrible movie The Rock, “I’m asking you as a friend . . . I’m ordering you as a superior officer . . . I’m giving you a last chance as a guy with a gun.”

5041288

I understood those directions & believe that I could follow them,
if I had the list handy as a reference guide.

Honestly, as much as I’d like to claim otherwise, a lot of my vast skills involve following reasonably simple directions. Do this, if this is X, do that, and if that is Y, replace the module.

With patience, and a bit of understanding of engineer speak, most things are doable for Joe Average.

5041289
Those diagnostic instructions were fifth-grade simple. Sadly, my manger isn’t always at that level.

5041297

Sounds like you had fun.

You know damn well how much fun I had. Both because you’ve been there, and because you know the kind of thing that becomes blog post worthy.

Also, is the Personal Audio Link bs the device in the back of the center console (usually) that allows the rear passengers to control the radio, and sometimes has a headphone jack?

That’s a good guess, but no. That’s the rear audio control module on this Denali. I honestly don’t know what the Personal Audio Link is.

5041333
Some of the complexity is in what your order--the customer didn’t have to have the side object detection, for example.

Having said that, you do have a valid point. I own a truck from the late 80s--probably one of the last of the truck trucks--that did away with all the silly options like power steering or power brakes or interior lights or a cigarette lighter or really anything that wasn’t needed for the most basic purpose of getting from A to B. Odds are that you can’t buy a thing like that any more.

5041359

What you're describing in this case is called a 'network storm' or 'broadcast flood' and different technologies have ways of handling it. Modern computer networks use control devices that will not spam a whole network if one device goes rogue. That's the principle behind 'switched' networks and it bemuses me that cars don't have that same architecture. That technology has existed since the 70s!

Car computers can handle that to a point. They do tend to ignore things that aren’t reasonable (that wasn’t always the case, but that’s a blog for a different time). As long as the Side Object Detection Module only spewed nonsense while the car was already running, it would just ignore it and flag a code. The problem was when it happened while the theft system was trying to communicate. And that wouldn’t be an unsolvable problem, of course, but the way this network was designed, it was.

My only thought is that switched networks require all the attached devices to go through one device (the switch or router) and perhaps that presents wiring challenges in a car environment.

In the most simplistic terms, it isn’t a challenge to get the vital systems away from the non-vital ones. But there are failure modes that the designers didn’t anticipate that throw a wrench in the works.

There are generally modules that communicate across networks, and to an extent, they can filter and ignore nonsense from other networks, but the architecture of a car sometimes renders that best practice moot. When it really comes down to it, the only parts of the car that have to perform flawlessly are the safety systems (airbags), and they generally do because of the way that the network is designed. In a nutshell, generally the airbag module is the quiet one, the Fluttershy of the group, but when s:yay:t gets real, that’s the module that has a megaphone, and no matter what any other module wants, the airbag wins.

Also that bat pony is so adorable it *hurts*.

:heart:

5041389

The network stuff is all magic pixie dust to me. Glad to see that translates into everything else as well.

I think the moral of the story might be that cars require magic pixie dust to run.

5041399

My understanding of logic exceeds my understanding of my native English language with those three instructions. To me they simplyfy to, doesnt matter what you test, if you are looking at the device, replace it. :twilightoops:

In some cases, that’s true. If you think it might be a certain module, it is. Replace it.

You trusted your Manager. He is a network problem.

Yes. And a moral problem, but that’s for a different blog.

I thought an Audio Access Module or whatever, especially if theres no documentation, is that thing thats one of the options for Hackers to walk past the network security by adding command codes as an audio file, SMS, phone call etc.

Theoretically, it could be.

Dont forget the basic tenant of complex technology.

Everything is Hostile.
Everythings out, to do what it can. :pinkiecrazy:

Yeah, pretty much.

5041409

Oh man. This reminds me of the absurd amounts of BS we had to go through doing avionics maintenance. After four years, I was convinced that the only way those aircraft managed to fly was because even the groundwanted nothing to do with them.

My brother is an aeronautical engineer, and to hear some of his stories . . . you’re not wrong, some airplanes fly because the ground wants nothing to do with that.

And let’s be honest, that would be a hell of an explanation for how pegasi fly. “Well, you see, we’re an abomination unto the Lord, and thus the very Earth itself repulses us.”

5041501

If I had to guess, the PALCM is like Apple CarPlay & others of that nature

That’s a very reasonable guess, and while I don’t know for certain, that’s what I’d assume unless given documentation to the contrary.

5041531

Thank you for making your suffering so entertaining for us.

:heart:

These are way better than mystery novels.

I do what I can!

5041594

But the troubleshooting never revealed the important question... Was it mud? (She did have horses after all)

Barring DNA tests, I’ll assume it was just mud, and nothing more.

These posts are usually over my head but I always have to make time to read them. I find them so fascinating.

Thank you!

I might not be there yet, but I draw my inspiration from various YouTube channels that explain complex ideas in ways that anybody can at least follow along. Numberphile’s a good example of that--I don’t know the math, but I can sort of follow along with how B follows A.

5041603

These days, does anyone get started just fooling around with cars, or are they too complex?

I can’t speak for these days these days, ‘cause I’ve been the business for almost 20 years. But, I got my start with fixing my own stuff and wondering what made it go, and I have to think that the modern tech wanna-be thinks the same way.

There’s still a fair bit of purely mechanical stuff to draw you in--cars still need brakes, spark plugs, etc., and from there it’s not a stretch to find something more complicated that failed, and do some research and come up with a diagnostic strategy. I think that overall, the guys that make decent techs are the ones who want to know why it works how it works, and everything tends to follow from that.

As an aside, when customers say that cars are so complicated that nobody can work on them, it really grinds my gears, since they’re giving it to me to work on--but I get where they’re coming from. They don’t understand it.

And in the interests of full disclosure, when I have computer problems, I take the mystery box to my more computer-savvy friends and say “fix it.”

it’s not HPHT, so it’s not fullprotection

So you're saying GM likes to build things only slightly skookum?

5041897 Also seen people buy what the dealer has in stock rather than ordering something
Think this is why things like UTVs, mini trucks from Japan are popular amoung some farmers

FTL

5041881

Consider yourself lucky. Your laptop might have a cute little screw, but you don’t have to deal with automakers inventing new fasteners. You got a Craftsman socket set? How do you like this Torx screw? Oh, you bought a set of Torx drivers? How about a triple-square fastener? And you got that covered? How do you feel about a three-lobe?

So true, every time you turn it seems that there is some new screw/bolt type which you need yet another new tool for to work on the latest automotive offering in front of you. The electronics industry is another that just loooves to use the latest random fastener type, seemingly just to drive techs crazy and convince you that it is just too difficult to fix and you should send it back to their 'certified' repairer. :twilightangry2: I'm pretty sure we now have in excess of twenty different screw/bolt driver set types in our lab arsenal... I don't think there are quite a full two dozen yet but its gotta be getting close.

5041455

Is it telling that after hearing so much about your manager that I was not surprised that he found those instructions confusing and that I was even less surprised that he had ordered parts before doing any tests?

There’s little that surprises me with him these days, honestly.

It does make me wonder just what GM's instructions were that he could apparently understand them easier... maybe more pictures? :pinkiehappy:

You know, I’m kind of curious about that, too. I don’t really see how they could be much simpler, but maybe they do have more pictures. These days, since everything’s online, and storage is dirt cheap, lots of newer instructions do have pictures, which I’ll be honest is an improvement over the little sketches they used to give mechanics (if you got anything at all). Ford’s gone with that in a big way, and it really is helpful when you’re taking something apart; they have a picture of the actual assembly with arrows pointing to the screws or clips you need to remove.

For the Acadia I was keen to have a shot and I was initially disappointed that I could not read the diagrams but then I thought to myself, "This is going to go to hell just like any other old 70's 'hub' or 'ring' based network" and I suddenly knew I wanted nothing to do with it!

I think I might be able to figure out a way to get you better diagrams if you want a look at how they do them in cars these days. I have a PDF and put it through MSPaint and then Discord as a webhost (’cause lazy), and it came out in potato-cam quality.

I’ll send you the link via PM; let me know if it works.

Those old networks suffered bus 'storms' and such just like your modern cars... it boggles my mind that what the computer industry worked out was a really bad idea around 30 years ago is the communications topology that automotive engineers use as 'cutting edge' today.

To be fair to the automakers, they’ve got to deal with legacy equipment, harsh environments for electronics, and older standards (I think most of the base of the network protocols date to the 90s, and were built on top of 70s architecture, but I’m not really up on the standards). There are things that they have to do in order to make it able to communicate with scan tools, anyway, and the connector and bitrates were at least initially decided in the early 90s.

The systems generally fail in a benign way, and generally the critical stuff is more robust, but they currently don’t have a way to shut off a module on the network that’s gone rogue, or at least not to my knowledge. I suppose that the BCM--or whoever tells those particular modules to power up--could be programmed to not let them power up.

And maybe if this becomes a pattern failure on GM vehicles, they’ll find a way to make that happen.

:twilightangry2: As you said, your reasoning was sound but the 'intermittent' fault part led you to distraction and heartache.

Intermittent problems are the worst. Even when you think you’ve fixed them, it’s hard to be sure.

In that "Train Wreck" sort of way, eh? :twilightsheepish:

Yes, in exactly that way.

We wait with patience... this sounds like another fun story from the annals of "The Memoirs of the Long Suffering Mechanic"

Luckily, I published that one straightaway, just so y’all didn’t have to spend forever on the edge of your seats. :heart:

5041885

"And we took five bucks off the bill because we pulled out all of the quad-lobed, three-angled special Ford-only connectors and replaced them with hex bolts. You're welcome."

I went the opposite way on an old Cavalier once. There’s a hidden bolt to the alternator (because of course there is) and I wound up losing it. Since I knew it needed all three, I replaced it with one I had that fit. But, instead of being a 13mm hex as it should have been, the one I had was a tailgate hinge mounting bolt off a Chevy truck, which was a Torx.

Assuming that car lived long enough to need another alternator, I bet the next guy was really confused when he finally figured out what drive tool he needed to get that bolt out.

5041922

So you're saying GM likes to build things only slightly skookum?

That’s exactly what I’m saying.

5041946

Also seen people buy what the dealer has in stock rather than ordering something
Think this is why things like UTVs, mini trucks from Japan are popular among some farmers

Oh, yeah, most people aren’t going to go to the trouble to special order a vehicle. I don’t know if you can even still do that, not if you’re just Joe Average anyway (I’ve got a special-order P30, which has an engine that GM didn’t even offer, but when you’re Frito Lay and buying these things by the dozens, you can tell GM what engine they’re going to put in it, and they will). And dealers probably typically don’t order very many of the absolute base model vehicles, since not many people are going to be attracted to them.

Some of those mini-trucks are pretty cool, and for the price you get a lot of nice options you wouldn’t normally get in a similar vehicle (such as a Gator or a quad). The downside is that they aren’t road legal and can’t be made so, but for a farmer, that often doesn’t matter.

5042091

So true, every time you turn it seems that there is some new screw/bolt type which you need yet another new tool for to work on the latest automotive offering in front of you.

I haven’t yet bought all the different sockets and bits that I might one day need, just the ones I use the most. Which has been a sizable investment thus far, let me tell you.

And it isn’t just screws and whatnot, either when you get to automobiles. There are all sorts of clever ways to attach pipes and hoses, and I’ve got a collection of disconnect tools for garter springs, jiffy-tite connectors, and so forth.

The electronics industry is another that just loooves to use the latest random fastener type, seemingly just to drive techs crazy and convince you that it is just too difficult to fix and you should send it back to their 'certified' repairer. :twilightangry2: I'm pretty sure we now have in excess of twenty different screw/bolt driver set types in our lab arsenal... I don't think there are quite a full two dozen yet but its gotta be getting close.

I’ve got a decent collection of those, too, but don’t do enough electronics work to really need the full set. On the automotive side, when modules fail, it’s remove and replace. For most shops, there isn’t a repair option (and assuming you read the next blog post, the condition of the inside of the Side Object Detection Module pretty much precludes repair, anyway, since it probably needs virtually every component and also a board).

5042440 Apparently in some states they are road legal some with some without restriction. Looked into it after seeing one at a local convenience store tagged and everything

5042517
There’s the federal regulations which would generally preclude them being used as an actual motor vehicle (they often lack airbags, emissions controls, etc.); however, I can believe that there are places where you can get around that and licence them as something other than a car. Supposedly in the UP of Michigan, side-by-sides/UTVs can be semi-road legal if they have lights on them and are tagged as an off-road vehicle. At least, that’s what my manager says; I haven’t looked into it.

5042536 Keep in mind some of those being imported are used and predate those emissions requirements. Ohio classifies them as a low speed vehicle and limits them to 35MPH. They are also or were popular at a number of zoos since they tend to have a higher carrying capacity than alot of UTVs
https://www.iihs.org/iihs/topics/laws/minitrucks?topicName=low--and-medium-speed-vehicles

5042544
Yeah, the laws can be weird on things like that, which is why I would never claim to be an expert on the subject. There are all sorts of weird clauses and so forth, as well as local applications of laws vs. an overarching federal regulation. And you’re right on the dates for equipment, too; I got away with having the exhaust fixed on one of my old trucks by claiming it was a 76 and had never had a catalytic converter, and the muffler shop wasn’t overly worried about actually verifying the model year (it was actually a 78, and the driver’s side sun visor had a printed label explaining how you should treat your catalytic converter, but you had to fold it down to see that. . .)

Likewise, my grandma’s car didn’t have seat belts, but since it had not been equipped with them from the factory, it was okay.

I had one of those damn fuel pump driver modules fail on my 08 half ton. God was it aggravating. haha, I looked it up and the first thing I saw was a recall for it saying, "Failure due to water intrusion" or something of the like... well golly gee guys, maybe if you hadnt put it right above the spare tire, it wouldnt get water in it! Turns out, due to the fast they used torque screws to hold the infernal thing to the cross member, which came in from the top mind you, the screws were rusty as hell and full of dirt. Had the break the original module out and wouldn't you know it, a couple tablespoon fulls of water poured out of it...

5042440
Farmers are generally gonna drive it down the road, legal or not. haha

Sorry couldn’t read the post I think my heart stopped when I saw the cover image :trollestia:

5042611

I had one of those damn fuel pump driver modules fail on my 08 half ton. God was it aggravating. haha, I looked it up and the first thing I saw was a recall for it saying, "Failure due to water intrusion" or something of the like... well golly gee guys, maybe if you hadnt put it right above the spare tire, it wouldnt get water in it! Turns out, due to the fast they used torque screws to hold the infernal thing to the cross member, which came in from the top mind you, the screws were rusty as hell and full of dirt. Had the break the original module out and wouldn't you know it, a couple tablespoon fulls of water poured out of it...

The problem wasn’t so much just water, it was galvanic corrosion. Since it was aluminum bolted directly to steel, if there was any salt in that water at all, the module casing was going to dissolve . . . and I bet that practically any of those trucks got some salty water up there. The improved design of the module by Ford has spacers that keep the module off the crossmember, and I haven’t had to replace one of those yet.

I don’t know why they didn’t realize that that would happen; certainly in the boating world galvanic corrosion is well-known, and it’s been a problem on some cars, too.

I hope they accounted for that with their new line of aluminum-bodied trucks; if not, they’re going to wind up with a fleet of F-150s where the bodies just slide off the frames.

Farmers are generally gonna drive it down the road, legal or not. haha

Yeah, that’s very true. And in smaller farming towns, the cops don’t pay much attention to what the farmers are driving.

5043205

Sorry couldn’t read the post I think my heart stopped when I saw the cover image :trollestia:

Well, that’s understandable. :heart:

Login or register to comment