WTH is this noise my car is making?!

First off I know I know diagnosing noises can be incredibly hard, especially with just a description over the internet, but I need more ideas on what to look for because this noise is driving me insane. The car is a 2001 Subaru Forester with 212k miles.My best description of the noise is it sounds like a marble being dropped into a large old school tupperware bowl.  It sounds like that but multiple times per second. The noise started last November and I've been chasing it ever since but I've got to get to the bottom of it because it's getting worse. The noise sound like it's coming from the rear, possibly from underneath the floor. The noise oftentimes is accompanied with a heavy vibration that can be felt through the floor. The noise occurs both while driving, and while stopped, which leads me to believe it's not suspension or drivetrain related. It mainly presents itself with two different stimuli. The first seems to be engine load/acceleration based, wherein accelerating at just the right rate or just the right throttle amount causes it to happen. A quick lift of the throttle can often cause it to stop. Conversely, and this is kind of weird, just the right rate of deceleration can cause the noise to. The second mode that does it, and much more predictable, is road surface irregularities. There are certain bumps and pavement textures (some that aren't even really that rough at all) that can almost always bring out the noise.

The first time I started hearing the noise was a day or two after putting my car back together from its engine out headgasket change. That day I installed a new aftermarket catalytic converter. The next day I drove is when the noise started. It started faintly but over the months it has become louder and more obnoxious with the vibration I mentioned. I have a hard time believing the cat could have caused this and have such a profound effect.

Since then I have been on a crusade to find this noise. Here are all the things I've done since the noise started, some were more maintenace related but I'll just throw them out there anyway. The common theme is that the noise still remains. New OEM axles (all 4 $$$), new OEM driveshaft, rear differential main bushings and outrigger bushings. Checked trailer hitch mounting bolts multiple times. Checked rear hatch for loose fit/rattle, nothing found. It even does it with the hatch open. I took off the interior panels of the hatch to see if anything was loose, nothing was, added thinsulate sound deadening while I was in there. I've tried driving with all of the loose parts stripped out of the rear like the spare tire, jack, etc. Still noise. All of my suspension parts are tight. License plate is held down with four screws. Checked engine mount torque.

I'm really kind of out of ideas at this point. I love this car and I really want to fix it for my sanity's sake. My best gues is that something is loose and that thing must be heavy enough to cause the string vibrations. But I have so far not been able to find that thing.

Look the entire exhaust system over carefully for a location where it is close enough to touch something (anything) else if it moves from bump or torque. Grab ahold of it in several places along its length and push/pull it towards the nearest obstructions. Your description of variable instigation/cessation points me that way. Amazingly loud harmonics can happen from exhausts striking chassis or suspension components. Especially the way Subarus vibrate.

In reply to freetors :

Ooooh I think I know, the second you mentioned your aftermarket catalytic converter I had a very good idea. You see catalytic converters have ceramic inside of them as part of their inner liner, and this cermaic can be shattered and cause a very similar noise to what you describe. I bet you 3 dollars (I'm a very conservative man) that either you or someone during shipping set your after market cat down a little too hard and broke part of that cermaic liner causing your sound.

I agree with TurnerX19.

I was thinking of exhaust before I even got to the part about the new converter.

In reply to freetors :

Does it sound like this?:   The sound right here

that sounds exactly like what you described

Interesting points about the exhaust/cat. I don't remember hearing anything loose in the cat when I installed it, but that doesn't mean it isn't messed up. It's kind of annoying to take off and check especially since the gaskets are so fragile, including the expensive donut gasket. I've shaken the muffler section around many a time now because I've had that thought and I could never get it to replicate. My trailer hitch has a provision on it to bolt the muffler hanger to and they're fairly close together but I can't get it to make that noise. I've considered removing the muffler section for a while to see if that makes any difference. I've also wondered if it's an exhaust heat shield or perhaps a broken baffle in the muffler.

You might be able to get away with just hitting the catalytic converter (gently) and see if it rattles 

Check the converter heat shield.  Also check the heat shield bolted to the car over the converter.

Subarus all have an extensive array of exhaust heat shields that will start to rattle at about 60,000mi.  No amount of crimping or clamping will solve it.

I tried mostly immobilizing the muffler section by packing really firm foam around it. It made no difference. It still definitely sounds like it could be coming from that area though. Is it possible that a chunk of catalytic converter could have broken off and got stuck somewhere in the muffler?

My heatshields are all bolted down. Though I'm sceptical that those flimsy things could be reasonable for such a heavy vibration that it could be felt in the floor. It honestly feels like somebody knocking on the floor with a hammer.

It has to be something exhaust related. Maybe try removing exhaust sections and see if it disappears? Trouble is that it might be hard to hear the presence or absence of the noise over the increased exhaust noise...

Have you checked your motor mounts? It sounds as though something is shifting from your description.

So here's a little update on this. It's still making the noise of course, but I believe I have finally found the actual source of the noise. Today I was experimenting with pulling off the rear exhaust heat shields and had to lower the rear of the exhaust to pull one of them. That's when I noticed that one of the hanger hooks on the muffler had a flat shiny spot on it. I looked at the trailer hitch and there was a corresponding mark on it too. When it's all installed the hitch tubing is about 1/2" behind the hook. I suppose that as the exhaust heats up it expands lengthwise just enough for the two to make contact. Remember in the first post how I said that this noise didn't start until after I installed the aftermarket cat? Well my theory is that the new cat must be just a little bit longer than the original and screwing everything up. I will be looking for a way to fix this long term.

that is a good place to look. Got a go-pro or something you can bolt on down to watch it while you drive?