Inside Matthias Nott's Brain

I’ve had increasing problems with two MacBook Pro, one I5, one I7, both from 2010. Reading about badly applied thermal paste on the net, I decided to give that a try. Using this excellent walkthrough from iFixit.com, I was able to eliminate the problems entirely. Monitoring the I7, I can see that the processor temperature dropped from 60 °C down to hardly over 35 °C doing the exact same job. The fan’s are very silent right now. The whole process takes about 30 minutes if you do it the second time; the first time may take you like an hour.

So here’s some material that I used, all from iFixit (the only additional tool I used was a head lamp with a magnifying glass that I recently bought in a road-side store in Taipei). First of all, I used a Magnetic Project Mat. This really helps keeping all those tiny screws in place. Next, I used the Pro Tech Base Toolkit. Those screw drivers are really excellent, very good selection of bits, and excellent grip. Don’t even try with your 5$ selection of screw drivers from Home Depot… The spudger from the set comes in handy for removing all those tiny ribbon cables. The anti static wrist wrap should be a no-brainer for you. Then, I used the “pro” Version of the Probe Set. The tiny tip served very well to remove any remaining old thermal paste from in between of those resistors. By the way, there was so much excess paste on the NVidia Chip that it had run over the resistors (see images below) – one effect that I had was that when watching Flash movies, I had diagonal flashes across the screen quite often. That’s also history now. Next, I used Arctic Silver ArctiClean to remove the old thermal paste. The easiest thing is to use QTips to remove the old paste, as you can work very precisely with them – and thus avoid spreading the old paste everywhere. The removal kit comes with a solvent for the old paste, and once you’ve done that, you use the second component (they are labelled “1″ and “2″) to prepare the surface for the new thermal paste. Bigger bits of paste can be removed with one of the tweezers from the Pro Tech Base Toolkit. Following some guides on the net, for the first computer I applied the new thermal paste using a very tiny droplet of Arctic Silver Thermal Paste on the tip of my index finger that I wrapped into some plastic film. For the second computer, I used QTips wrapped in plastic film, which allowed even more precise application. Since the Thermal Paste is basically metallic silver, you want to avoid at all costs spreading it outside of the application area. I applied a very thin film on both the chips as well as their counter parts of the heat pipes.

Total cost of material used (except the lamp): 141.75 $.

Here’s the first computer after umounting the heat pipes:

As you can clearly see, there is much too much thermal paste, it is everywhere where it shouldn’t be, i.e. outside the chips but not really on them. No wonder that the fans were inefficient to remove the heat!

Here’s the same view, after removing the old thermal paste:

Then, with the new paste applied:

And here are the products used:

The second computer had the exact same problem. This confirms this to be a general problem with MacBook Pro’s:

Again, after cleaning:

This shows how (and, in particular, how little) to apply:

After application of the thermal paste:

So that’s all for now. While assembling everything back together, you can actually benefit from the occasion to remove the battery, pry the white lamp that stupidly keeps flashing when the laptop is in standby mode, and put some black adhesive over the lamp. I haven’t documented that, but you’ll find out yourself. The lamp sits in front of the battery, so you’ll have to remove the battery first (and you’ll need to remove the three very special screws for that; the right bit is part of the Pro Tech Base Toolkit); then you can use one of the metal spudgers of the same toolkit to remove the lamp. It is just glued there, but comes off pretty easily; it can be put back just as easily afterwards).

Hope, this helps…

Having recently upgraded my server by an external housing and an additional SCSI controller (an LSI 9750-8e in addition to the 9690SA-8i), I was experiencing unexpected server crashes that had never happened before on that server.

The server just became unresponsive, sometimes still responded to a ping, sometimes not. Before, it had an uptime since fall 2008, only interrupted by the occasional kernel upgrade.

The server runs on an UPS, and an additional Filter. It has a redundant power supply. For that reason I was excluding problems on that side.

Server Specs: dual 3.00 GHz Xeon E5472 with 32 GB RAM, 7 x Seagate ST310006400SS 1TB drives running in a RAID 10 on the internal controller (1 hot spare; leaving 2.7 TB net space); 5 x SEAGATE ST33000650SS 3TB drives running in a RAID 10 on the external controller (1 hot spare; leaving 5.8 TB net space); both controllers have 512 MB cache each and are both battery buffered. Of course, the NIC is also redundant. The server was provided by Silicon Mechanics.

Now, since the problems arose after the upgrade I immediately thought about the controller. I dumped a log and fired it off to Silicon Mechanics, who showed an excellent level of responsiveness – yet they could not see any problems. They suggested memory to be failing. Yet, because of the correlation with the upgrade, I wasn’t really buying into the idea of memory modules failing just by accident around the same time.

Someone on #linuxger suggested using netconsole. Netconsole is basically a driver that fires off logging information (dmesg) to a remote host using UDP – beginning at a very early stage of the boot process. Since – once the server had failed – I could not log on to the system, I really wanted to get some “pre mortem” error messages. So I took out an old linux notebook, configured netconsole on the server and syslogd on the notebook, and waited.

For several days, nothing happened.

Now since I had to do some productive work, I took up my development VMs and continued my normal day job. I recently started a new project on the nice “Swiss Riviera”, close to my home, where I am to test and automate all sorts of installation procedures for Business Objects products. Since large customers are slow to provide machines, I was as usual resorting to using my own server for setting up all those testbeds. Each of those VMs consumes 8 GB memory. I was using snapshots a lot in order to roll back and try “the next scenario.”

The reason I’m giving you this context is this: In addition to the modification to the server hardware, there was also a modifiation in how the server was used. Even though I had previously used large VMs on that server, I had rarely used a number of them at the same time. Since VMWare is able to release unused memory to the host operating system, I could have started about any number of VMs.

As you can imagine, the server did fail.

Back to netconsole. I saw all sorts of things, but the most relevant were messages like this:

vmware-vmx invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Out of memory: Kill process 332 (upstart-udev-br) score 1 or sacrifice child
Killed process 332 (upstart-udev-br) total-vm:17052kB, anon-rss:40kB, file-rss:440kB
smbd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0

Out of memory: Kill process 622 (portmap) score 1 or sacrifice child
Killed process 622 (portmap) total-vm:8376kB, anon-rss:0kB, file-rss:400kB
rsyslogd invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0

etc. etc.

For some reason, someone was killing my processes arbitrarily, which did explain the fact that a) the server was becoming unresponsive and b) the server was sometimes responding to ping, sometimes not.

What was going on? Google revealed several hits of oom-killer in context with VMWare. The basic idea is that this process kicks in as a last resort of the operating system should it have to reclaim memory. It should start with killing the big memory hogs; in my case, it ended up killing about everything useful. I assume that the actual breakdown happened when e.g. all client machines decided at the same time to do some windows upgrade, leading to a spike in memory usage and thus the oom-killer to kick in. What I found out was that you can actually switch off the whole process by saying

echo 1 >/proc/sys/vm/overcommit_memory

Having said that, I decided to stress test the server a little. I started some of those VMs, used a tool to force memory allocation inside some of those VMs to let them shoot up in memory usage. I was barely using that tool in the first VM when the system resorted to coming to a screaching halt:

top - 11:51:57 up  1:02,  1 user,  load average: 64.24, 49.45, 29.38
Tasks: 187 total,  19 running, 166 sleeping,   0 stopped,   2 zombie
Cpu(s):  0.0%us, 100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  33014440k total, 32857916k used,   156524k free,    87920k buffers
Swap: 31455228k total,     2968k used, 31452260k free, 30577144k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  294 root      20   0     0    0    0 R   34  0.0   0:51.89 jbd2/sda1-8
 1337 root      20   0  586m  41m 6216 S   34  0.1   4:28.28 vmware-vmx

A load of 64… Wow. That’s quite impressive I thought. I started investigating some more. And mind the fact that memory was basically unused (total – cache!), and the swap was also not used.

The conclusion is that the extreme system load is a result uf iowaits while the disk backed memory files (*vmem) were updated. Some people say that you should set mainMem.useNamedFile=”FALSE” in the .vmx files; the idea behind that is to turn of the usage of *vmem files. However, this does not solve the problem, as VMWare Server will use an unlinked (deleted) file in $TMPDIR instead.

In addition, since I was using snapshots a lot, this made matters worse, as apparently this doubles the load on the disks. I went on to fiddle with the vm.swappiness and related parameters, and am giving you here the values that I ended up using (set through /etc/rc.local):

# cat /proc/sys/vm/overcommit_memory
# 0
echo 1 >/proc/sys/vm/overcommit_memory 

# cat /proc/sys/vm/dirty_background_ratio
# 5
echo 2 >/proc/sys/vm/dirty_background_ratio 

# cat /proc/sys/vm/dirty_ratio
# 10
echo 100 >/proc/sys/vm/dirty_ratio 

# cat /proc/sys/vm/swappiness
# 60
echo 0 >/proc/sys/vm/swappiness 

echo 9000 >/proc/sys/vm/dirty_expire_centisecs
echo 3000 >/proc/sys/vm/dirty_writeback_centisecs

From this post, which is a mirror from here and which I’m going reuse here, I learned that first of all, vm.dirty_background_ratio defines the percentage of memory that can become dirty before a background flushing of the pages to disk starts. Until this percentage is reached, no pages are flushed to disk. When the flushing starts, it is done in the background without disrupting the running foreground services.

The parameter vm.dirty_ratio defines the percentage of memory which can be occupied by dirty pages before a forced flush starts. If the percentage of dirty pages reaches this number, all the processes become synchronous, and they are not allowed to continue until the io operation they have requested has actually been performed and the data has been sent to disk.

Now, my understanding is that every virtual machine’s RAM is mapped (with mmap in 1MB blocks) to a file on disk.

I don’t know why VMWare server does not actually use the hosts’s RAM and rather puts the client’s RAM into a file on disk. While this does come in handy if you have to give to your client machines more memory that you physically have on the host, the tradeoff is – well – the extreme load I was experiencing.

Imagine now, that one of my virtual machines thinks it has 8 GB RAM, and I use my memory hog inside the VM to actually claim that memory. Every time the guest RAM is written, the Linux host sees a new dirty page that must then be written to the mmap’ed file on disk. Now 8 GB RAM in the VM is a lot of pages on the host, and therefore the dirty page count can increase rapidly even when the host does simple tasks like caching disk reads, which the guest kernel believes is improving performance, when in fact it is actually slowing down things since it is causing the linux host to perform slower disk writes as a result.

So that’s the onset: The guest changes its RAM, creates dirty pages, and the Linux host kernel needs to decide when to commit these dirty pages back to disk. This can happen in two ways: in the background by a pdflush process, with other processes still running, or synchronously, causing any processes that were dirtying RAM (e.g., vmware-vmx) to wait for the dirty page writes to complete. Clearly the synchronous write option is going to have severe consequences for the guest VM process, explaining the system coming to a standstill.

When the oom-killer was still active, he was at some point taking care of the “situation” by shooting from the hip.

Now as a solution, we need to find a better way to managing writing back the vmem files,

1. avoiding synchronous writes, i.e. having all writing done by the background pdflush process,
2. avoiding accumulation of dirty pages as long as synchronous write becomes unavoidable, and
3. not updating the memory backing file too often since guest RAM can change fast, and we do not want to cause a disk write for every change.

The first point calls for a high value for vm.dirty_ratio. There will surely be good reasons why this is set to a default of 10% in recent kernels. However, for the configuration we’re having here, those 10% are quickly exhausted. For this reason, I maxed this value out at 100%, believing that all suspending processes for disk writes should be a last resort.

The second point calls for a low value for vm.dirty_background_ratio. The pdflush process does not appear to have the same adverse effect on the system performance, which is why I set this down to 2%, assuming that the earlier the background writes are started, the lower the chances are for reaching vm.dirty_ratio and hence forced synchronous writes.

The third point can be controlled by vm_dirty_expire_centisecs which I set to 9000, causing Linux to wait 90 seconds before committing changes to disk. This does affect the whole system, not just VMWare vmem, and caching disk writes for 90 seconds could be considered a “bad thing”. But since both controllers are battery backed and also, the whole system is running on a decent UPS that keeps everything up and running for 30 minutes, and finally we’re on redundant power supplies, this should be safe to do. In addition, this value seems to be ignored once vm.dirty_background_ratio is reached, and thus it does not counter point 1.

Having made those changes, I did the above stress test again, running loads of VMs and maxing out their memory (but just to about 80 % of their memory in order to avoid swapping inside the client machines). While before, already one single memory hog was bringing the system to a grinding halt, I am now able to max out memory allocation for all VMs without any problems.

I hope this information is of use for others. I’m particularly thankful for Graham who’s given me the right way. On the above link, there’s an additional discussion, in particular w.r.t caching on the hardware raid controller: “There is an issue, particularly with later distro’s, using (hardware?) RAID with VMware. This manifests itself with huge iowaits (64s not uncommon). The underlying cause seems to be vmware writing all dirty memory to disk even though everything is configured so that no swapping should take place.” The person posting that has a very similar hardware to this one here. According to 3dm2, caching is active on both controllers here.

A related discussion went on here, and the solution (towards the end of the thread) is in the same context as what I wrote above.

I’m going to investigate the caching further, but the above settings seem to already have improved the situation a lot.

You know what, I really do think you should get a job. I mean you. The kind of brain dead blog spammers who have nothing else to do than to add a “positive comment” into other people’s blogs along with a link to whatever useless service you think you’re providing the world with. Stay away. Please.