Linux threading mystery

[sawyl]

Just before Easter, I was asked to look into the behaviour of threading under Linux. One of my colleagues had noticed that some multi-threaded processes being run on a machine running 2.6.9 generated multiple processes, whereas others generated a single process.

After confirming that both programs were using the same threading libraries, we attempted to investigate whether they were running with the same process scoping. We varied the scope with pthread_attr_setscope() only to discover that the call is not supported under Linux.

We then looked at the applications in more detail and found that the multi-process executable was actually a shell script. This set the environment variable LD_ASSUME_KERNEL to 2.4.10 prior to launching the actual application binary.

We then used a small piece of C to test the hypothesis:

#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <pthread.h>

#ifndef MAX_THREADS
# define MAX_THREADS 2
#endif

void
*sleep_for(void *x)
{
sleep(60);
pthread_exit(NULL);
return 0;
}

int
run_threads(pthread_attr_t *a)
{
pthread_t threads[MAX_THREADS];
int i, status;

printf("Parent process ID: %d\n", getpid());
for(i = 0; i < MAX_THREADS; ++i) {
printf("Starting thread %d\n", i);
if(pthread_create(&threads[i], a, sleep_for, NULL)) {
perror("thread create error");
exit(1);
}
}
for(i = 0; i < MAX_THREADS; ++i) {
pthread_join(threads[i], NULL);
printf("Thread %d rejoined\n", i);
}
return 0;
}

int
main(int argc, char *argv[])
{
pthread_attr_t a;
int scope;

pthread_attr_init(&a);
if(argc == 2 && strncmp(argv[1], "-p", 2) == 0) {
if(pthread_attr_setscope(&a, PTHREAD_SCOPE_PROCESS) != 0)
perror("error setting the scope");
} else {
if(pthread_attr_setscope(&a, PTHREAD_SCOPE_SYSTEM) != 0)
perror("error setting the scope");
}
pthread_attr_getscope(&a, &scope);
if(scope == PTHREAD_SCOPE_SYSTEM) {
printf("Running with PTHREAD_SCOPE_SYSTEM\n");
} else if(scope == PTHREAD_SCOPE_PROCESS) {
printf("Running with PTHREAD_SCOPE_PROCESS\n");
}
run_threads(&a);
exit(0);
}


When the test application was run without any special options, it generated a single process with three threads:

1 linux> ./threads &
[1] 637
2 linux> Running with PTHREAD_SCOPE_SYSTEM
Parent process ID: 637
Starting thread 0
Starting thread 1
2 linux> ps -o pid,ppid,cmd --forest
PID PPID CMD
18773 18771 /bin/bash
637 18773 \_ ./threads
781 18773 \_ ps -o pid,ppid,cmd --forest
3 linux> ps -o pid,ppid,cmd -L
PID PPID CMD
637 18773 ./threads
637 18773 ./threads
637 18773 ./threads
740 18773 ps -o pid,ppid,cmd -L
18773 18771 /bin/bash
4 linux> Thread 0 rejoined
Thread 1 rejoined
[1]+ Done ./threads
4 linux>


When the same application was run with LD_ASSUME_KERNEL set to 2.4.10, it generated four separate processes:

4 linux> LD_ASSUME_KERNEL=2.4.10 ./threads &
[1] 3242
5 linux> Running with PTHREAD_SCOPE_SYSTEM
Parent process ID: 3242
Starting thread 0
Starting thread 1
5 linux> ps -o pid,ppid,cmd --forest
PID PPID CMD
18773 18771 /bin/bash
3242 18773 \_ ./threads
3243 3242 | \_ ./threads
3244 3243 | \_ ./threads
3245 3243 | \_ ./threads
3366 18773 \_ ps -o pid,ppid,cmd --forest
6 linux> ps -o pid,ppid,cmd -L
PID PPID CMD
3242 18773 ./threads
3243 3242 ./threads
3244 3243 ./threads
3245 3243 ./threads
3469 18773 ps -o pid,ppid,cmd -L
18773 18771 /bin/bash
7 linux> Thread 0 rejoined
Thread 1 rejoined
[1]+ Done LD_ASSUME_KERNEL=2.4.10 ./threads
7 linux>


Mystery solved!