Neural network: Using genetic algorithms to train and deploy neural networks: Multithreading

  

Total views

1,679,827

Posted by: Tuyen Pham Thanh 1 year, 5 months ago

When building a model the network will learn the parameters ie its weights. But that requires a set of hyperparameters such as number of hidden nodes, learning rate, number of training steps. Hyperparameters must be appropriate. Since then set a new problem: Optimizing hyperparameters.

We use Genetic Algorithms to optimize hyperparameters. During evolution, the network population will learn hyperparameters.

In principle, we only need the main thread with one network population to perform. However, two networks with different number of hidden nodes will not be convenient for hybridization. So many populations are needed, each with a configuration of number of hidden nodes. This will take a lot of time if running serially on the thread. So we create multiple threads, each for one population. Parallel threads will take advantage of multi-core computer resources and the performance speed of total work will increase many times.

A multithreaded program needs to consider the following issues:

Threading control: When to end a thread. Get its results.
Shared resource protection: When multiple threads access shared resources, we must control which threads are allowed.
Thread-safe functions: Use safe versions if possible. These are functions that use local variables instead of global variables. Some cases are not safe but not serious. For example the erand48 () function is that we only need a little trick to run it once in the main thread before creating new threads. Then the function calls only check the global state and can be considered safe.

See pthread_create (3) to create a thread.
In order to control the termination of a thread we use the futex. After creating sub-threads, the main thread enters the standby state with FUTEX_WAIT. A sub-thread after finishing the job will store the training result in a global list. If it is the last sub-thread, it will call futex with FUTEX_WAKE to wake up the main thread. The main thread gathers the results and then selects and stores the best model. See futex (2) for more details.

Protection of shared resources is done with the semaphore. Using semaphore is very simple. Because the threads are in the same process, we only need to use memory-based semaphore. When passing the sem_wait () function, its value decreases by 1 and will be locked if its value becomes 0. Otherwise, sem_post () function increases the semaphore value and unlocks.

We declare a semaphore as a global variable. Because only one thread is allowed to access shared resources at a time, we initialize the value of the semaphore to 1. When the first thread accesses the data, the semaphore is immediately locked and the other threads must wait until it is done and unlock with sem_post (). When no longer used, the semaphore is destroyed with sem_destroy ()

#define handle_error(en, msg) \
    do { errno = en; perror(msg); exit(EXIT_FAILURE); } while (0)

sem_t sem;
int ret;

ret = sem_init(&sem, 0, 1);
if (ret != 0)
    handle_error(ret, "sem_init");

/*
Use sem_wait () / sem_post () pair whenever you need
to manipulate shared data or access shared resources
*/

sem_wait(&sem);

// Manipulating shared data
// ...

sem_post(&sem);

ret = sem_destroy(&sem);
if (ret != 0)
    handle_error(ret, "sem_destroy");

The program below outputs a random integer in the range [0, 9). The random_r () function is a thread-safe function although it is a non-standard glibc extension

#include <stdlib.h>
#include <stdio.h>
#include <time.h>

int rand_range(int range_min, int range_max,
               struct random_data &rand_data) {
    
    int32_t i;
    random_r(&rand_data, &i);
    int u = (int)((double)i / ((double)RAND_MAX + 1) *
        (range_max - range_min) + range_min);
    return u;
}
                        
int main(int argc, char ** argv) {
    
    struct timespec ts;
    clock_gettime(CLOCK_REALTIME, &ts);
    
    char state[256];
    struct random_data rand_data;
    rand_data.state = NULL;
    initstate_r(ts.tv_nsec, state, 256, &rand_data);
    
    int i = rand_range(0, 9, rand_data);
    
    printf("%d\n", i);
    
    return 0;
}